Scheduler Activations: Effective Kernel Support for the User-Level Management of Parallelism THOMAS E. ANDERSON, and HENRY (University BRIAN N. BERSHAD, EDWARD D. LAZOWSKA, M. LEVY of Washington Threads are the vehicle for concurrency in many approaches to parallel programming. Threads can be supported either by the operating system kernel or by user-level library code in the application address space, but neither approach has been fully satisfactory. This paper addresses this dilemma. First, we argue that the performance of kernel threads is inherently worse than that of user-level threads, rather than this being an artifact of existing implementations; managing parallelism at the user level is essential to high-performance parallel computing. Next, we argue that the problems encountered in integrating user-level threads with other system services is a consequence of the lack of kernel support for user-level threads provided by contemporary multiprocessor operating systems; kernel threads are the wrong abstraction on which to support user-level management of parallelism. Finally, we describe the design, implementation, and performance of a new kernel interface and user-level thread package that together provide the same functionality as kernel threads without compromising the performance and flexibility advantages of user-level management of parallelism. Process Management; D.4.4 D.4. 7 [Operating [operating systems]: Communications Management– input/ output; tems]: Organization and Design; D.4.8 [Operating Systems]: Performance Categories Sys- General Additional and Subject Terms: Design, Descriptors: Measurement, D.4. 1 [Operating Systems]: Performance Key Words and Phrases: Multiprocessorj thread 1. INTRODUCTION The effectiveness performance lparallelism of parallel computing of the primitives within programs. that Even depends to a great extent on the are used to express and control a coarse-grained parallel program the can This work was supported in part by the National Science Foundation (grants CCR-8619663, CCR-8703049, and CCR-8907666), the Washington Technology Center, and Digital Equipment Corporation (the Systems Research Center and the External Research Program). Anderson was !supported by an IBM Graduate Fellowship and Bershad by an AT&T Ph.D. Scholarship. Anderson is now with the Computer Science Division, University of California at Berkeley; Bershad is now with the School of Computer Science, Carnegie Mellon University. Authors’ address: Department of Computer Science and Engineering, University of Washington, Seattle, WA 98195. Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct commercial advantage, the ACM copyright notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Association for Computing Machinery. To copy otherwise, or to republish, requires a fee and/or specific permission. 01992 ACM 0734-2071/92/0200-0053 $01.50 ACM Transactions on Computer Systems, Vol. 10, No. 1, February 1992, Pages 53-79. 54 T. E. Anderson et al . exhibit poor performance high. Even a fine-grained of creating if the cost of creating program can achieve and managing parallelism One way to construct collection of traditional is low. a parallel UNIX-like program is to share memory processes, each consisting address space and a single sequential space. Unfortunately, because such gramming in a uniprocessor for general-purpose parallel parallelism well. The shortcomings and managing parallelism is good performance if the cost execution stream within processes were designed environment, programming; of traditional that address for multipro- they are simply they handle only processes between a of a single too inefficient coarse-grained for general-purpose parallel pro- Threads separate the notion of a gramming have led to the use of threads. sequential execution stream from the other aspects of traditional processes such as address spaces and 1/0 descriptors. This separation of concerns yields a significant 1.1 performance advantage relative to traditional processes. The Problem Threads can be supported either at approach has been fully satisfactory. User-level threads are managed each application so that intervention. The result PCR  and an order threads are also flexible; level or by runtime in library the , of magnitude the cost of the they of user-level cost kernel. routines thread management operations can be excellent performance: FastThreads within user in linked into require no kernel systems such as thread of a procedure can be customized Neither operations call. is User-level to the needs of the language or user without kernel modification. User-level threads execute within the context of traditional processes; indeed, user-level thread systems are typically built without any modifications to the each process executing underlying operating system kernel. and treats processor,” as a “virtual under pulls threads off virtual processors the underlying its control; each virtual the ready are being kernel. The thread package views it as a physical processor processor runs list and runs them. In multiplexed across real, “Real world” operating user-level reality, physical system code that though, these processors by activity, such as multiprogramming, 1/0, and page faults, distorts the equivalence between virtual and physical processors; in the presence of these factors, user-level threads built on top of traditional processes can exhibit poor performance or even incorrect behavior. Multiprocessor operating systems such as Mach , Topaz , and V [71 provide Programming exhibited direct kernel with kernel by user-level support threads threads, for multiple avoids the because the threads per address space. system integration problems kernel directly schedules each application’s threads onto physical processors. Unfortunately, kernel threads, just like traditional UNIX processes, are too heavyweight for use in many parallel programs. The performance of kernel threads, although typically an order of magnitude better cally an order of magnitude than worse that than of traditional the best-case ACM TransactIons on Computer Systems, Vol. 10, No 1, February processes, performance 1992. has been typi- of user-level Scheduler Activations: threads (e.g., in the absence Effective Kernel Support of multiprogramming and 1/0). . As 55 a result, user-level threads have ultimately been implemented on top of the kernel threads of both Mach (CThreads ) and Topaz (WorkCrews [241). User-1evel threads are built on top of kernel traditional processes; they have suffer exactly The parallel the same problems. programmer, then, employ user-level threads, provided the application kernel 1.2 threads exactly threads, which exactly as they are built the same performance, has been faced with on top of and they a difficult dilemma: which have good performance and correct behavior is uniprogrammed and does no 1/0, or employ have worse performance but are not as restricted. The Goals of this Work In this paper we address this a user-level thread package kernel threads with the dilemma. We describe a kernel interface that together combine the functionality performance and flexibility of user-level and of threads. Specifically, —In the tion, common case when our performance existing user-level system integration). —In the thread operations is essentially thread infrequent case the management when the —No processor idles —No high-priority in the presence thread waits interven- achieved by the best (which must system need kernel as that systems kernel processor reallocation or 1/0, our kernel thread management system: do not same from be involved, can of ready suffer mimic the poor such as on behavior of a threads. for a processor while a low-priority thread runs. – When a thread page fault), to run —The another user-level specific traps thread part between to block the same system is easy (for the thread or from in achieving the the necessary kernel and these control each goals and to the To be able support cations software and 1/0 application’s to manage the needs to be aware request /completions) address simplify policy scheduling an model such application’s multiproces- information space. space. application- for concurrency address of a can be used in a multiprogrammed scheduling allocate processors among applications, the kernel scheduling information (e. g., how much parallelism space). because a different is structured to change example, was running threads, or even to provide a different , Actors , or Futures . The difficulty sor is that from It kernel on which of our customization. application’s as workers to the the processor is distributed To be able to needs access to user-level there is in each address parallelism, the user-level of kernel events (e. g., processor reallothat are normally hidden from the application. 1.3 The Approach multiprocessor, an Our approach provides each application with a virtual abstraction of a dedicated physical machine. Each application knows exactly ACM Transactions on Computer Systems, Vol. 10, No. 1, February 1992. 56 . T. E, Anderson et al. how many control (and which) over operating which system processors of its kernel have threads been are has complete sors among address spaces including processors assigned to an application allocated running to it and has complete on control over those the processors. allocation the ability to change during its execution. The of proces- the number of To achieve this, the kernel notifies the address space thread scheduler of every kernel event affecting the address space, allowing the application to have complete knowledge of its scheduling state. The thread system in each address space notifies the kernel of the subset that can affect processor allocation decisions, for the majority The uler of operations kernel that of user-level preserving thread operations good performance do not need to be reflected to the kernel. mechanism that we use to realize these ideas is called A scheduler activation vectors control from the kernel activations. sched- to the address space thread scheduler on a kernel event; the thread scheduler can use the activation to modify user-level thread data structures, to execute user-level threads, and to make requests of the kernel. We have implemented a prototype of our design on the DEC SRC Firefly multiprocessor workstation . While the differences between scheduler activations that the and kernel threads are crucial, the similarities kernel portion of our implementation required straightforward operating modifications system on the to Firefly. the kernel threads Similarly, the are great enough only relatively of Topaz, user-level the portion implementation involved relatively straightforward modifications Threads, a user-level thread system originally designed to run Topaz Since threads kernel our can when threads are the We emphasize, implemented suffer from solved by implementing the 2. USER-LEVEL In to Faston top of threads. goal is to demonstrate that the exact functionality be provided at the user level, the presentation in assumes that user-level programmer or compiler. models, native of our same at user level problems them THREADS: PERFORMANCE LIMITATIONS this motivate we on top of kernel as user-level our work threads threads—problems on top of scheduler FUNCTIONALITY section concurrency model however, that other used by the concurrency or processes, that are activations. ADVANTAGES by of kernel this paper describing AND the advantages that user-level threads offer relative to kernel threads, and the difficulties that arise when user-level threads are built on top of the interface provided by kernel threads or processes. We argue that the performance of user-level threads is inherently better than that of kernel threads, rather than this being an artifact of existing implementations. additional advantage of flexibility with respect environments. Further, we argue that the lack ited by user-level threads is not but is a consequence of inadequate ACM Transactions inherent kernel User-level threads have an to programming models and of system integration exhib- in user-level support. on Computer Systems, Vol. 10, No. 1, February 1992 threads themselves, Scheduler Activations: Effective Kernel Support 2.1 The Case for User-Level It is natural thread that that could Unfortunately, in the kernel: cost within as user-level of accessing thread optimizations the threads there 57 Management the performance be applied are as efficient tionality. threads — The to believe systems Thread . are kernel, without significant management found yielding the threads compromises inherent costs With operations: in user-level kernel in func- to managing kernel threads, the program must cross an extra protection boundary on every thread operation, even when the processor is being switched between threads in the same address space. This involves not only an extra kernel trap, but the kernel against must buggy thread also copy and or operations techniques malicious can check parameters programs. By be quite are used to expand inexpensive, code inline in order contrast, to protect invoking particularly and perform itself user-level when compiler sophisticated regis- ter allocation. Further, safety is not compromised: address space boundaries isolate misuse of a user-level thread system to the program in which it occurs. With kernel – The cost of generality: ing implementation is used by all kernel thread system must provide application; particular system use it, this imposes feature. overhead In contrast, can be closely since different These were factors inherently instance, However, applications provided that do not use a by a user-level thread matched to the specific needs of the applications that applications can be linked with different user-level policy would on those the facilities thread libraries. As an example, preemptive priority scheduling, can use a simpler thread management, a single underlyapplications. To be general-purpose, a any feature needed by any reasonable most kernel even though thread systems implement many parallel applications such as first-in-first-out not expensive. be important Kernel trap [241. if thread overhead management and priority operations scheduling, for are not major contributors to the high cost of UNIX-like processes. the cost of thread operations can be within an order of magnitude of a procedure implementation, level thread well-written To illustrate implementations call. This implies that any overhead added by a kernel however small, will be significant, and a well-written user- system will have significantly kernel-level thread system. this better performance than a quantitatively, Table I shows the performance of example of user-level threads, kernel threads, and UNIX-like pro- cesses, all running on similar Topaz kernel threads were hardware, measured a CVAX processor. FastThreads and on a CVAX Firefly; Ultrix (DEC’S derivative of UNIX) was measured on a CVAX uniprocessor workstation. (Each of these implementations, while good, is not “optimal.” Thus, our measurements are illustrative and not definitive). Fork, the time to create, schedule, execute The two benchmarks are Null and complete a process/thread that invokes the null procedure (in other ACM Transactions on Computer Systems, Vol. 10, No. 1, February 1992. 58 . T. E. Anderson et al. Table 1, Operation the (the was executed the Table 11300 441 1840 there tuned that with while this of the the path arises between they Flexibility call Topaz written in . and improve 2.2 Sources of Poor Integration Kernel Interface Unfortunately, have the same may and in User-Level in threads code and is highly in choos- flexibility threads, both performance in thread systems modifying likelihood Threads and since there may require specialthreads, supporting require the however, the kernel, of errors in the Built on the Traditional it has proven difficult to implement user-level threads level of integration with system services as is available kernel threads. level, but rather cost management, Topaz User-level important overhead, on assembler. performance models programming complexity, across 7 ,usec. difference thread multiple parallel about between the models, each of which system. With kernel increases a benchmark takes thread are many parallel programming ized support within the thread which kernel. Each averaged kernel that simultaneously is particularly for on a condi- were of magnitude Topaz services time wait 19 psec. difference fact system results order and critical implement tradeoffi is an then together). the about of magnitude despite a tradeoff to there and a procedure takes the Signal-Wait, threads and management is much where flexibility. trap order This two comparison, process another Commonly, avoid For and process/thread, processor, a kernel I shows is yet a thread), a waiting a single while FastThreads. ing of forking of synchronizing on Ultrix Ultrix processes 948 repetitions. between Topaz threads 37 overhead Firefly, (ysec ) 34 to signal tion multiple FastThreads Latencies Fork overhead processlthread Operation Signal-Wait Null words, Thread This is not inherent in managing parallelism is a consequence of the lack of kernel support that with at the user in existing abstraction for supporting user-level systems. Kernel threads are the wrong thread management. There are two related characteristics of kernel threads that cause difficulty: —Kernel the threads user —Kernel thread block, resume, are preempted without notification to threads are scheduled obliviously with respect to the user-level state. These can cause thread system will problems even on a uniprogrammed often create as many kernel threads processors” as there are physical to run user-level threads. When ACM and level. TransactIons on Computer Systems, system. A user-level to serve as “virtual processors in the system; each will be used a user-level thread makes a blocking 1/0 Vol. 10, No. 1, February 1992 Scheduler Activations: Effective Kernel Support request or takes a page fault, though, the kernel threads A plausible on the just-idled solution to this serving 59 as its virtual processor is lost to the address is no kernel thread to run other processor also blocks. As a result, the physical space while the 1/0 is pending, because there user-level thread . processor. might be to create physical processors; when one kernel thread thread blocks in the kernel, another kernel user-level threads on that processor. However, 1/0 completes or the page fault returns: there more kernel threads than blocks because its user-level thread is available to run a difficulty occurs when the will be more runnable kernel threads than processors, each kernel thread in the middle of running a user-level thread. In deciding which kernel threads are to be assigned processors, the operating system will implicitly choose which user-level threads In are assigned a traditional processors, the processors. system, operating when there system could are more employ runnable some kind threads than of time-slicing to ensure each thread makes progress. When user-level threads are running on top of kernel threads, however, time-slicing can lead to problems. For example, a kernel thread could be preempted while its user-level thread is holding a spin-lock; any user-level threads accessing the lock will then spin-wait until the lock holder is rescheduled. Zahorjan et al. [281 have shown that time-slicing in the presence of spin-locks can result in poor performance. As another example, a kernel thread running a user-level thread could be preempted to allow another kernel thread to run that happens to be idling in its user-level scheduler. Or a kernel thread running a high-priority user-level thread could be rescheduled in favor of a kernel low-priority user-level thread. Exactly the same problems page faults. If there machine’s processors; should preempt occur thread with that happens to be running multiprogramming as with a 1/0 and is only one job in the system, it can receive all of the if another job enters the system, the operating system some of the first job’s processors to give to the new job [231. The kernel then is forced to choose which of the first job’s kernel threads, and thus implicitly which user-level threads, to run on the remaining processors. The need to preempt processors from an address space also occurs due to variations the in parallelism dynamic variations within reallocation jobs; of processors in parallelism is important Zahorjan and among address to achieving McCann [271 show spaces high that in response to performance. While a kernel interface can be designed to allow the user level to influence which kernel threads are scheduled when the kernel has a choice , this choice is intimately tied to the user-level thread state; the communication of this information between the kernel and the user-level negates many of the performance threads in the first Finally, ensuring and place. the logical flexibility correctness advantages of of a user-level using thread user-level system built on kernel threads can be difficult, Many applications, particularly those that require coordination among multiple address spaces, are free from deadlock based on the assumption that all runnable threads eventually receive processor time. When kernel threads are used directly by applications, the kernel ACM Transactions on Computer Systems, Vol 10, No. 1, February 1992. 60 . satisfies runnable number T. E. Anderson et al this assumption by time-slicing the processors among all of the threads, But when user-level threads are multiplexed across a fixed of kernel threads, the assumption may no longer hold: because a kernel thread blocks when its user-level thread blocks, an application run out of kernel threads to serve as execution contexts, even when there runnable user-level threads and available processors. 3. EFFECTIVE KERNEL SUPPORT FOR THE USER-LEVEL can are MANAGEMENT OF PARALLELISM Section the ity) 2 described programmer and when behavior problems, system the problems that arise to express parallelism user-level threads are in the presence of multiprogramming we have designed a new kernel that together combine the own chine virtual except machine this multiprocessor, that during the the the kernel may execution kernel and interface functionality performance and flexibility of user-level The operating system kernel provides its when 1/0). and of kernel threads. each user-level abstraction change of the threads are used by (poor performance and poor flexibilbuilt on top of kernel threads (poor To address these user-level thread threads thread of a dedicated the program. number with system with physical of processors There the are several ma- in that aspects to abstraction: –The kernel allocates processors to address spaces; the kernel has complete control over how many processors to give each address space’s virtual multiprocessor. —Each address space’s user-level thread system has complete control over which threads to run on its allocated processors, as it would if the application were running on the bare physical machine. –The kernel changes thread notifies the number system the user-level of processors whenever thread system assigned a user-level whenever to it; the kernel thread blocks the or wakes up kernel (e.g., on 1/0 or on a page fault). The kernel’s role is to vector to the appropriate thread scheduler, rather than to interpret these on its own. —The user-level thread system notifies the kernel when kernel also notifies the in the the events events application needs more or fewer processors. The kernel uses this information to allocate processors among address spaces. However, the user level notifies the kernel only on those subset of user-level thread operations that might affect processor allocation decisions. As a result, performance promised; the majority of thread operations do not suffer the communication — The from with application the kernel. programmer programming directly system manages its virtual mer, providing programmers user-level different sees with no difference, kernel multiprocessor a normal runtime system could easily parallel programming model). ACM Transactions is not comoverhead of except for performance, Our user-level thread transparently to the programTopaz thread interface [41. (The threads. be adapted, on Computer Systems, Vol. 10, No. 1, February 1992 though, to provide a Scheduler Activations: Effective Kernel Support In the remainder to the user-level tion to allow user-level 3.1 of this thread section system, the kernel we describe what to allocate how kernel information processors events is provided among . 61 are vectored by the applica- jobs, and how we handle spin-locks. Explicit Vectoring The communication of Kernel between Events to the User-Level the kernel processor Thread allocator Scheduler and the user-level thread system is structured in terms of scheduler activations. The term “scheduler activation” was selected because each vectored event causes the user-level thread system to reconsider its scheduling decision of which threads to run on which processors. A scheduler activation serves —It serves as a vessel, exactly the same way —It notifies –It provides or execution that the user-level space three in roles: context, a kernel thread the system kernel for running thread for user-level threads, in does. of a kernel saving the event. processor context of the activation’s current user-level thread, when the thread is stopped by the kernel (e.g., because the thread blocks in the kernel on 1/0 or the kernel preempts its processor). A scheduler traditional activation’s kernel — one mapped space. Each starts running into data thread. the user-level ; when Each structures scheduler are quite activation kernel and thread is allocated one mapped a user-level its thread activation’s kernel stack. The user-level tion’s user-level stack. In addition, the into own calls thread kernel similar to those has two execution the application user-level into the scheduler maintains address stack kernel, of a stacks when it it uses its runs on the activaan activation con- trol block (akin to a thread control block) to record the state of the scheduler activation’s thread when it blocks in the kernel or is preempted; the user-level thread scheduler maintains a record of which user-level thread is running in each scheduler When activation. a program is started, the kernel assigns it to a processor, and upcalls into fixed entry point. The user-level thread creates a scheduler the application management activation, address space at a system receives the upcall and uses that activation as the context in which to initialize itself and run the main application thread. As the first thread executes, it may create more user threads and request additional processors. In this case, the kernel will create an additional scheduler activation for each processor and use it to upcall into the user level to tell it that the new processor user level then selects and executes a thread in the context is available. The of that activation. Similarly, when the kernel needs to notify the user level of an event, the kernel creates a scheduler activation, assigns it to a processor, and upcalls into the application address space. Once the upcall is started, the activation is similar to a traditional kernel thread— it can be used to process the event, run user-level threads, and trap ACM into Transactions and block on Computer within Systems, the kernel. Vol. 10, No. 1, February 1992. 62 T. E. Anderson et al. . Table II. Add this processor (processor a mnnab!e Execute Processor has been ready context of the preempted activation The blocked Scheduler list has scheduler activation to the ready context of the blocked (preempted the user-ievei scheduler Points blocked (blocked activation state) zn the #) using (unblocked wet--level scheduler # and its machine was executing that as no longer unblocked l~st the activation thread activation. activation has Return Upcall thread preempted to the Activation #) user-level Return Scheduler Scheduler thread itsprocessor. activation that was # and its machine executing in state) the act~uatton. The crucial distinction that once an activation’s between scheduler activations and kernel threads is user-level thread is stopped by the kernel, the thread resumed is never activation directly is created has been stopped. thread from to notify The user-level the old activation, by the the kernel. user-level thread tells Instead, thread system the kernel system then removes that a new that scheduler the thread the state of the the old activation can be reused, and finally decides which thread to run on the processor. By contrast, in a traditional system, when the kernel stops a kernel thread, even one running a user-level thread in its context, the kernel never notifies the user level of the event. Later, the kernel directly by implication, its user-level thread), again resumes without the kernel notification. thread (and By using scheduler activations, the kernel is able to maintain the invariant that there are always exactly as many running scheduler activations (vessels for running user-level Table scheduler II threads) lists the as there events activations; that are processors the kernel the parameters assigned vectors to the address to the to each upcall user space. level using are in parentheses, and the action taken by the user-level thread system is italicized. Note that events are vectored at exactly the points where the kernel would otherwise be forced to make a scheduling decision. In practice, these events occur in combinations; when this occurs, a single upcall is made that passes all of the events that need to be handled. As one example of the use of scheduler activations, Figure 1 illustrates what happens on an 1/0 request/completion. Note that this is the uncommon case; in normal operation, threads can be created, run, and completed, all without kernel intervention. Each pane in Figure 1 reflects a different time step. Straight arrows represent scheduler activations, s-shaped arrows represent user-level threads, and the cluster of user-level threads to the right of each pane represents the ready list. At time Tl, the kernel allocates the application two processors. On each processor, the kernel upcalls to user-level code that removes a thread from the ready (thread ACM list and starts 1) blocks Transactions on in the Computer running it. At time kernel. Systems, To notify Vol 10, No. T2, one of the user-level the 1, February user level 1992, of this threads event, the Scheduler Activations: Effective Kernel Support ‘m . 63 Time User Program ............................ .. .... .......... .......... J($ (A) p fq? ~ u (B) u ‘2 ,,: ‘(c) A’s thread has blocked 1 I ..“U!!!!-J ................................. Time T4 User Program . .. .. .. .. ... .. .. . .. ... .. . .. ... . .. .. ... . .. .. .. ... . .. .. . User-Level Runtime System ~m~ ‘(c) “(B) ‘(.4) ‘(D) Operating A’s thread System and B’s Kernel thread can continue Fig. kernel takes upcall in the scheduler 1. the processor context can then Example: that 1/0 completes. em running scheduler use the processor I request/completion. had been of a fresh and start running it. At time T3, the 1/0 i to take Again, thread activation. another the kernel 1 and performs The user-level thread must an thread off the ready notify list the user-level thread system of the event, but this notification requires a processor. The kernel preempts one of the processors running in the address space and uses it to do the upcall. (If there are no processors assigned to the address space when This the 1/0 upcall preemption. puts the completes, notifies The upcall thread that the upcall the user level invokes had been must wait until of two things: the kernel the 1/0 code in the user-level blocked on the ready allocates completion thread list system and one). and the that (2) puts (1) the thread that was preempted on the ready list. At this point, scheduler activations A and B can be discarded. Finally, at time T4, the upcall takes a thread off the ready list and starts When a user level thread running it. blocks in the kernel or is preempted, most of the state needed to resume it is already at the user level —namely, the thread’s stack and control block. The thread’s register state, however, is saved by low-level kernel routines, such as the interrupt and page fault handlers; the kernel passes this state to the user level as part of the upcall notifying the address space of the preemption and/or 1/0 completion. We use exactly the same mechanism to reallocate a processor from one ACM Transactions on Computer Systems, Vol. 10, No. 1, February 1992. 64 T. E. Anderson et al. . address space to another due to multiprogramming. For example, suppose the kernel decides to take a processor away from one address space and give it to another. The kernel does this by sending the processor an interrupt, stopping the old activation, and then using the processor to do an upcall into the new address space with a fresh activation. The kernel need not obtain permission in advance from the old address space to steal its processor; to do so would violate the semantics could have address the the old dress space space on its an address then but control we could instead, it as a processor. processors it has simply we reallocates been over (When skip delay the notifying case the it is running into the The ad- user-level threads should be is preempted the address until allows in stopped. processor old an upcall notifying of these last notification Notification assigned, been which the still to make space the The kernel processor activation, have address However, occurred. is used scheduler threads full space). the preemption processor a fresh has (e. g., the new address on a different second processors. space, preemption, that user-level remaining old preemption The using two scheduler run the be notified space. that space priorities than another address address thread still by doing old of address priority space must does this in higher the user from space kernel level of to know explicitly the eventually which managing cache locality). The above threads beyond 3 is description have the ones lower thread will system system level to put can which terms of the of the page same data Third, In was in the event. until ACM manager a different loop, only must page Transactions added for may in and when on in For whose be made kernel page it Systems, Vol. 10, No 1, February user or no user-level its on the the by stack, thread is processor during processing upcall same the allows an event an a thread is saved own handling is level. a preemption to continue 1992 other no knowledge if a preempted delay of in needs is that fault occurs, any if a user-level example, the turn way context build state with one the behavior when completes. Computer user exactly is entirely to when if it was can this, The kernel’s at the even activation is necessary; fault free kernel manager if not. of a page fault the manager a new saving the parallelism thread complication kernel thread it knows application is properly switch The user-level 2 instead. and the activations; particular, check for the thread user-level 3’s processor. application to recover context the if place processors. thread way no action a user-level The the in First, take 1, suppose because with in to case, thread as stopping it is the thread program kernel case, that run interaction work upcall, idle an upcall, the this and of its user-level Figure allowing kernel In in thread and respects. have In preemption to represent subsequent (reentrant) running, in the list The case. used 2. upcall, of scheduler activations occurs The top every structures is running. kernel. the in scheduler fault kernel’s on an additional the minor may example 1 and ready the activations. model the to preempt do on each the scheduler concurrency exactly for we described threads, In to 3 on the several preemption threads kernel is running while user-level the to ask thread both processor thread know Second, ask that in additional above. than can use an described priority then is over-simplified priorities, location; subsequent the to notify the upcall Scheduler Activations: Effective Kernel Support Finally, a user-level execute further resumes point the it part of the 3.2 Notifying has mode the leave user level, straints are not by the Kernel met any key of User-Level affect the while minimizing best our more to to notified an away, be to user on top level but the state as Allocation next as we system about the By — a thread that cache need they not subset contrast, traps are tell the that can when to the —is kernel kernel respects context and con- threads. small a processor preserving know, of kernel thread only decision. run and all that there the Table III even priorities within the same lists the that address has when the idle may that simply since apparent the longer to If their get only kernel has more The has not taken kernel to this address needed thread for none need approach spaces, an are the the system if have time must space it gets state that that idle there, the the updated serialize its but in an address it has happens, becomes gives with these kernel nothing one kernel kernel on the address found, if the space notifies a processor by to address space hints: processor user-level ordering drawback in reporting extra application system. a or additional Creating an extension than an searches processors. the by address kernel are is no returns Of course, in the rather made an eventually notifications a processor threads, the (An has it busy. if makes processors, application be and work it than reassigned must processors. calls example, space These idle not Similarly, processors be no other kernel processors, unique idle an has system has where needs more the whenever threads priorities). For kernel, in kernel Provided allocator constraints. must situation transitions. registered it the runnable the of additional meaningful more threads. processors kernel then notifies has processor violate notified space it runnable the cannot globally not it address than handles An the parallelism, where and then the them honest the that register Processor as far built allocation for for a state parallelism the Affecting systems; system operation, overhead run processors, tion. Events thread is that thread thread processors space reaches occurs thread’s tO kernel policies that both respect priorities exist. These idle if runnable threads thread directly system, threads future. user-level or latter need the space. transition the again the still so, last subsection is independent of the policy processors among address spaces. Reason- kernel processor used the not most kernel’s when In by user-level every are address may If however, must be based on the available parallelism In this subsection, we show that this information can observation about threads blocks is when the kernel completes. either It passing be efficiently communicated for guarantee that processors do not The it kernel. in the 1/0 65 upcall. able allocation policies, in each address space. kernel the until the The mechanism described in the used by the kernel for allocating met blocked when temporarily, would notifies that kernel thread where kernel thread in . the space address informa- notifications to matters. to this approach parallelism multiprocessors: to the a dishonest ACM Transactions is that operating or applications system. misbehaving on Computer Systems, Vol may This not problem program 10, No. 1, February be is can 1992. 66 T. E. Anderson ● Table III. Add et al. Communication more processors Aliocate more running them This processor sor an unfair as well. feedback for In can that elsewhere, has penalize similar goal; of idle and with this policy idle when the in could easily processors the be future, will system left to do in the avoid something the similar. performance, service, often service is improved approximated time. multiprocessors be adapted This needed is created. accumulate multiprogrammed the address more. are should systems remaining use if overall near work favor they that hand, interactive as they that when the information can processors in multi-level honest those is likely work uniproces- systems, allocator other operating least at. We by expect to achieve a the to encourage honest reporting a user-level thread could processors. 3.3 Critical One issue There SectIons we executing have not in a critical are two yet possible continue to test , deadlock ready list lock; preempted are not per-processor improve avoided ; and Prevention onto through and user use of upcall lists approaches a scheduling occur of thread also must to dealing and has by the could pre-empted to place even when locking a number uses control critical the protocol to atomically. problem preemptions of serious the the unlocked blocks be done with 1 other be holding attempted inopportune prevention, because FastThreads lists free Prevention can example, free two With if the Problems For held thread be or preempted. (e.g., spin-lock list). to these are level. performance occur a lock. it is blocked preempted per-activation) the the the ready accesses recovery when poor would by preemption. that application-level (e.g., the (really, latency inopportune an protected is instant effects: if so, deadlock thread sections at the ill threads and addressed section thread) kernel the create of jobs to be used it the especially the priority policy to reallocation time, jobs the On processors, likely start needs to provide processors that needed. uniprocessor response favoring same than most up imply are of processor reducing give priorities they production Average by the applications to space thread and spaces and on a multiprogrammed user-level processors threads Many or processor spaces overhead of resources address The when fewer if another kernel-level fewer needed) space () decisions. since address activattorw. to encourage address be returned address allocation use encourages scheduler proportion Space to the Kernel # of processors to this be used processor spaces (additional thzs processor either the Address processors is idle Preempt consume from between of are the drawbacks, 1 The need for critical sections would be avoided if we were to use wait-free synchronization . Many commercial architectures, however, do not provide the required hardware support (we assume only an atomic test-and-set instruction); in addition, the overhead of wait-free synchronization can be prohibitive for protecting anything but very small data structures. ACM Transactions on Computer Systems, Vol. 10, No. 1, February 1992 Scheduler Activations: Effective Kernel Support particularly kernel in a multiprogrammed to yield user-level, control violating is inconsistent will vention the with describe in address Finally, “pinning” to in (at in requires priorities. presence memory section; 67 the temporarily) of critical the physical a critical least space implementation 4.3. while Prevention allocation of efficient Section be touched environment. processor semantics the requires might over . to the Prevention sections of all page virtual identifying that we faults, pre- pages these that pages can be cumbersome. Instead, we user-level adopt thread thread system course, this a solution system checks check based that if must the a user-level context temporarily via exits the section, upcall, again user-level continue activation on the if it locks). switch. When back At this point, list. We use the in the middle preempted the the section. If so, the control switch. ready was a critical any relinquishes context back in informs or unblocked, the to it continued the original is safe same (Of thread to place mechanism of to processing a event. This technique ensure the thread an kernel a user-level it an upcall preempted executing acquiring is continued the was When been before thread via has thread be made critical on recovery. a thread that presence supports lock from spin-locks, is to relinquish the is a spin-lock processor faults. lock the not affected, takes spinning processor for system continue a page the time fault; a while in technique thread to we even this user-level it holder, released, Further, allowing holder after the eventually since occurs, correctness when continuing is always or page a preemption Although spin-waiting By it preemptions user-level when holder. deadlock. is acquired, of processor notified wasted free a lock arbitrary always this is once is spin- may be a solution to . 4. IMPLEMENTATION We have implemented the native operating tion, and We FastThreads, modified scheduler it now Table processors the Topaz Where In address described the DEC spaces to thread upcalls which existing blocked, the modified they Topaz user modifying Topaz, worksta- Topaz routines to implement resumed, or preempted level Topaz to to take do scheduled belonged. (and 3 by multiprocessor package. formerly formerly, Section Firefly management to allow we spaces; in SRC thread Topaz addition, to address compatibility; for kernel performs II). design a user-level the activations. a thread, (see the system We therefore also UNIX) these actions explicit allocation of threads obliviously to object code maintained applications still run as before. FastThreads sections, allocation In all, lines modified to provide decisions we was concerned a few (For over with and (see added to Topaz. threads below), was and not to process Topaz with Table III). hundred comparison, 4000 lines the lines of code). implementing the with scheduler ACM upcalls, the of code original The for to FastThreads majority per on Computer interrupted needed Topaz processor activations Transactions to resume information its and implementation of the allocation new critical processor about 1200 of kernel Topaz policy code was (discussed se. Systems, Vol. 10, No 1, February 1992. 68 T. E. Anderson et al. . Our is “neutral” design address pair spaces of policies briefly Processor The processor Zahorjan specting do. in allocation guaranteeing divided the Our rather the tions. Continuing compatibility In implementation, our in kernel requiring with that every Topaz kernel existing same way allocator processors or application can be neither state; processor, received in it, and interface described in scheduler activations; address spaces dress spaces scheduler threads using are is no need Thread An important aspect to application’s lists cache locality; choose The accessed is essentially by policy our design ACM Transactions kernel threads compete it each has for user-level it for to ad- thread using As a The spaces assigned spaces scheduler. yet). provide Processors thread (An for address structures to the idle. asked information address address are processor handed for activations. another to the at as appropriate; default each used it the policy by idle for in work kernel a result, there the Each can FastThreads list struc- application to is fit the per-processor order ready of an data be tuned uses last-in-first-out own of to improve is empty. This . includes processor is available no knowledge or level. they if its Multilisp has policy, user in processor scans kernel scheduling these an that is that or parallelism implementation reallocations; the to preserve applications. that if kernel activa- Topaz whether data use of processors. model a processor the addition, notifying to needs. ready In of our manage free Topaz to Policy concurrency used are assigned only number scheduler scheduler know directly. activations partitioning Scheduling completely processor static this of the necessary processors kernel threads original use for provides processors kernel that instance, asked time-sliced space was use to share, Space-sharing use sequential) for internal kernel to the space some address them. address needs of re- is work in their are address that policy while if there multiple an threads has not 3.2 scheduler handed application’s tures use upcalls; for 4.2 Section that via only has that want applications more use debug- priority remainder. an integer (possibly use could idles processors the for spaces as dynamic highest of the that processor space we and processors processors is not possible address the all among priority) it to support binary processors same to the the reallocations; makes than need processors implementation threads, some implementation; no processor among evenly of processor (at course, follow. that do not divided of available spaces to processors Of enhancements “space-shares” evenly spaces are number prototype is similar chose policy and number that we The are of address The . priorities reduces our allocating processors. performance subsections policy McCann processors for as some for onto Policy if some address those selected the of policies threads Processors spaces; if the be as well Allocation and choice scheduling to these, considerations, 4.1 on the for had describe ging to and for hysteresis spins for to a reallocation. on Computer Systems, Vol. 10, No. 1, February 1992. avoid short unnecessary period before Scheduler Activations: Effective Kernel Support 4.3 Performance While the as just equivalent considerations The Section In a user-level can be resumed), the thread flag when thread check to see if it being temporarily upcall when acquisition even though since we thread and We adopt collector section in the post-process would the is for the when it this to the imposes set and a then thread to the original overhead or page fault is particularly in and whether thread so that a preemption Latency kernel leaves, processor sections place code the the on occurs, important building our user-level not the A second it notifies its copy we then copy. This compiler support. At critical section, we execution kernel this to see if it was a new activation in one of these corresponding relinquishes uses starts system; at the critical each critical the Normal the thread thread place control can user in level the the in to the back can upcall ACM be be an old and cached must call, the of an in call explicit but flag). to the management activation is created for is it the free, had eventual case user been reuse. running level Vol. The that userit to is removed the of blocking be- Instead, by returning processing Systems, however, initialized. activation after on Computer not and for in the notifies latency relates allocated of preemption, that clearing scheduler preemption; on lock a procedure scheduler thread ‘Ikansactions impact case, we bracket this activation to user-level case of the no enhancement scheduler recycle as the In setting a new structures activations system context: the is occasionally section. with a new data there implementation, Logically, Creating processing The performance as soon the resumer. counter preemptions, our path, scheduler kernel of the user-level the make version occurs, program a critical line requires thread from about (In activations. discarded level the section. significant upcall. cause notify low-level package; to and to the common Trellis/Owl labels, thread code the the every language original back of assembler user-level preemption thread’s within straight of scheduler the a critical case. from the if so, continues concerned be made not copy special in in at the end of the critical section. Because normal execution code, and this code is exactly the same as it would be if we upcall uses the original common exact assembly processor to and of the original for no overhead a uniprocessor to do given When preempted copy each the activation the an with code but code. sections, were source to yield scheduler critical make on compiler-generated copy, original checks We imposes used be straightforward of the that was by delimiting, C the also end after func- additional to check is needed the Unfortunately, critical in be able flag check or not solution . We do this the The blocks do this the infrequent. continuable technique section. not to it must relinquish place. are when clear whether events different a a related garbage the a safe these way continued. release these use One will some critical sections, described in continuation of critical sections system section, continued user-level are system. case; the a lock. is being (or thread a critical it reaches lock to is preempted holding enters there performance. relates temporary user-level to provide threads, for these to provide the was it of order is sufficient of kernel are important significant 3.3. when described to that that most 69 Enhancements design tionality . upcall in the that kernel, resumption 10, No. 1, February is 1992. 70 T. E. Anderson et al. . possible. A similar tions: kernel future thread in occasional scheduler bulk, bulk traditional deposit same kernel kernel 4.4 have There the two its system itself single-steps use speed the to same the number of as a preemption system, when returned Ignoring the one 1/0 crossing is completes. The system. with the with debugging Firefly their own application Topaz debugger. needs: debugging code running on top the processors, but debugged. a debugger of instructions user-level this is Instead, logical the these thread system kernel do not state the each of thread scheduler debugger cause kernel of the assigns the as little The when when events have debugged. inappropriate processor; activation, should being stops upcalls into or the system. the the debugging–the informs debugged thread user-level facilities code thread of the running system thread in is working system the context to combine to correctly, stop and the examine of a user-level debugger the thread state of [181. PERFORMANCE The goal with the the sequence a scheduler application 5. to on the is being Assuming can to and a time. processor thread each and is crucial described being user-level or our activations system physical activation in at makes 1/0 is needed environments, thread we have of collected be one a kernel In occur scheduler as possible each can implementa- destroyed system. Transparency support thread when system on another crossings separate thread effect kernel cached Considerations user-level of the and boundary our crossings system. 1/0, integrated are many be returned of discards, thread an Debugging We of being boundary to start in can activations instead application-kernel needed created, [131. discarded kernel is used once creations Further, the optimization threads, of our user research and level each flexibility within issues performance, thread we have cost fically, flexibility fork, of upcalls)? what is space. the The previous in kernel overall effect In cost system? and the on at functionality is the our threads parallelism sections. what yield) the of kernel of managing First, and between Third, in questions. block of communication functionality address addressed three (e, g., the advantages application been consider operations is the is performance user the and terms of of user-level Second, what level (speci- performance of applications? 5.1 Thread The cost of user-level as those —that for in thread FastThreads running on integration. original tained ACM of the is, system Performance FastThreads, Table Transactions I. on top Table Our Computer operations package of Topaz IV adds Topaz system Systems, in kernel the kernel 10, system on the threads, performance threads, preserves Vol. our running the No order 1, February is essentially Firefly with the of our and associated system Ultrix of magnitude 1992 the to our prior to the processes same work poor data con- advantage Scheduler Activations: Effective Kernel Support Table Signal- Wait that user-level tion in Null and the kernel on threads Fork decrementing must inform number be notified. (This available). There the (in plus which an when a 48 lock psec. is (The than 5.2 Upcall Thread nel Null does Fork (Section is important, point, level that to those threads held When to be Our we by the this time an onds, a factor We see from time in its of execution is by 1/0 the request of five nothing difference, in application the inherent which upcall we ACM may is threads similar between it a to the activations latency have user or preempting scheduler the the at user-level activations then determine be done of blocking reschedules we overhead two user-level to the to when determines to wait for how a critical the One to signal test kernel. in and Table This operations. measure IV, approximates fault. signal-wait time in scheduler through except of making The of upcall wait machinery Topaz Transactions that. performance thread activation than attribute than upcall kernel threads in our Topaz Signal-Wait be a page expected of slower scheduler or can for ker- infrequent thread. forced worse cost Further, the case when it helps that thread, when the for First, scheduler a kernel is analogous added using is considerably is the pleting this perfor- of marking optimization frequent needed If the implementation, with the intervention, preempted synchronization overhead codes). a Signal-Wait sections operations on a uniprocessor. our implementation the to this our performance—the reasons. threads. and commensurate kernel; several kernel running began performance the critical Upcall of thread preempting even is preempted resource more characterizes kernel the or be practical other for kernel in of blocking long 5.1) ratio require thread a thread has and are resumed condition way this ,usec. is due threads, it as is being the kernel Removing of 49 necessary. the to outperform user-level could not though, “break-even” to begin which restore that processors a zero-overhead 4.3). time benchmark many thread than without Section Fork to running parallelism Performance performance case—is done better whether Signal-Wait.) involvement cost (see a Null as to increment- a program in Signal-Wait, be worse for a preempted of magnitude held yielded path must is due determining sufficient wants whether be significantly FastThreads always processes 11300 1840 is a 3 psec. degrada- and be eliminated with Ultrix which threads degradation work order There threads. running it of checking extra still would that ToPaz threads 948 441 FastThreads, could 71 Latencies(Wsec) of busy or is a 5 psec. cost case Although mance machine kernel factor over kernel to original the the Operation FastThreads on Scheduler Activations 37 42 offer relative a uniprogrammed can Thread FastThreads On Topaz threack 34 37 Operation Null Fork ing IV. . the and is 2.4 that com- millisec- threads. activations to two on Computer that is responsible implementation Systems, Vol issues. 10, No. 1, February for First, 1992. 72 T. E. Anderson et al, . because we built implementation state, the and thus kernel in have from written in + . For assembler. commensurate comparison, by we expect with than what our if tuned, kernel thread be Topaz to the existing more maintain that portion thread Burrows by our  the reduced As next a production on application + in would be a result, the ap- are somewhat section in SRC Modula-2 performance performance. is is entirely recoding upcall of system implementation four in achieved designed and of that, had of the kernel a factor measurements might if we much Schroeder over Topaz performance worse than importantly, assembler; costs Thus, as a quick modification thread system, we must overhead, As tuned processing plication more scratch. carefully Modula-2 RPC scheduler activations of the Topaz kernel scheduler activation implementation. 5.3 Application Performance To illustrate sured the the FastThreads ning effect same built on on scheduler solution to the ing the center the tree masses were all this force on each the cache cache data. fit are processor Firefly’s we demonstrate when With the are sequential system no to kernel threads Transactions of a cluster of size was if they cache for and were into physical account; memory systems, tests by buffer 50 msec.; measurements disk the simplifica- kernel the generation the a further (Our the for used so that the for ) All memory, modified of memory As in performance illustrative. We chosen memory. current available amount contention than on Topaz threads the number three of the the for Computer enough of the all parallelize the application on our were size our measure- run on Systems, makes minimal as on original were used. a six so that running. for 2 each there use graphs of the the three is negligible (Speedup is of FastThreads Figure of processors 1/0 relative to algorithm). systems added perform overhead application. either system memory applications because threads portions exert as a buffer the access. took to explicitly a disk when has other processor, implementation, ACM if versus implementation one represent- by would bound. cause as quickly than speedup there 1/0 block only that it runs faster application’s systems be they log N) Firefly. services, much to exerted that simply than a tree traverses speed or physical point less force cache we then run- an O(N cluster. problem floating intended CVAX First, and normally of magnitude force us to control the when by the memory Firefly’s in The of processor of its enough our body. mea- original FastThreads was constructs and we threads, modified algorithm compute allowed miss would the orders and in similar because kernel This that misses ments a part kernel we measured of space ratio either a small qualitatively are be to manage threads The of the relative can always tion, of mass the application; . be approximated center and application portion performance, Topaz threads, The of each on application’s of Topaz problem application the top using N-body at the application system of mass can Depending our application activations. to compute distant of parallel This user-level Vol 10, worse overhead thread No than of creating 1, February system. 1992. and is the sequential synchronizing greater for Topaz a Scheduler Activations: Effective Kernel Support 5- -0- l“opaz + orig + new o 73 threads FastThrds Fastl’hrdS 4- 3- 2- 1 o~ 1 2 number Fig.2. Speedup of N-Body As processors tially improves release an kernel, provided acquire a busy greater both user-level bottleneck during our tests, wake up lock is the of original processors. the Topaz periodically, that Thus, so that fora kernel though to and be kernel lock has good enough paral- performance. when are are using less ifthe less When the application tries (in this case, six processors), thread systems is similar. impact on the duration). Next, because performance of original and our system for a short system diverges applications has several time, Transactions if slightly were running daemon and then threads go back to spaces, proceswhich Fast- to use all of the processors of the the number of preemptions for both (The preemptions have only a small FastThreads we show that when the application requires it does 1/0, our system performs better than ACM is crucial Because Threads. machine user-level ofa lock our system explicitly allocates processors to address these daemon threads cause preemptions only when there are no idle sors available; this is not true with the native Topaz scheduler, controls the kernel threads used as virtual processors by original sleep. is attainedby level optimizations the overhead sections atuser no other operating execute to threads. FastThreads Even time iniand tries application critical threads acquire trapping speedup prevents available. a thread the Topaz ofthe short user-level in application its [121; these If good that performance can without block the kernel lock. The threads it byspinning with will released. 100%ofmemory Topaz the of contention. shows 6 a thread section for thread ofkernel to the with Topaz, critical the restructuring is built which a ofprocessors, In contention systems trapping or five on no to improve by performance four out. the overhead application performance presence or perhaps before The the thread threads the however, when in number flattens is 5 processors of versus lock lock, be able kernel with then there itisthe Wemight the added, only much busy are application rescheduled lelism; application and 4 3 on Computer Systems, because of their short kernel involvement either original FastVol. 10, No. 1, February 1992. 74 T. E. Anderson et al. . 1oo0 Topaz + orig + new 80- threads Fast Thrds Fast Thrds 60- 40- c o 20- o~ 100% 90% 80% Fig. 3. Execution time of N-Body 60% 70% % available 40% 50% memory application versus amount of available memory, 6 processors, Threads or Topaz threads. Figure 3 graphs the application’s execution on six processors as a function of the amount of available memory. For all three systems, performance degrades slowly at first, and then sharply once the application’s application performance with than with the other two working original systems. time more set does not fit in memory. However, FastThreads degrades more quickly This is because when a user-level thread blocks blocks, in the kernel, the kernel and thus the application of the parallel 1/0. The curves for modified FastThreads and for Topaz threads each other because both systems are able to exploit the parallelism of the application to overlap thread serving as its virtual processor also loses that physical processor for the duration some of the 1/0 latency with As in Figure 2, though, application performance is fied FastThreads than with Topaz because most thread useful computation. better with operations modican be implemented without kernel involvement. Finally, while Figure 3 shows the effect on performance of applicationinduced kernel events, multiprogramming causes system-induced kernel events that result in our system having better performance than either original N-body averaged system; Table FastThreads or Topaz threads. To test this, we ran two copies of the application at the same time on a six processor Firefly and then their execution times. Table V lists the resulting speedups for each note that a speedup of three would be the maximum possible. V shows that application performance with modified FastThreads is good even in a multiprogrammed that obtained when the application environment; the speedup is within 5% of ran uniprogrammed on three processors. This small degradation is about what and the need to donate a processor thread. In contrast, multiprogrammed either When ACM original FastThreads applications using Transactions on Computer we would expect from bus contention periodically to run a kernel daemon performance is much worse with or Topaz threads, although for different reasons. original FastThreads are multiprogrammed, the Systems, Vol. 10, No. 1, February 1992 Scheduler Activations: Effective Kernel Support Table V. Speedup of N-Body Application, Multiprogramming 100% of Memory Topaz threads 1.29 Level . 75 = 2, 6 Processors, Available Original FastThreads 1.26 New FastThreads 2.4.5 operating system time-slices the kernel threads serving as virtual processors; this can result in physical processors idling waiting for a lock to be released while the lock holder threads than with our expensive. In addition, tion, it may than from for Topaz is rescheduled. system because because Topaz end up scheduling more Performance is worse with Topaz common thread operations are more does not do explicit processor allocakernel the other; Figure 2 shows, threads when more than threads from one address space however, that performance flattens three processors are assigned to out the application. While the Firefly is an excellent vehicle for constructing proof-of-concept prototypes, its limited number of processors makes it less than ideal for experimenting with significantly parallel applications or with multiple, mul tiprogrammed parallel applications. For this reason, we are implementing scheduler activations in C Threads and Mach; we are also porting Amber , a programming system for a network of multiprocessors, onto our Firefly implementation. 6. RELATED The two IDEAS systems with goals most closely related to our own—achieving properly integrated user-level threads through improved kernel support —are Psyche  and Symunix . Both have support for NUMA multiprocessors as a primary goal: Symunix in a high-performance parallel UNIX implementation, and Psyche in the context of a new operating system. Psyche and Symunix provide “virtual processors” as described in Sections 1 and rupts are same 2, and that like augment notify upcalls, stack the these user except and thus virtual level that processors of some kernel all, interrupts are not reentrant). of multimodal parallel programming kinds, in different address spaces, by defining events. on the Psyche software (Software same processor has also explored in which user-defined can synchronize while inter- interrupts threads sharing use the the notion of various code and data. While Psyche, Symunix, and our own work share similar goals, the approaches taken to achieve these goals differ in several important ways. Unlike our work, neither Psyche nor Symunix provides the exact functionality of kernel threads with respect to 1/0, page faults and multiprogramming; further, the performance of their user-level thread operations can be compromised, We discussed some of the reasons for this in Section 2: these systems events that affect the notify the user level of some but not all of the kernel ACM Transactions on Computer Systems, Vol. 10, No 1, February 1992. 76 T E. Anderson et al. . address space. For example, neither when a preempted virtual processor thread system threads Both are running Psyche and does not know on those Symunix Psyche nor Symunix notify is rescheduled. As a result, how many processors processors. provide shared it writable the user level the user-level has or memory what user between the kernel and each application, but neither system provides an efficient mechanism for the user-level thread system to notify the kernel when its processor allocation needs to be reconsidered. The number of processors needed by each application could be written into this shared memory, but that would give no efficient way for an application that needs more other application has idle processors. Applications in both Psyche and Symunix the kernel in order to avoid preemption processors share to know that synchronization at inopportune state moments (e.g., some with while spin-locks are being held). In Symunix, the application sets and later clears a variable shared with the kernel to indicate that it is in a critical section; in Psyche, the application checks for an imminent preemption before starting a critical latency, section. which performance The setting, clearing, and checking of these bits adds to lock constitutes a large portion of the overhead when doing high- user-level thread latency management unless . By contrast, a preemption actually our occurs. system has no effect on lock Furthermore, in these preempt other systems the kernel notifies the application of its intention to before the preemption actually occurs; based on this a processor notification, the application can choose to place a thread in a “safe” state and voluntarily relinquish a processor. This mechanism violates the constraint that higher priority threads are always run in place of lower priority threads. Gupta et al. [9a] share our goal of maintaining a one-to-one correspondence between physical processors and execution contexts for running user-level threads. being texts When more until’ context. Our a processor contexts than the application kernel preemption eliminates the need application thread system of the event constant. Some systems provide asynchronous some of the problems with or 1/0 processors, Gupta reaches a point user-level for while kernel thread completion results in there et al.’s kernel time-slices conwhere it is safe to suspend a time-slicing by notifying keeping the number 1/0 as a mechanism management the of contexts to solve on multiprocessors [9, 251. Indeed, our work has the flavor of an asynchronous 1/0 system: when an 1/0 request is made, the processor is returned to the application, and later, when the 1/0 completes, the application is notified. There are two major differences between our work and traditional asynchronous 1/0 systems, though. First, and most important, scheduler activations provide a single uniform mechanism to address the problems of processor preemption, 1/0, and page faults. Relative to asynchronous 1/0, our approach derives conceptual simplicity from the fact that all interaction with the kernel is synchronous from the perspective of a single scheduler activation. A scheduler activation that blocks in the kernel is replaced with a new scheduler activation when the awaited event occurs. Second, while asynchronous 1/0 schemes may require significant changes to both application and kernel code, ACM Transactions on Computer Systems, Vol. 10, No 1, February 1992 Scheduler Activations: Effective Kernel Support . 77 our scheme leaves the structure of both the user-level thread system and the kernel largely unchanged. Finally, parts of our scheme are related in some ways to Hydra , one of the earliest multiprocessor moved of the out performance the kernel could policy to a scheduling without in decisions policy scheduling this and policy separation required server, to the kernel. are the kernel’s be delegated in which Hydra, communication then back was came at Longer-term responsibility, to the to a distinguished processor although application-level a through kernel to switch. In our system, an application can set its its threads onto its processors, and can implement trapping sions in our system systems, However, cost because implement a context policy for scheduling policy operating kernel. own this allocation deci- as in Hydra, this server. 7. SUMMARY Managing parallelism parallel at computing, operating described but the user kernel level threads is essential to or processes, high-performance as provided in systems, are a poor abstraction on which to support this. the design, implementation and performance of a kernel many We have interface and a user-level thread package that together combine the performance user-level threads (in the common case of thread operations that can implemented (correct entirely behavior at user in the level) with infrequent the functionality case when Our approach is based multiprocessor a virtual on providing in which each application the application many processors and which those processors. application it address –Processor has exactly Responsibilities are of its divided of kernel the kernel must threads be involved). address space knows exactly threads between are the of be kernel with how running on and each space: allocation (the allocation of processors to address spaces) is done threads to its of every event by the kernel. —Thread scheduling (the assignment processors) is done by each address —The kernel affecting –The address can affect The notifies the address the space address thread space’s scheduler space. space notifies processor address of an space. the kernel allocation of the subset of user-level events that decisions. mechanism that we use to implement these ideas is called A scheduler activation is the execution context for vectoring control from the kernel to the address space on a kernel event. The address space thread scheduler uses this context to handle the event, e.g., to modify user-level thread data structures, to execute user-level threads, and to make requests of the kernel. While our prototype implements threads as the concurrency abstraction supported at the user level, scheduler activations are not linked to any particular model; scheduler activations can support any user-level concurrency model because the kernel has no knowledge of userlevel data structures. scheduler kernel activations. ACM Transactions on Computer Systems, Vol. 10, No. 1, February 1992. 78 T. E. Anderson et al. . ACKNOWLEDGMENTS We would like to thank Andrew Black, Mike Burrows, Jan Edler, Mike Jones, Butler Lampson, Tom LeBlanc, Kai Li, Brian Marsh, Sape Mullender, Dave Redell, Michael Scott, Garret Swart, and John Zahorjan for their helpful comments. Center for providing We would also like us with their to thank Firefly the DEC hardware Systems Research and software. REFERENCES Actors: A Model of Concurrent Computation in Distributed Systems. MIT Press, 1. AGHA, G. Cmabridge, Mass. 1986. 2, ANDERSON, T , LAZOWSKA, E., AND LEVY, H. The performance implications of thread manIEEE Trans. Comput. 38, 12 (Dee. agement alternatives for shared memory multiprocessors. 1989), 1631-1644. Performance Calif., Also appeared ’89 Conference May 1989), 446-449. 4. BIRRELL, A., GUTTAG, multiprocessor: Operating A D. system. A hierarchical J., Principles Comput. Msg. programming Symposwm D. Systems J., LIPKIS, systems. 1988), Mellon In E., LEVY) The process R, Program. M. ceedings 2nd June algorithm. the 11th Nature 324 primitives ACM for Symposium a cm parallelism in the Mach operating H., AND LITTLEFIELD, In Park, ACM. Rep. R. The Proceedings Ariz., 31, Dee (Mar. 3 Amber 1989), ACM pp. 147-158. 1988), CMU-CS-88-154, system: of the 12th 314-333. School of Computer 1988. Process USENIX July management Workshop on for highly UNIX Systems and parallel UNIX Supereomputers (Sept. for 1985), Wash., concurrent Mar S.-P., AND GLIGOR, V. 1987), pp 15, MARSH, B., Proceedings Oct. 1990), pp. In Systems ACM Trans. and structures. Practice In Pro- of Parallel 197-206. S. Empirical Calif., with A comparative data Principles Oct. processes analysis Conference studies of the 1991), of competitive 13th ACM spinning Symposium on pp. 41-55. and monitors of multiprocessor on Distributed in Mesa scheduling Comm un. algorithms. Computing Systems (Sept. 356-363. SCOTT, M., of the 13th 1991), ACM T , AND MAR~ATOS, Symposium P., AND STAUNSTRUP, Parallel on LEBLANC, E. on Operating First-class Systems user-level Prlnclples threads. (Pacific In Grove, pp.110-121. Cornput. J., AND KOHLER, Transactions computation. concurrent on Proceedings Grove, of the 7th International 16. MOELLER-NIELSEN, algorithms. highly M., AND OWrC~I, (Pacific symbolic Symposium multiprocessor Principles multipro- Computer 501-538. SIGPLAN MANASSE, use of shared-memory CSL-TR-91-475A, 1991. LAMPSON, B., AND REDELL, D. Experiences ACM. 23, 2 (Feb. 1980), 104-117. Proceedings effective Rep. for implementing ACM a shared-memory Making Tech. A language 7, 4 (Oct. (Seattle, A., LI, K., Operating ACM Univ., Syst. the Programming 17. Moss, and (Oakland, pp. 94-102. and Tech. approach A methodology of 12. KARLIN, control Multilisp: Lang. 11. HERLIHY, Calif, 1987), Commun J,, AND SCHONBERG, E Stanford 10. HAI,STEAD, In Systems Synchronization of (Litchtield system. of the R. of multiprocessors. C Threads. Umv., Proceedings Laboratory, 14. Lo, SIGME’Z’RICS Pp. 1-17. cessors: 13 ACM 1990), 35-43. 9A. GUPTA, A , TUCKER, A., AND STEVENS, L for Nov. concurrency Principles The V distributed Carnegie 9. EDLER, 1989 of Computer Proceedings Tex., for 23, 5 (May 8. DRAVES, R., AND COOPER, E. Science, the Modeling AND LEVIN, In on a network on Operating 7. CHERITON, J., (Austin, support 6. CHASE, J., AMADOFt, F., LAZOWSIiA, Parallel of and O(N log N) force-calculation specification. Scheduling IEEE HORNING, formal Systems 5. BLACK, Proceedings pp. 49-60. 3, BARNES, J., AND HUT, P. (1986), in on Measurement Computer W. 4, 1 (Feb. Concurrency Systems, Vol. J. Problem-heap: 1987), features 10, No. A paradigm for multiprocessor 63-74. for the Trellis/Owl 1, February 1992 language. In Proceed- Scheduler Activations: Effective Kernel Support . 79 ings of European Conference on Object-Oriented Programming 1987 (ECOOP 87) (June 1987), pp. 171-180. Proceedings of the ACM 18. REDELL, D. Experience with Topaz teledebugging. In SIGPLAN/SIGOPS Workshop on Parallel and Distributed Debugging (Madison, Wise., May 1988), pp. 35-44. Performance of Firefly RPC. ACM Trans. Cwnput. SW 19. SCHROEDER,M., AND BURROWS, M. 8, 1 (Feb. 1990), 1-17. 20. SCOTT, M., LEBLANC, T., AND MARSH, B. Multi-model parallel programming in Psyche. In Proceedings of the 2nd 21. TEVANIAN, Threads ACM SIGPLAN Symposium on Principles and BLACK, COOPER, E., AND Practice of Parallel YOUNG, M. (Mar. 1990), pp. 70-78. Programming A., and RASHID, R., Unix Kernel: the GOLUB, The D., battle for D., control. In Proceedings of the 1987 Mach USENIX Summer Conference (1987), pp. 185-197. A multiprocessor worksta22. THACKER, C., STEWART, L., AND SATTERTHWAITE, JR., E. Firefly: tion. IEEE Trans. Comput. 37, 8 (Aug. 1988), 909-920. 23. TUCKER, A., AND GUPTA, A. Process control and scheduling issues for multiprogrammed of the 12th ACM Sympo.mum on Operating shared memory multiprocessors. In Proceedings Systems Principles (Litchfield Park, Ariz., Dec. 1989), pp. 159-166. 24. VANDEVOORDE, M., AND ROBERTS, E. WorkCrews: An abstraction for controlling paralProgram. 17, 4 (Aug. 1988), 347-366. lelism. Int. J. Parallel common runtime approach to 25. WEISER, M., DEMERS, A., AND HAUSER, C. The portable interoperability. In Proceedings of the 12th ACM Symposium on Operating Systems Principles (Litchfield 26. WULF, W., Park, LEVIN, tem. McGraw-Hillj 27. Dec. New York, ZAHORJAN, J., AND MCCANN, Proceedings Computer 28. Ariz., ZAHORJAN, overhead 1991), Received of the 1990 Systems J., 1989), R., AND HARBISON, C. ACM LAZOWSKA, Hydra/ C.mmp: An Experimental Computer Sys- 1981. (Boulder, in shared pp. 114-122. S. Processor Colo., E., memory scheduling SIGMETRICS May 1990), AND EAGER, in shared Conference D. multiprocessors. memory multiprocessors. on Measurement and Modeling In of pp. 214-225. The IEEE effect of scheduling Trans. Parallel Distrib. discipline on spin Syst. 2, 2 (Apr. 180-198. June 1991; revised August ACM 1991; accepted Transactions September on Computer 1991 Systems, Vol. 10, No. 1, February 1992.
* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project