Design of the TruCluster Multicomputer System for the Digital UNIX Environment. Digital Common Directory Interface, OM, Version 2, Atom, TruCiuster, Integrated Directory Services
विज्ञापन
विज्ञापन
I
Wayne
M.
Cardoza
FrederickS. Glover
William E. Snaman, Jr.
Design of the TruCiuster
Multicomputer System for the Digital UNIX
Environment
The TruCiuster product from Dig ital provides an available and scalable multicomputer sys tem for the UNIX environment. Although it was designed for general-purpose computing, the first i mplementation is directed at the needs of large database applications. Services such as distri buted locking, fai lover ma nagement, and remote storage access a re layered on a high-speed cl uster i nterconnect. The initial implementation uses the M E MORY CHANNEL, an extremely relia ble, hig h-performance i nter connect specially designed by Digital for the cl uster system .
The primary goal for the first release of the TruClustcr system for the Digital UNIX ope r a ting svsn.:m was to develop J high-pertorm;�nce commen:ial database server environment running on a cluster of several nodes. Database applications oti:en require comput in g power and I/0 connectivity and bandwidth greater than that provided b�· most single systems. In addi tion, Jvaibbilitv is a kev require me nt tor enterp r ises that are dependent on datJbase services for no rm a l operations. These req uirements led us to implement a cluster of computers tllJt cooperate to provide services but fail independently. Thus, both performance and
Jvailability are addressed.
We chose an industry-st:mdard benchm;�rk to gauge our success in meeting pert(>rmance g o als
.
The
Transaction Processing Performance Council TPC-C benchmark is a widely accepted measurement of the capabi l it:v of large servers. Our goal was to achieve industry-leading numbers in excess of 30,000 t r a n sac tions per minute (tp m C) with a four-node TruCluster system.
The TruCiuster vers i on 1.0 product provides reliable, shared access to large amounts of storage, distributed synchr o niz
�u
i on k>r applications, efticient cluster communication, and application f:1ilover. The focus on database servers docs not mean that the
TruCJuster system is not suitable for other applica tions, but that the inC\·itJble design decisions and trade-offs tor tbe first product were made with this goal in mind .
Although other aspects of providing a singl e -syste m view of'' cluster are important, they are secondary objectives and will be phased into the product over time.
This paper begins wirh a brief comparison of com puter systems and presents the advantages of clustered computing. Next, it introduces the TruCiuster prod uct and describes the design of its key software compo nents and their relationship to database applications.
The paper then discusses the design of the MEMORY
CHANNEL interconnect for cluster systems, along with the design of the low-level software toundation for cluster synchronization and communication.
Finally, it addresses application tailover and co n f
i
g u r at i ons
. hardware
Digital
Tcdmic�l journal
Vol. 8 No.
I 1996
5
Brief Comparison of Computing Systems
Contemporary computing systems evolved ti·om centralized, single-node time-sharing systems imo sev eral distinct styles of multinode computer systems.
Single-node systems provided uniform accessibility to resources and services and a single-management donuin. They were limited with respect to scalability, however, and system tailures usuaUy resulted in a com plete loss of service to clients of the system.
Multinode computer systems include symmetric multiprocessing (SMP) systems and massively parallel processors (MPPs). They also include network-based computing systems such as the Open Software
Foundation Distributed Computing Environment
(OSF DCE), Sun Microsystems Inc.'s Open Network
Computing
(ONC), and workstation brrnsu Each of these systems addresses one or more of the bendits associated with clustered computing.
SMP configurcltions provide for tightly coupled, high-performance resource sharing. In their effective range, SMP S)'Stems provide the highest-performance single-system product for shared-resource applicJ tions. Outside that range, however, both hc1rdware and software costs increase rapidly as more processors are added to an SMP system. In addition, SMP avail ability characteristics are more closely associated with those ofsingle systems because an SMP svstem, by dd� inition, is composed of multiple processors but not multiple memories or
I/0 subsystems.
MPP systems such as the Intel Paragon series were developed to support complex, high-performance parallel applications using systems designed with hun dreds of processors. The indi,·idual processors of an
M
PP system were typically assigned to specific t:tsks, resulting in fairly special-purpose machines.
The DCE and ONC technologies provide support for common naming and :tccess capabilities, user account management, authentication, and the replica tion of certain services tor improved availability.
Workstation farms such as the Warson Research Cen tral Computer Cluster deliver support for the parallel execution of applications within multiple computer environments typicallv constructed using offthe-shelf software and hardware.' ONC, DCE, and f:mns pro vide their services and tools in support of heteroge neous, multivendor computing environments with hundreds of nodes. They are, however, much further away from realizing the benefits of a single-system view associated with clustered computing.
In the continuum of multi node computer systems, the advantage of the cluster system is its �1bility to provide the single-svstcm ,·iew and ease of manage ment associated with SMP systems and at the same time supply the E1ilure isolation and sca!Jbility of dis tributed systems.
Cluster systems have clear advantages over large scale parallel systems on one side and heterogeneous distributed systems on the other side. Cluster systems provide many cost and availability advantages over large parallel systems. They are built of standard build ing blocks with no unusual packaging or interconnect requirernenrs. Their I/0 bandwidth and storage con nectivity scJic well with standard components. They are inherently more tolerant of Eli lures due to looser coupling. Parallel or multiprocessor systems should be thought of as cluster components, nor as cluster replacements.
Cluster systems have a different set of advantages over distributed systems. First they are homogeneous in nature and more limited in size. Cluster systems can be more efficient when operating in more constrained environments. Data formats arc known; there is a single-security dom:1in; failure detection is certain; and topologies are constrained. Cluster systems also are likely to have interconnect pertorm:1ncc advantages.
Protocols are more specialized; interconnect charac teristics are more uniform; and high performance can be guaranteed. finally, the vendor-specific nature of cluster systems Jllows them to evolve Elster than het erogeneous distributed systems cllld will probably always allow them to have advJntJges.
There ln.: numerous examples of general-purpose clusters supplied by most computer vendors, including
AT&T, Digital, Hewlett-Packard, Imcrnational Busi ness 1\tbchines Corporation, Sequent Computer Sys tems, Sun Microsysrems, and Tandem Computers.
DigitJJ's Open VMS cluster system is generally accepted as the most complete cluster product oftering in the industrv, and it lchieves many of the single-svsrem m:magemcnt Jttributes' Much of the functionality of the OpenVMS cluster system is retained in Digital's
TruCiuster product offerings.
Structure of the TruCiuster System
Digital's TruCluster multicomputer system is :1 highly avJiL1ble and scal<lble structure of
UNIX servers that preserves mJny of the benefits of a centralized, single computer system. The TruCluster product is a collec tion of looselv coupled, general-purpose computer systems connected by a high-perfornlJnce intercon nect. It maintains J single security domain Jnd is ma.n
<lged as a single system. Each cluster node may be a uniprocessor or a multiprocessor system executing the Digital UNIX operating system. figure l shows a typical cluster configuration.
Each cluster member is isolated fi·om sofrware and hardware bults occurring on other cluster members.
Thus, the TruCluster system does not hcwe the tightly coupled, "bil together" characteristics of multiproces sor systems. Cluster services remain availab.le even
"vben individual cluster members are temporarily
6
Di�;iral Technical )ollrnal Vol. 8
No. I 1096
DISKS
Figure 1
Contiguration of a four node Cluster System unavailable . Other important availability objectives of the TruCluster server include quick detection of com ponent and member failures, on-line reconfigurarions to accommodate the loss of a fai led component, and cont inued service while sate operation is possible.
The TruCiusrer prod uct supports large, highly available database systems through several of its key components. First, the distri buted remote disk ( DRD ) facility provides reliable, transparent remote access to all cluster storage from any cluster node. Next, the d is tributed lock manager
( D LM )
enables the elements of a distributed database system to synchronize activity on i ndependent cluster nodes. Finally, elements of
Digi tal's D ECsafe Available Server Environment
( AS E )
provide application tailover.5 In support of all controls cluster membership and the transition of nodes in and out of the cl uster. Figure 2 is a block dia gram showing the relationships between components.
Each major component is described in the remain der of this paper. I n addition, we describe the high performance
M E M O RY
CHANNEL interconnect that was designed specifically tor the needs of cluster systems.
Distributed Remote Disk Subsystem
The distributed remote disk
( DRD) su bsystem was developed to support database appUcations by present ing a cl usterwide view of disks accessed through the character or raw device i nterface. The Oracle Parallel
Server
( OPS ) , which is a parallelized version of the
Oracle database technology, uses the DRD subsystem .
The DRD su bsystem provides a cl usterwide name space and access mechanism tor both physical and loa ical (logical storage manager or
LS M )
� olumes. Tl
� e
LSM logical device may be a concatenated, a striped , or a mirrored volume. D RD devices arc Jccessible ti-om any cluster member using the DRD device name.
This location independence allows databJse software to treat storage as a uniformly accessible duster resource and to easily load balance or tai l over activity between cluster nodes.
Cluster Storage Background through the U N I X tile system and an associ
� ted block device special til e . A disk device mav also be accessed through a character device special t ile or r:1w device that provides a direct, unstructured interface to the device and bypasses the block buffer cache.
Database mJnagement systems :1nd some other high-performance U N I X applications Jre often designed to tJke advantage of the character device spe cial file interbces to improve pertorm:1nce by avoiding additional code path length associated with the tile svs tem cache -'' ' The
I/0 profil e of these systems is ch acterized by large files, random acce
�
� r s to records, private datJ caches, and concurrent read -write sharing.
Overall Design of the DRD
The DRD su bsystem consists of four primary compo nents. The remote raw disk ( RRD ) pseudo-d river red i rects DIU) Jccess requests to the cluster member serving the storage device. The server is identified by information maintained in the DRD device data base
( RRDB
)
. Req uests to Jccess local
DRD devices are passed through to local device drivers. The block ship ping client ( BSC) sends requests for :1cccss to remote
DRD devices to the appropriate DIU) server and returns responses to the cal ler. The block shipping server (BSS) accepts requests ti·om
BSC clients, passes them to i ts locJI driver tor service, :1nd returns the results to the calling ESC client. Figure 3 shows the components of the DRD subsystem.
The D RD management component supports D RD device naming, device creation a n d deletion, device relocation, and device status requests. During the
DRD device creation process, the special device file designating the
D RD device is created on each cluster mem ber. In Jddi tion, the DRD device num ber, its cor responding physicJl device number, the network address of the serving duster member, and other con figuration parameters are passed to the DRD driver, which upd:1tes its local database Jnd communicates the information to other cluster members. The D RD driver may b e queried for device status and D RD data base information .
Clusterwide Disk Access Model
D uring the design of the
DRD ered both shared (multiported) and s
� rved
� iisk models.
A multiported d isk configuration provides good fai lure recovery and load balancing characteristics. On the
Digital Technical }ournal
Vol . 8 No. 1
1 996
7
8 r - - - - - - - - - - - - - - - - - -
I
NODE A
I
I r---
I
DATABASE INSTANCE t
I I
.....--:-t ----.
LOCK
I
MANAGER
INTERFACE r-----------t---+--+·
STANDBY
CONFIGURATION
MANAGER
�����
AVAILABI LITY
MANAGER
I
RAW DISK
INTERFACE
ER
I
I t'-------t----11--
1
I
COM
� l
I
I
I
I
I
II
I
I
I
I
I
, - - - - - - - - - - - - - - 1
I
NODE B
I
DATABASE INSTANCE
I
I
I
I
I
1 0 1
I
II
I
I
LOCK
MANAGER
I NTERFACE
DECSAFE ASE AVAILABILITY SERVICES
DISTRI BUTED LOCK MANAGER
I
DISTRIBUTED DIRECTORY SERVICE
I
I
RESOURCE MANAGER
I I
RESOURCE MANAGER
I
--
-+-1
!
DISTRIB UTED REMOTE DEVICE
I
'
.
RAW DISK
INTERFACE t
ACTIVE
I
CONFIGU RATION
MANAGER
I
..J
CONNECTION
MANAGER
AGENT l
AVAILABILITY
MANAGER
I
I
I + I
I
I
I
I
I
I
I
LOCAC
I
ORO
1/0
REDI RECTION
'"'cc cmvms t
I I
ORO BLOCK
SHIPPING
SERVER
ORO BLOCK
SHI PPING
CLIENT
MCMORV CHANNeL
SERVICES
1
I
I
I
I
:
DATA
MOVER
I I
:
I
I
ORO BLOCK
SHIPPING
CLI E NT
I
I
ORO BLOCK
SHI PPING
SERVER r
MCMORV CHANNEL
SERVICES
DAD
1/0
REDI RECTION
J
I
I I
LOCAL ""cE CR'"Rs
- - - l - - - - - - - ��-�" ;;;;���,"�� - - - - - -r - - -
J
I
I
II
MEMORY CHANNEL BUS ll
NODE C NODE 0
I
Figure 2
Software Componen ts
I
I
I
I
I
,
- - - - - - - - - - - -
ORO CLI ENT
- ,
I
I
I
I
I
I
DEVIC E DRIVER
I
L - - - - - - - - - - - - - - -
L - - - - - - - - - - -
Figure 3
D istri buted Remote D isk S u bsystem
Vol. 8 No. 1 1 996
other hand, !jO bus contention and hardware q ueuing delays from fu lly connected, sh:1red disk configurations can limit scalability. In addition, present standard I/0 bus technologies l imit configuration dist:mces.s As 3 consequence, we selected a served disk model for the
DIU) implementation. With this model, software queuing ;1lleviates the bus contention ;md bus queuing delays. This approJch provides improved scalabiljty Jnd fau l t isolation ;JS well as flexible storage configura tions 9·10 Full connectivity is not required, and extended machine room cluster configurations can be con structed usi ng standard networks ;1nd !jO buses.
The DlU) implementation supports cl usterwide access to DRD devices using a software- based emula tion of a fully con nected disk configuration . Each device is assigned to a single cluster mem ber at a time .
The member registers the device into the cluster wide namespacc and serves the device data to other cl uster members. �Jilure recoverv and load-balancing support are included with the DlU) device i mpJemcn tJtion. The bilure of a node or con trol ler is transpar ently masked when another node connected to the shared bus tJkcs over serving the d isk. As an option, au tomatic lo3d bJiancing can move service of the d isk ro the node generating the most requests.
In the Tru Ci ustcr version 1 .0 prod uct, data is transferred between req uesting and serving cl uster mem bers using the high - bandwidth, low-latency
MEMORY C H A N N E L i nterconnect, which a lso sup ports direct me mory access ( D MA) between the !jO ad3pter of the serving node and the m;lin memory of the req uesting node. The overall cluster design, how ever, is not dependent on the MEtVIO R.Y CHAN NEL intercon nect, :md alternative cl uster i nterconnects will be supported in future software rcle:�scs.
DRD Naming
The DigitJI U N I X operatjng system presently supports character device special file names f(>r both physical disk devices and LSM logical volumes and maintains a sepa rate device namespace for each. An important D lU) design objective was to develop a cl usterwide naming scheme integrating the physical and logical devices within the Dlill mmespace. We considered defining
J new, single mmespace to support all cluster disk devices. Our research, however, revealed plans to intro duce significant changes into the physic:1l device nam ing scheme i n a fu ture base system release and the compl ications of l icensing the logical disk technology ti-om a third parrv that maintains control over the logi cal volume namespace. These issues resulted i n deter ring a true cl usrerwide device namespace .
As an interim approach , we chose ro create a sepa rate, clusterwide DRD device namespace layered on the existing physical and logical device naming schemes. TrJnslations from D IU) device names i nto the underlying physical and logical devices are main tained by the DRD device mapping database on each cluster node. DRD device "services" are created by the cluster administrator using the service registL1tion facil ity. 1 1 E:1ch "add Service'' mamgement oper3tion generates a unique service number that is used in constructing the Dlill device special file name.
This operation also creates the new D RD device special ti le on each d uster member. A trad itional
UNIX-device-naming convention results i n the ere arion of D RD special device fi le n;Jmes in the form of
/dev/drd/d rd (service number f . ' 2
DRD Relocation and Failover
ASE fai lover (see the discussion in the section
Application Fai lover) is used to support DRD bilover and is fu l l y integrated within the cl uster prod uct. The device relocation policv defined d u ring the creation of a D RD device indicates whether the device may be reassigned to 3norher cluster member as a result of a node or controller fail ure or a load - balancing opera tio n . In the even t of a cl uster member fail u re, DRD devices exported by the failed member arc reassigned to an alternate server attached to the same shared ljO bus. During reassignment, the DRD device databases are updated on all cl uster mem bers and DRD ljO operations Jre resumed. Cl uster device services may also be reassigned d u ring a pl:mned relocation, such as for load babncing or mem ber removal. Any D RD operation in progress during a re location triggered by a fai l u re will be retried based upon rhc registered D RD retry pol icy. The retry mechanism must revalidate rhe database trJnslation map for the target D RD device because the server binding may have been modified .
Failover is thus transparent to database applications and allows them to ignore configu ration changes.
Several challenges result ti·om the support of m u ltiported disk configurations under various fai l u re scenarios. One of the more difficult problems is distin guishing a fjilcd member from a busy member or a communication bult. The ASE failover mechanism was designed to maintain data i nregriry d uring service tai lover, and to ensure that subsequent disk oper;Jtions are not honored ti-om a member that has been declared
"down " by rhe remai ning cluster members. This ASE mechanism, which makes usc of small computer sys tems interface (SCSI) target mode and device reserva tion, was integrated into the Tru Ci uster version 1 .0 product and supports the DRD service guarantees.
Other challenges relate to preserving seriali zation guarantees in the case of cluster member fai lure.
Consider a parallel application that uses locks to serial ize access to shared DRD devices. Suppose the applica tion is holding a write lock for 3 given data block and
Digir31 T<:chnical Journal Vol. 8 No. J 1996
9
issues an update for that block. Before the update operation is acknowledged, however, the local mem ber fai ls. The distri buted lock manager, which wi l l have been notified o f the member failure, then rakes action to release the lock. A second cooperating appli cation execu ting on another cluster member now acquires the write Joc k for that same data block and issues an update tor that block. If the fail ure had not occurred, the second appl ication would have had to wait to acquire a write lock for the data block unt i l the first application released the lock, presu mably after its write request h ad completed. This same serialization must be mai ntained during failure cond itions. Thus, it is imperative that the write issued by the first (now fai led) application partner not be applied after the write issued by the second application, even in the presence of a timing or net\vork retransmission Jnom aly that delays this first write.
To avoid the reordering scenario just described, we employed a solution cal led a sequence barrier in which the connection manager increments a sequence number each rime it completes a recovery transition rlut results in rcle:1sed locks. The seq uence number is comm u nicated to each D RD server, which uses the sequence n u mber as a barrier to prevent apply ing stale writes. This is similar to the im mediate com mand feature of the Mass Storage Control Protocol
( MSCP) used by Open VMS cluster systems to provide similar guarantees. Note that no application changes arc required.
As another example, client retransmissions of
DRD protocol requ ests that are nor id empotent em cause serious consistency problems. Request transac tion IDs and DRD server dupl icate transaction caches are employed to :woid undesirable efkcrs of client generated retransmissions . " '
Cluster member fa ilu res are mostly transparent to applications executing on client member systems.
Nondistributed applications may fail, but they can be
Jutomatiea l ly restarted by ASE facilities. DlU) devices exported by a serving member become unavailable for
�1 small amou nt of rime when the member tails. C luster f::�ilover activities that must occur before the DRD service is again available include detecting and veri�' i ng the member fai lure, purging the disk device SCSI hardware reservation, assigning an alrern::�re server, establishing the new reservation, and bringing the device back on - l i ne . A database application serving data from the DRD device at the time of the fai l ure may also have registered to have a restart script with a recovery phase executed prior to the restart of the database appl ication. A possible lack of transp:�rency may result if some client applications are not designed to accommodate this period of inaccessible DRD ser vice. The DRD retry request poli cy is configurable to accommodate applications interacting directly with a D RD device.
1 0
Digital Tcdm ical )ourn01 l
Vol . S No. 1 1 996
Distri buted Lock Manager
The distributed lock manager ( DLM ) provides syn chronization services appropriate for a highly parai
J.elized distri bu ted database system . Databases can use locks to control access to distributed copies of data buffers (caches) or to li mit concurrent access to shared disk devices such as those provided bv the DRD sub system. Locks can a lso be used tor controlling applica tion i nstance start-up and for detecting application i nstance fail u res. In addition, appl ications can use the locking services for their other synchronization needs.
Even though this is a completely new implementa tion , the lock manager borrows from the original design and concepts i ntroduced i n 1984 with the
VAXcl uster d istributed lock manager. '' These concepts were used in several recen t lock man:1.ger implementa tions for
UNIX by other vendors. In addition, the
Oracle P::trallel Server uses a locking :1pplication pro gramming i merface (API) that is conceptually similar to that oHcred here.
Usage of the DLM
The lock manager provides an API for request ing, releasing, and altering locks. "·'" These locks are requested on abstract names chosen by the applica tion . The names represent resources and may be orga nized in a hierarchy. When a process requests a lock on a resource, that request is either granted or denied based on examination of Jocks already granted on the resource. Cooperating components of an application use this service to achieve mutually exclusive resource usage . In addition, a mode associated with each lock request al lows tradi tional levels of sharing such as mul tiple readers excluding aU writers .
The API provides optional asynchronous request completion to allow queuing requests or overlapping mu ltiple operations for i ncreased performance.
Queuing prevents retry delays, elimin:�tes pol ling overhead, and provides a first in, first our ( FI FO) fair ness mechanism . In addition, asynchronous requ ests can be used as the basis of a signaling mechanism to detect component failures in a distri buted system . One component acquires an exclusive lock on a n amed resource. Other components queue incomparjble requests with asynchronous completion specified . If the lock holder fails or otherwise releases its Jock, the w:�iring requests are granted . This usage is sometimes referred to as a "dead man" lock.17
A process can request noti fication when a lock it holds is blocking another request. This allows elimina tion of many lock ca.lls by effectively cac hing Jocks.
When resource conten tion is low, a lock is acquired and held u mil :lllother process is blocked by that lock.
Upon receiving blocking noti fication, the lock can be released . When resou rce contention is high , the lock is acq uired and released immediately. I n addition, this
noti fication mechanism can be used as the basis of a ge neral signaling mec hanism . One component of the
3pplication acq uires an exclusive lock on a named resource with blocking noti fication specified. Other components then acquire incompatible locks on that resource, thus triggering the blocking notification.
This usage is known as a "doorbe ll" lock.'7
The D LM is often used to coordi nJte access to resources such as 3 distributed cache of database blocks. M ultiple copies of the data are held under compatible l ocks to permi t re:.1d but not write access.
When
:.1 writer wants an incompatible lock, readers are notified to downgrade their locks and the writer is granted the lock. The writer modifies the data before downgrading i ts lock. The reader's lock requests are again granted, and the reader tetches the latest copy of the data. A value block can also be associated with each resource. I ts value is obtained when a lock is granted and can be changed when cerrain locks are released .
The value block can be used to communicate any use ti.d information, i nduct ing the latest version n u mber of cached data protected by the resource .
Design Goals of the DLM
The overall design goa l of the lock manager was to provide services tor highly scalable database systems.
Thus correctness, robustness, scJ iing, a nd speed were the overrid i ng subgoals of the project.
Cardi.1l attention to design details, rigorous testing, internal consistency checking, and years of experience working with the VMS distri buted lock manager h ave
�11 1 contributed to ensuring the correctness of the implementation for the Digital U NI X system. Because the lock manager provides guarantees about the state of :1l l locks when either a lock holder or the node upon which it is running fails, it em ensure the i nternal lock state is consistent as far as surviving lock holders are concerned. This robustness permits the design of applications that can continue operation when a clus ter node f:1ils or is removed �or scheduled service . The choice of a kernel-based service :.1nd the use of a mes sage protocol also contri bute to robustness as dis cussed below.
I n terms of performance and scaling, the lock man
Jger is designed for minimal overhead to its users. The kernel-based service design provides h igh perfor mance by eliminating the context switch overhead
JssociJted with server daemons. The lock manager uses the kernel -locking features of the Digital
UNIX operating system for good scaling on SMP systems. A kernel- based service as opposed to a li brary also all ows the Jock manager to make strong guarantees about the internal consistency state of locks when a lock-holding process fails.
The message protocol contributes to cl uster scaling and performance through a scaling property that mai ntains a constant cost as nodes are added to the d uster. '< The message protocol also provides suffi ciently loose coupling to allow the lock manager to maintain internal lock state when a node fails. The use of messages controls the amou nt of i nternal state visi ble to other nodes and provides natural checkpoints, which limit the damage resulting ti·om the failure of
J cluster node.
DLM Communication Services
The D LM session service is a commun ication layer that takes advantage of M EMO RY CHANNEL fea tures such as guaranteed ordering, low error rate, and low latency. These features allow the protocol to be very simple with an associated reduction in CPU over head . The service provides connection establishment, delivery and order guarantees, and bufter manage ment. The connection manager uses the communi cation service to establish a c lunnel for the lock manager. The Jock manager uses the communi cation services to com municate between nodes. Because tl1c service h ides the details of the communication mecha n ism, Jlternative intercon nects can be used without changes to the lock manager's core routjnes.
The use of the MEMORY
CHANN EL i n terconnect provides a very low latency com munication path for small messages. This is ideal for the lock manager since lock messages tend to be very small and the users of the lock manager are sensi tive to latency since they wai t for the lock to be granted betore proceeding.
Small messages are sent by simply writing them into the receiving node's memory space. No other com munication setup needs to be performed . Many net work adapters and commu nication protocols are biased toward providing high throughput only when relatively l arge packets are used . This means that the performance drops off as the packet size decreases.
Thus, the MEMORY CHANNEL interconnect pro vides a better alternative for communicating smal l, latency-sensitive packets.
Connection Manager
The con nection manager deti nes an operating envi ronment for the Jock manager. The design allows gen eralization to other clients; b u t in the TruCluster version 1 .0 product, the lock manager is the only con sumer of the connection manager services. The envi ronment hides the details of dynamically changing con tlgurations. From the perspective of the lock man ager, the connection manager manages the addi tion
:tn d removal of nodes and maint ains a communication path between each node. These services a l l owed us to simpli�' the lock manager design.
The connection manager treats each node as a mem ber of a set of cooperating distributed components.
It maintains the consistency of the set by admitting and removing members under controlled conditions.
Dig;ir,ll Technical Journal VoU!
No. l
1996
1 1
The connecrion , .l:mager prm·ides contlgu r:.Hion rclatcd event notification and other support services to each mem ber of a set. It provides notification when members are added and removed. It
�1iso maim a in s
J list of current members. The connection manager J!so provides notification to clients when unsafe operation is possible as a result of partitioning. Partitioning exists when a member of a set is un JwJrc of the existence of a disjoint set ofsimibr clients.
The connection m�w:-�gcr can be extended in clicnt-spccitic wavs to tJcilitatc hJndling of mem bership change n·ents. Extensions are integral, well synchronized parts of the membership change mechanism. The lock mJnagcr uses :1n extension to distribute a globally consistent directory database :111d to coordinate Jock dat:tb:-�se rebuilds.
The connection manager nuintains a fully con nected web of communication clunnels between members of t he set. Jv1embership in the set is comin gcnt upon being
able
to comm u nicJte with J l l other me mbers of that set. The usc of the communication channels is ent irelv under the comrol of the lock m:Jn agcr or
:tny other client that may usc the connection manager in the future. 'When a client requests :-�dmis sion ro a set, the connection tn:-�Juger establishes a communication channel between the new client :1nd
:11! existing clients. It monitors these connections to ensure they remain fi.mcriona l . A connection hils when a comm unication ch:1nncl is unus:tble
between
a p:1ir of clients or when a client �1t either end of the channel tails. The connection lll<lllJgcr detects these cond itions and reconfigu res the set to contain onlv fu l ly connected members.
The combination of
::1 highly ;w�1ibblc communi cation channel, together wirh set membership :tnd synchronized membership chJnge responses, allows optimiz::Jtions in the lock manager's message prot o c ol.
The lock manager can send a mcss�1ge ro another node and know that either the mess:tgc will be delivered or th:Jt the configuration wi I I be altered so that it d oes not matter.
The use of the connection manager greath· sim pli tics the design and implementation of the J ock manager. The connection m<lll<lgcr allows most of the logic tor hand ling contigur:ttion changes and com munication errors to be moved away from main code p�Hhs. This increases mainline per�(mnance and simpli fies
the
logic, al lowing mon: e m phasis on correct and efticicn t operation.
Memory Channel Interconnect
Cluster performance is criticallv dependent on the cluster interconnect. This is due both to the high bandwidth requirements of bulk data transport tor
DRD and to the low latency required
tor
DLM opera tions. Although the cluster architecture allows for any high-speed interconnect, the initial implementation supports onlv the new
M EM O RY CHANNEL inter connect designed specitic:t l ly tiJr the needs of cluster svstems. This verv reliable, high-speed interconnect is based on a prn·ious interconnect designed
bv
Encore
Computer Corporation '' I t has enhanced been signitic:mtly
bv
Digital to impro1·c d�Ha integrity :tnd provide �iJr higher perh
> rman c e i n the future.
EJch cluster node h a s a
1VI EMORY C H ANNEL intcrbcc card that connects to a hub. The hub can be thought of as a switch that provides either broadcast or point-to-point connections berwcen nodes. It also pro vi des ord ering guarantees �md docs a portion of the error detection. The current implementation is an eight-node hub, but brger hubs arc planned.
The MEMORY CHANN EL interconnect pro
,·i des a 1 00-megabvte-per-sccond , memor y-mapped connection to other cluster mem be rs. As shown in figure
4,
cluster members mav map transkrs fi·om
t
he
MEM ORY CHANNEL interconnect di rectlv into their memory. The efkct is of
�1 write-only wi ndow into the memory of other cluster systems. Transkrs
�1 r e done with standard memory �Kccss instructions r�nhcr than special I/0 instructions or device access
MEMORY
CHANNEL
TRANSFER
NORMAL
MEMORY
WRITE
PAGE
In
PAGE
1--
NODE
0
ADDRESS SPACE
MEMORY CHANNEL
BUS ADDRESS SPACE
Figure 4
Tr:llls k rs Performed lw the M EM O RY CHAN NEL I nt e rconnect
MEMORY
CHANNEL
TRAN SFER
I
PAGE
NODE
1
MEM ORY
1 2
Digital
Tcchnid journal
Vol. 8 :-.io. I 1 996
protocols to �woid the overhe:�d usu�1 l l v present with rhc..:se techniq ues. The usc of mc..:morv store instruc tions resu l ts in extremely low I J t<..:�Ky ( two microsec onds) :�nd low overhead f(x J rranster of any length .
The MEMORY CHANNEL interconnect guaran tees essentially no undetectc..:d c..:rrors ( :1pproximately the same und etected error r;ne as Cl'Us or memory ),
Jl iowing the elimination of c..:hc..:cksums and other mc..:ciJJllisrns that detect software.: errors. The detected error r:Jte is also extremely low ( on rhe order of one nror per year per connec..:rion ) . Since recovery code oecutes verv infreq uently, we.: Jre assured that rela rivclv simple, brute-forc..:c recovery fi·om software errors is adequate. Using hardware error insertion, we have rested recovery code at error rates of many per second . Thus we are confident there :1re no problems at the :1crual rates.
Low-level MEMORY CHANNEL Software
Low-level sofuvare interfaces :1rc provided ro insulate the next bver of sofuvare (e .g., lock manager and dis tributed disks) trom the dctJils of r he MEMORY
CHANNEL i mplementation . We have taken the
:1ppro:1ch of providing a verv thin l:1yer to impact per f(xmance as l ittle as possible and :Jilow direct use of the
MEMO
RY C HANN EL inrerconncct. H igher-level sofrw�m: rhen isolates its use of MEMOR
Y
CHAN N E L in : 1 rcmsport layer that can I :Jter be modi fied for addi tional <.:luster interconnects.
The.: writc-onlv nature ofrhc..: MEMORY CHA.l'\JNEL inrnconnect le:1ds r.o some chal lenges in designing
:111d i m p lementing software. The onlv wa�' to see a copv of d ata written to the MEJ\,I ORY CHANNEL interconnect is to m�
1 p MEMORY CHANNEL trans fers to :1 norher region of memory on rhe same nod e .
This le:1ds t o two very visi ble programming con straints. hrsr, data is read and written rrom different addresses. This is not �1 natural programming style, and code must be written to treat a location as two vari
:1blcs, one t()l' read and one t('Jr write . Second , the ctkct of a write is delayed by the transkr latency. At t\vo microseconds, this is short but is enough ti me to execute hundreds of instructions. Hardware teatures
Jrc provided ro stall u nti l d:tta h:1s been looped back, bur \'CI')' c:�refi.d design is necessary to minimize t hese sr::dls and place them correctly. \tVe have had several subtle problems when an algorith m d id not include a stall and proceeded to read stale dar�1 that was soon overwritten by data in transi t. finding these problems is especially di fficu lt because much evidence is gone by the time the problem is observed . for example, con sider
�1 linked l ist that is implemented in a region of memory mapped to all duste r nodes through the
J\II EMORY CHANNEL interconnect. l f t\vo demems are inserted on the l ist without inserting proper waits tor rhe loop back delav, the efkct of the first insert will not be visible when the second insert is done. This resu lts in corrupting the list.
The difficulties just descri bed <1rc most obvious when de:1ling with distributed shared memory. Low level software i ntended to support appl ications is instead oriented toward a message- passing model .
This is especial ly apparent in the tcaturcs provided for error detection. The pri mary mechanisms allow either the receivi ng or the send ing node to check for anv errors over a bounded period of rime. This error c heck requires a special hardware transaction with each node and involves a loopback delay. If
�111 error occurs, the sender must retransmit a l l messages and the receiver must not use any data received in that time.
This mechanism works wel l with the expected error rates. H owever, a shared munory model makes it ex tremely difticult to bou nd the data affected by an error, unless each modification of a data element is separately checked for errors. Since this involves
J l oopbJck delav, man�' of the perceived efficiencies of sh ared memory may disappear. This is not to say that a shared memory model cannot be used . It is just that error detection and con trol of concurrent Jccess must be wel l - integrated, Jnd node tai l ures requ ire careful recovery. I n addition, the write-only nature of
MEMORY CHANNEL mappi ngs is more su ited to messJge passing than shared memory due to the extremely carefu l programming necessary to hand le de laved loopback at a separate ad dress .
AP!s :tre provided pri marily to mJnage resources, con trol memorv mappings, and provide synchroniza tion. MEMORY CHA.NNEL A Pis perti.)rn1 the follow ing tasks:
•
•
•
•
A l locJtion and mapping
- Al locate or deallocate the
CHANNEL address space.
M EMORY
- Map the MEMORY CHAN N E L int erconnect for receive or transmit.
Unmap the interconnect.
M E M O RY CHANNEL
Spin lock synchronization
- Create and delete spin lock regions.
- Acquire and release spinlocks.
Other synchronization
- Create :1nd delete write :1cknowledgment regions.
- Request write acknowledgment.
- Cre:Jte and delete sofuvare notification channels.
- Send noti fication.
- W:tit f(x notification.
Error detection and recovery
- Get cu rrent error coum.
- Check tor errors.
- Register for call back on error.
Vol .
8
No. 1
1 996 1 3
Higher layers o f software are responsi ble for transfer ring data, checking for errors, retrying transfers, and synchronizing their use of
M EM O RY CHA!'\J N EL address space after it is allocated .
Synchronization
Efficient synch ron ization mechanisms are essential for high-performance protocols over a cl uster inter connect.
M E M O RY CHA!'\JN E L hardware provides two i mportant synchronization mechanisms: first, an ordering guarantee that all writes arc seen in the same order on al l nodes, including the looped- back write on the originating node; second , an acknowledgment request that returns the current error state of al l other nodes. Once the acknowledgment operation is com plete, all previous writes are guaranteed either to have been received by other nodes or reported as a transmit or receive error on some node. We have implemented clusterwide software spinlocks based on these guaran tees. Spinlocks are used for many purposes, including i n ternode synchronization of other componen ts and concurrency control for the cl usterwide shared-mem ory data structures used by the low- level
1YI E M O RY
CHAN N E L software .
A spin lock is structured as a n array with one clement for each node. To acquire the spinlock, a node first bids for it by writi ng a value to the node's array ele ment. A node wins by seeing its bid looped back by the
M EM O RY CHAN N EL i nterconnect without seei ng a bid from any other node. The orderi ng guarantees of the
M E M ORY CHAN N E L ensure that no other node could have concu rrently bid and believed it had won .
Multiple nodes can realize they have lost, but more than one node cannot wi n. In case of a con flict, many different back-off techniques can be used . The win ning nod e then changes i ts bid va lue to an own value.
This last step is not necessary for correctness, but it does help with resolving contention and with various fai l u re recovery algorithms. AJI higher- l evel synchro nization is bui l t on com binations of spi n l ocks, order ing guarantees, and error acknowledgments.
Error Recovery and Node Failures
Most of the difficult problems in the low- level soft ware relate to error recovery and node tailures. I n spire of its reliabil ity, errors wil l occur in the
M E M O RY
C HAN N E L interconnect, and they must be hand led as transparently as possible . Transparency is key to sim plifYing the com munication model seen by higher level software. In addition, node fail ures from hardware or software faults are more freq uent than
M E MO RY C H A N N E L errors and must be dealt with even in the most inconvenient portions of the low level code. The
M EM O RY CHAN N E L interconnect is managed through a collection of distri buted data structures that must be kept consistent. Software locks arc used to synchronize access to these structures, but errors may l eave them in an inconsistent state .
Guaranteed error detection before the release of a lock allows operations to be redone in case of an e rror.
Thus, all sequences of
M EM ORY CHAN N E L writes must be idempotent to take advantage of rhis straight forward error-recovery technique.
If a node failure occurs, a su rviving node must make a l l data structu res consistent before it rekases locks held by the fai led node. To keep this a manageable task, we have written carefully structured algorithms to handJe each inconsistent state . I n general, struc tures arc changed such that a single atomic write com mits a change. ff a node fails before this lasr write, no recovery is necessary. As an example, consider a data structure that is completely initialized before being added to a l ist.
A single write is used to accomplish the list addition . I f a node tails, the last write was either done or not and, in either case, the list is consistent.
Complications arise when another node has
;1 receive error on the l ast write done by a failing node. In this case, the fai led node cJ.nnot retry after detecting the error, so the node with the receive error has a difrerenr view of the list than a l l other surviving nodes. To resolve this event, one node must propagate its view of the list to a l l other nodes before it releases the lock held b�, the tailed node. Any node can do this because each has a sel rconsistcnt view of r he list. If the node with the receive error propagates i ts view, the last ele ment added by the failed node is lost. This situation is no different, however, from having the node fai l a few instructions earl ier. The chal lenge is to design recov ery for all these cases and maintain our sanity by mini mizing the number of such cases.
Another interesting problem is maintaining a con sistent cou nt of errors across a l l nodes. This count is key to the error protocols of both the low-level
M EM ORY C HA N N E L software and higher layers since comparisons of a saved and a current value bou nd the period over which data is suspect. The cou nt may be read on one node, transferred with a message, and com pared to a cu rrent va lue on another node. Thus, a consistent value on all nodes is critical and must be mai ntained in the presence of arbi trary combi nations of receive and transmit errors .
( Although errors arc very infrequent, they may be cor related; so algorithms must work well tor error bursts . )
T h e write acknowledgment, described earl ier, guaran tees that other nodes have received a write without error. It is used both to imp.lement a Jock protecting the error cou nt and to guaran tee that all nodes have seen an updated cou nt. Updating the count is a slo\\' operation due ro multiple round-trip delays and long error ri me-outs, bur it is performed very infreq uently.
1 4
Digital Tt:chn ictl )ound
Vol . 8 No. I 19')6
Future Enhancements to MEMORY CHANNEL
Software
Fully supported MEMORY CHANNEL AP!s are cu rrently available only to other layers in the UNIX kernel tor two important reasons: First, M EMORY
CHANNEL is a new type of interconnect and we want to better understand i ts uses and advantages before commi tting to a ful l y fu nctional API for general use.
Second, many difficult issues of secu rity and resource l i mits will affect the final i nterface. To help Digital and i ts customers gain the necessary experience, a lim ited functionality version of a user-level M EMORY
CHANNEL API has been impl emen ted i n the version
1 .0 product. This interface su pports al location and mapping of MEMORY CHANNEL space along with spinlock synchronization . It is oriented toward sup port of paral lel computation in a cl uster, but we also expect it will serve the needs of many commercial appl ications. Once we have a better understanding of how high- level applications will use the M EMORY
CHA.t'\INEL intercon nect, we wil l extend the design and provide add itional AP!s oriented toward both commercial applications and technical computing.
Application Fa ilover
Digital's TruCi uster multicomputer system is a logical evolution of the DECsafe Avai lable Server Envi ronment (ASE). An ASE system is a m u l tinode con figuration with all nodes and all high l y available storage connected to shared SCSI storage buses.
Figure 5 shmvs an ASE configuration. Software on each node monitors the status of a l l nodes and of shared storage. In case of a fai l ur e , the storage and associated applications are tailed over to surviving sys tems. Planned application failover is accomplished by stopping the appl ication on one node and restarting the application on a surviving node with access to any storage associated with the appl ication. Application specific scripts control tailover and usually do not require appl ication changes.
Figure 5
Typical ASE
Con tiguration
In addition to supporting the appl ication fai l over mechanisms from ASE, the TruCluster system sup ports paral lel applications running on m u l tiple cluster nodes. In case of a fai l ure, the application is not stopped and restarted . I nstead , it may continue to exe cute and transparently retain access to storage through a distributed disk server. I n addition, more general hardware topologies are supported .
Hardware Configurations
The TruCi uster version 1 . 0 prod uct supports a maxi m u m of tour nodes connected by a high-speed
MEMORY CHAN N EL interconnect. The nodes may be any Digital UNIX system with a peripheral compo nent i nterconnect ( PCI ) that supports storage and the
MEMORY CHANNEL interconnect. Highly available storage is on shared SCSI buses connected to at least two nodes. Thus, a c luster looks l i ke multiple ASE systems joined by a cl uster interconnect.
Although tl1e l imitation to four nodes is temporary, we do not intend to support large numbers of nodes.
Ten to twen ty nodes on a high-speed interconnect is a reasonable target. A cluster is a component of a dis tribu ted system , not a replacement for one. If very large numbers of nodes are desired, a distributed system is built with cl uster nodes as servers and other nodes as cl ients. This allows maintaining a simple model of a cluster system without havi ng to allow for many comp lex topologies. Aside from simpl icity, there are performance advantages from targeting algoritllms for relatively sma ll and simple cluster systems.
Although the nu mber of nodes is intended to be small, the individual nodes can be high-end mu ltiprocessor systems. Th us, the overall computing power and the f/0 bandwidth of a cluster are extremely large.
Conclusions
With the comp letion of the first release of Digita l 's
TruCluster prod uct, we believe vve have met our goal of providing an envi ronment tor high- performance commercial database servers. Both the d istributed lock manager and the remote d isk services are meeting expectations and providing reliable, high-performance services for paral lel ized applications. The M EM O RY
CHAN N EL i nterconnect is proving to be a n excel lent cluster i n terconnect: Its synchronization and fai l ur e detection are especially compatible with many cluster aware components, which are enhanced by its low l atencies and simplified by i ts elimination of compJex error hand ling. The error rates have also proven to be as pred icted . With over 100 units i n use over the last year, we have observed only a very smal l nu mber of errors other than those attributable to debugging new versions of the hardware.
Digital Tedmical j ou rnal Vol . 8
No.
1
1996 I S
Detailed component performance measurements are stil l i n progress, but rough comparisons of D RD against local I/O have shown no signitlcant penalty in latency or throughput. There is of course additional
CPU cost, b u t it has n ot proven to be significant for real applications. DLM costs are comparable to VMS and thus meet our goals. Audited TPC-C results with t11e Oracle database also validated both our design approach and the implementation details by showing that database performance and scaling with addi tional cluster nodes meet our expectations.
The previous best reported TPC-C num bers were
20,9 1 8 tpmC on Tandem Computers' H imalaya
Kl0000- 1 12 system with the proprietary NonStop
SQL/MP database software. The best reported num bers with open database software were 1 1 ,456 tpmC on the Digital AlphaServer 8400 5/350 with Oracle7 version 7 . 3 . A four-node AlphaServer 8400 5/350 cluster with Oracle Paral lel Server was recentJy audited at 30,390 tpmC. This represents i ndustry-leadership performance with nonproprietary database software.
Future Developments
We will continue to evolve the TruCiuster prod uct toward a more scalable, more general computing envi ronment . In particu lar, we will emphasize distributed fi l e systems, configuration flexibility, management tools, and a single-system view tor both i n ternal and client applications. Work is u nder way tor a cluster fi le system wiili local node semantics across the cluster sys tem. The new cluster file system will not replace D RD but will complement it, giving applications the choice of raw access through D RD or ti.tiJ, local-file-system semantics. We are also lifting the four-node limitation and allowing more flexibility in cluster i n terconnect and storage configurations. A single network add ress for tl1e cluster system is a priority. Finally, fu rther steps in managing a mu lti node system as a single system will become even more i mportant as the scale of cluster systems i ncreases.
F urther in the future is a true single-system view of cluster systems that wil l transparently extend all process control, communication, and synchronization mechanisms across the entire cluster. An i mplicit trans parency requirement is pertormance.
Acknowledgments
In addition to the au thors, the fol lowing i nd ivid uals contribu ted d irectly to the cl uster components described in this paper: Tim B urke, Charlie B riggs,
Dave Cherkus, and Maria Vella for D RD ; Joe Amato and M itch Condylis for DLM; and Ali Rafieymehr for
MEMORY CHANN EL. Hai H uang, Jane Lawler, and
especially project leader Brian Stevens made many direct and indirect contributions to the project.
Thanks also to Dick B uttlar for his editing assistance.
References and Notes
1 . "Introduction ro DCE,"
OSF DCE Documentation Set
( Cambridge, Mass . : Open Soltware Foundation, 1991 ).
2. Internet RFCs 1 0 1 4, 1057, and 1094 describe ONC
X D R, RPC, and NFS protocols, respectively.
3. G . Pfister,
ln Searcb of Clusters
( U pper Saddle River,
N.J . : Prentice- HaJ J , Inc., 1 995): 1 9 -26.
4.
N.
Kronenberg, H. Levy, and W. Strecker, "VAXclusters:
A Closely-Coupled Distributed System,"
ACil1
Trans actions on Computer S)stems,
vol. 4, no. 2 ( May
1 986): 1 30-146.
5. L. Cohen and
].
Wi lliams, "Techn ical Description of the DECsatC Available Server Environment,"
Digital
Tecbn ical.fournal,
vol. 7, no. 4 ( 1 995 ) : 89-100.
6. TPC performance numbers tor U N IX systems are typi cally reported for databases using the character device interface.
7. The rile system inrerfaces on the Digital UNI X operat ing system arc bei ng extended to support di rect ljO, which results in bypassing the block bu ffer cache and reduci ng code path length for those applications that do nor benefit from use of the cache.
8. A fast wide differential (FVV D ) SCSI bus is limited to a maximum distance of about 25 meters tor example.
9. M. Devarakonda cr al., " Evaluation of Design Alterna tives for a Cluster File System,"
USENTX Conference
Proceedings, USEN IX Association, Berkeley, Calif.
(J anuary 1 995 ).
10.
J.
Gray and A. Reuter,
Tra nsaction Processin[;
Concepts and Techniques
(San Mateo, Calif. :
Morgan Ka utinan Publishers, 1 993 ) .
1 1 . This mechanism i s inherited fi·om the DECsafe Avail a ble Server management faci.lity, including rhe asemgr inrertace.
1 2. As an example, if the first DRD service created for a cluster is 1 , the D lW device special fr le name is
/dev/drd/drd
I and irs minor device number is also 1 .
1 3 . C. Juszczak, " Improving the Performance and Cor rectness of an NFS Server," USL�·.VJx Conference Pro ceedings,
USENJX Association, San Diego, Calif.
( Winter 1 989).
1 4. W. Snaman, Jr. and D . Thiel, "The VAX/VMS Distrib uted Lock Manager,"
Digital Techn ical journal,
vol. 1, no. 5 (September 1 987): 29-44.
1 5 . R . Golden berg, L. Kenah, and D. Dumas, VAX/VMS
Internals and Data Structures ( BecHard, Mass. :
Digital Press, 1 99 1 ) .
1 6
Digital Technical Journal
Vol.
8 No.
I
1 996
1 6.
TruC/uster AfJj>!icatinn Pmgrc.nnm ing inter/aces
Gu ide ( M�wncml , 1'vlass . : D i g it
� l
Eq uipment Corpora tion, Order No.
AA-QL8 PA-TE, 1 996).
1 7.
T.
Rengclr<l)'ln, P. Spi r o
, and W. Wri ght, " H igh Avai l clbilitY MedwJisms o f VAX DBMS Software," Digital
Techuica/ .fou rna!. vo l .
1 , no.
8 ( l:;ebruarv 1 989 ) :
88-98.
1 8 . f:'ncnre 91 Series
Tech nical St11111nary ( Fort La ud erd a l e , rh1.: Encore Compu ter Corporation,
1 99 1 )
.
Biographies
Wil l iam
E.
Snaman, Jr.
Sandy Snaman joined Digital in 1 980. H e is cu rrenrlv a consulting sotn1-are engineer
Group, where in Digital's U N I X Sott\\'Jre he c o nt r i bu t ed ro the TruCiusrc:r architec run: and design . He and members of his group designed and im plemented c luster eomponems such as the con nection mcmager, lock manager, cm d various aspects of cluster commun ications. Previously, in ing Group, he was the pr o j ec t the VMS Engi neer leader �(>r the port of the
VMSclustcr svstcm ro t he AJpha pl att(>rm and the technica l supervisor a n d project kad er tor area .
Sandy rhe VAXcluster cxcc uri,·e a l so reaches MS Windows programm in g and
C++ at
Uni
D<1niel W e b st e r Col lege. H e has a B.S. i n compu ter science a n d an M.S. in intormarion systems trom the v e rs i t y of Lowell.
Wayne M. Cardoza
Wavnc
C a rdo z c 1 is c 1 senior consulting engineer in the
U N I X E n gi n e er ing Group. He joined DigitJI i n 1 9 79
Jnd c o n t ri b ute d to v�rious areas ofrhc VMS kernel prior to joi ning rhe U N I X C roup ro work on rhe U N I X cl uster producr. W:1ync was also one: of the architects of PIUSM,
Jn e<Jrly Digital t()r this work. More recently, he participated ofthe Alplu AX
Al ph<L B e t o r
RISC: arc h i tecru n: ; he holds several patems
I' architecture and the
Open VMS port to e coming to DigitJI, Wc1vne wc1s in the design employed bv
Bdl Laboratories. He rec e iv ed a B.S. E. E . trom Southeastern
Massachuse tts Uni1·crsit\· and an
M.S . E . E. trom MIT.
Frederick S. Glover
Fred Gl o v e r is '' soft:wc1n: consulting engi neer <l nd the rec h
nicll di rector of the
Digi t a l U�J
X Bc1se Operating System
Croup. Since joining the Di g i ta l
U N I X
Group in 1 9 8 5 ,
!-'red has contributed to the de ve l o pm e nt of networking serv i c e s, local c1nd remote ti le systems, <1nd c l uster tech nol ogy.
He has served as the chair of the
I 1:-:TI-'
/lS IG Truste d
NFS Working
Croup,
:JS the chc1 i r ofrhe
OS!-'
Distribu ted
File Svsrem Working Group, to the I E F.F. POS I X
:md as Digita l 's representative
1 00 3 . 8 Transparem hie
Access ·work ing Group. P1·ior to Jo in i n g D ig i ta l , Fred was e m pl oyed bv l\T&T Bell Laboratories, where co-development of the R.I\1AS h i s contri butions i ncluded network communication s ub s ys t e m. He re n: i v e d B .S. and M.S. degrees in computer science trom Ohio State Uni v ersit y cmd con ducted h i s t h e s is resc:u·ch i n the areas oftault-tolcrant distributed computing and data flow arcbi tecru n: .
Digital TechnicJI /oum<ll
Vol .
8
No. 1 1 996
1 7
डाउनलोड
विज्ञापन
संबंधित