Digital VAX 8974, VAX 8978, VAX 8700, Local Area VAXcluster, MicroVAX II VAX Technical journal
विज्ञापन
विज्ञापन
VAX cluster Systems
Digital Technical Journal
Digital Equipment Corporation
Number 5
September 1987
Cover Design
VAX cluster systems are featured in this issue. The central connection between the elements in a cluster is called the
Star Coupler. Our star-filled cover evokes the thousands of
VAXcluster systems now operating worldwide. The image was created using the Lightspeed System.
The cover was designed by Barbara Grzeslo and Tim Roberts of the Graphic Design Department.
Editorial Staff
Ediwr- Richard W. Beane
Production Staff
Production Editor- Jane C. 131ake
Designer- Charlotte Bell
Interactive Page Makeup- Terry Reed
Advisory Board
Samuel H. Fuller, Chairman
Robert M. Glorioso
John W. McCredie
Mahendra R. Patel
F.
Grant Saviers
William D. Strecker
The Digital Technical journal is published by
Digital Equipment Corporation, 77 Reed Road,
Hudson, MassachusettS 01749.
Changes of address should be sent to Digital
Equipment Corporation, attention: Media Response
Manager, 444 Whitney Street, NR02-1/J5, Northboro,
M.A
01532-2599
Comments on the content of any paper are welcomed.
Write to the editor at Mail Stop HL02-3/K11 at the published-by address. Comments can also be sent on the ENET to RDVAX::BEANE or on the ARPANET to
BEANE%RDVAX.DEC@DECWRL.
Copyright© 1987 Digital Equipment Corporation.
Copying without fee is permiued provided that such copies are made for use in educational institutions by facuhy members and are not distributed for commercial advantage. Abstracting with credit of
Digital Equipment Corporation's authorship is permitted Requests for other copies for a fee may be made to the Digital Press of Digital Equipment
Corporation. All rights reserved.
The information in this journal is subject to change without notice and should not be construed as a commitment by Digital Equipment Corporation. Digital
Equipment Corporation assumes no responsibility for any errors that may appear in this document.
ISBN 1-55558-004-1
Documentation Number EY-8258E-DP
The following are trademarks of Digital Equipment
Corporation: CI, DEC, DECnet, DECnet-VAX,
DECsystem-1 0, DECSYSTEM-20, Digital Network
Architecture (DNA), Digital Storage Architecture
(DSA), the Digital logo, HSC, Local Area VAXcluster,
MicroVAX, MicroVAX II, MicroVAX 2000, Q-bus,
RJ\1S- I I , SA482, UNIBUS, VAX, VAX-11/750,
VAX-11/780, VAX-11/782, VAX-11/785, VAX 8600,
VAX 8650, VAX 8700, VAX 8974, VAX 8978,
VAXcluster, VAXstation, VAXstation
JJ,
VAXstation ll/GPX, VA.Xstation 2000, VMS,
VT, VT220
IBM is a registered trademark of International Business
Machines, Inc.
Intel is a trademark of Intel Corporation.
Lightspeed is a trademark of Lightspeed Computers,
Inc.
Book production was done by Educational Services
Media Communications Group in Bedford,
MA.
Contents
V AXcluster Systems
7
The VAX cluster Concept: An Overview of a Distributed System
Nancy P. Kronenberg, Henry M. Levy, William D. Strecker, and Richard J. Merewood
22
The System Communication Architecture
Darrell
J.
Duffy
29
The VAX/VMS Distributed Lock Manager
William E. Snaman, Jr. and David W. Thiel
4 5
The Design and Impl emen tation of a Distributed File System
Andrew C. Goldstein
56
Local Area VAXcluster Systems
Michael S. Fox and John
A.
Ywoskus
69
VAXcluster Availability Modeling
Edward E. Balkovich, Prashant Bhabhalia, William R. Dunnington, and Thomas F. Weyant
80
System Level Performance of VAX
8974 and
8978
Systems
Daeil Park, Rekha D. Von Ehren, Tzyh-jong Wang, and Nii N. Quaynor
93
C/ Bus Arbitration Performance in a VAXcluster System
Xi-ren Cao, Nii N. Quaynor, and Fernando C. Colon Osorio
Editor:ls Introduction
Richal"d
Editor
W.
Beane
VAXc luster systems are closely cou pled configu· rations of VAX CPUs and storage devices . The VAX
C PU a t a n y n o d e c a n c o m m u n i c a t e w i t h the processor and storage dev ices at a n y other node in the cluster . The i nterconnects and soft· ware used to activate this unique concept allow data transfers at u p to 70 megabits per second between nodes. This issue of the
Digital Techni· cal journal conta i ns papers about some of the key hardware and software features i n these sys· terns , as wel l as some measures of their perfor· mance . Since several orga n i zations within Digital are responsible for various VAXcluster features , these papers are comribured by engineers from a wide spectrum of engineering groups.
S in c e the VAXc luster concept spans s u c h a r a n ge of t e c h n o l o g i e s , t h e f i rst p a p e r i s a n overview ex p l a i ni n g genera l ly how t hese sys tems work. Nancy Kronenberg, Han k Levy, B i l l
Strecke r , and Richard Merewood describe t h e architecture, the storage control , t he VMS soft· ware a lterations, and the mu l ti tude of activi ties that control access to the storage devices.
The System Commu n i c a t i o n Arc h i tecture , described by Darre ll Duffy, is the structure that a l l ows the nodes in a VAXcluster system to coop erate. This relatively si m p l e framework governs t he s h a r i n g of data betwe e n resou rces at t h e nodes a n d binds together applications that run on d i fferent VAX CPUs.
Add i t ional features were needed i n the VMS software to accommodate ac cess i n g d i s ks on multiple systems. The distribu ted lock manager, described by Sandy Snaman and Dave Thie l , pro· vides the synchron ization needed to accomplish transparent data transfers between c l uster mem· hers. Other changes were a lso needed to broaden the file functions performed by the VMS software .
Andy Goldstei n relates some alternative ways to expand those functions and how the QJO proces· sor was extended ro sync hroni ze fi l e accesses.
The resul ting system of locks and queues pro· v ides a con s i stent sequence for managi n g d is tributed fi les.
The next paper. by M i ke Fox and joh n Ywoskus, descri bes the extension of the VA.-'Ccl uster con cept ro systems connected w i t h a n E t herne t .
Tbcsc Local Area VAXcluster systems use speci a l software t o provide fu nctions needed b y clusters, but not p rov ided by E t hernet software . Thus,
M icroVAX II and other sma l l VAX systems can be clustered to yield sign i ficant a mounts of process· ing power.
The last t hree papers deal with performance aspects of VAXcluster systems. The paper by Ed
Ba l kovich, Prashant Bhabhalia, D ick Dunnington, and Tom Weyant d iscusses the results of a VAX· cluster model that demonstrates how redundancy im proves ava i l ab i l i ty. The n , Dale Pa r k , Re kha
Von Ehren, T·J. Wang, and Nii Quaynor describe two mode ls they developed to measure the per· formances of VAX
8974 and 8978 systems. These models, based on ben c hmarks ru n in d i fferent environments, use a VAX
8700 CPU for a base l i ne comparison .
The fi nal paper relates the resu lts of a model to measure the c haracterist ics of the CI bus . Xi -rcn
Cao , N i i Quaynor, and Fernando Colon Osorio describe how t h e i r model measures the p e r· formance of t he arbi tration a l go r i t hm i n t h i s bus. They suggest some i n teresting schemes to improve uti l i zation and reduce response time.
2
Biographies
Edward E. Balkovich
Ed Ba lkov ich is the manager of V�'{c luster System
Engi neering, which addresses issues of VAXcluster pcrformance, availabi l i ty and architecture for H igh Performance Systems. He was Digita l 's associate director of Project Athena at M . I .T. and is an Adjunct A.o;sociate Professor at
Brandeis University. Before joining Digital in 1 9R I, Eel was a faculty member at the University of Connecticut. He earned his B.A. degree ( 1968) from the
University of Cali fornia at Berkeley, and his M . S. ( 197 1) and P h . D. ( 1 976) degrees from the University of Californi a at Santa Ba rbara . He is a member of t he ACM and IEEE.
Prashant Bhabhalia
A principal engi neer i n V�'Ccl uster Systems Eng i· neering, Prashant Bhabhalia develops and in terprets re liabi l i ty and avai labi l· ity mode ls. Earl ier, he was a program manager in Computer Systems Manu facturing and a sen ior engineer in GlA Manufacturing. Before joi n i ng Digital in 19RO, Prashant was an industrial engi neer at Norton Company and Gits
Plastic Corporation . He holds an M.S. I .E. degree ( 1 97 4) from the Polytech nic I nstitute of Brooklyn and a B . S . M.E . degree (
197 2) from the M.S. Univer sity in India . Prashant is a senior member of l . I .E.
Xi-Ren Cao
A.<> a principal software engineer in the H igh Performance Sys tems and Clusters Group, X i - Ren Cao mode ls and evaluates VAXcluster con figurations. Before joining Digital i n 1986, he was a research fe llow at Har vard Universiry. Xi -Ren has published over 20 technical papers on performance evaluation, si mulation, srochastic systems, queuing networks, and control t heory, and has co-authored a book "Perrurbation Analysis of Dis crete Event Systems ," to be published in 1988. He received h is P h . D . degree from Harvard Un iversity i n
19R4 and is a member of IEEE.
Fernando C. Colon Osorio
Fernando Colon Osorio graduated from the
Un iversity of Puerto Rico ( B .S . E . E . , 1970) and the University of Massachu setts ( M . S . , Ph . D ,
1976). Joining Digital i n l97(J. he helped design the PDP-
1Ij60 and PDP-11/74 systems and managed the
LAN group i n Corporate
Research . Fernando also managed the overall design verification for the V�'{
8600 project . In H igh Performance Systems. he now manages the systems research and advanced development group, responsible for VAXclusters. fault tolerance, advanced arc h itectures, and performance ana lyses. He was
Associate Ed i tor of the IEEE Transactions on Com puters and is the co-author of "Engineering I n te l l igent Systems."
Darrellj. Duffy As a consulting software engi neer. Dan-ell Duffy works on the network architecture for VAXcluster systems . On prev ious projects , he led the deve lopment of operating systems for parallel processors and wrote software for the Local Area Terminal protocol . Darre l l helped to deve lop
DECnet software after joi n i ng Digital in 1977 He received a B . S . in com puter science from West Virginia University in
197 2 and worked at the Un i versity of Florida . Darre l l and three other Digital engi neers have applied for a parent on the LAT protocol .
.-1
4
Biographies
William R. Dunnington
Dick Dunni ngton is a principal qual i ty engineer working on availabil i ty modeli ng in the Com puter System Manufactu ring
Grou p . Prev iously, he was a quali ty engi neer in the Far East Manufactu ring
Grou p , working on personal computer memories . Before join ing Digital i n
1979, D ick was a captain i n the U .S . Army. H e received a n A<>sociates degree in liberal arts from S.U . NY ( 1973) and a B.S. degree in engineering science from the University of Nebraska (
1 97 4). Dick, a member of SIAM and ASQC, is a l so a Certified Qua l i ty Engineer.
Michael S. Fox
I n
J
977, Mike Fox joined Digital after earning his M.S.
( 1977) and B.S. ( 1976) degrees i n computer science from Rensselaer
Polytechnic Institute. Initial ly, he helped to develop the RSXI 1 M-PLUS soft ware, then served as archi tect and supervisor on the PRO/SERVER project . I n
D i gi tal's Graduate Engineering Education Program , M i ke returned to Rensse l aer for a year as a faculty member in computer science. Back at Digita l , he joined the VMS Engineering Group and lead the p roject that developed the
Local Area VAXcluster software. M i ke is now a consulting software engineer.
Andrew C. Goldstein
Andy Goldstein received h is B . S . E . E . and M . S . E . E . degrees from M. l .T in
197 1, and joined Digital in 1973. H e was ini tially responsible for the fi le system in the RSX-11 D and RSX-11 M systems, and became a charter member of the VMS Deve lopment group. Andy designed and implemented the VMS fi le system, and worked as wel l on the VMS ljO and executive software . More recently, he designed the security features i n
VMS version 4 . 0 and helped w i t h the VAX c l uster file system . Andy i s now a sen ior consu ltant software engineer, and is a member of Tau Beta Pi , Eta
Kappa Nu , Sigma Xi , and ACM .
Nancy P. Kronenberg Nancy Kronenberg i s a sen ior consu ltant software engineer in the Advanced VAX Deve lopment Grou p . She is currently project leader of the microcode team for a new VAX CPU. Previously, Nancy worked i n the VMS Development Group where she assisted with the SCA specifica t ion and wrote the CI port driver and pan of the VMS SCA services. Before joi n i ng Digita l i n
1978, she was a systems analyst at Massachusetts Computer
Associates and at Appl ied Data Researc h . Nancy earned her AB degree i n physics from Corne l l Un iversity in
196 7 .
Henry M. Levy
A consul tant engi neer on leave from Digita l , Hank Levy is currently an A'isistant Professor working on distributed systems and com puter architecture research at University of Washi ngton . Hank joined D igita l i n 1 97 4. H e was a member o f t he original VAXjVMS team and later worked for the VAX Architecture Group on i nterconnect and workstation archi tectures. He has publ ished over a dozen papers and the books Capab i l i ty
Based Computer Systems and Computer Programm ing and Arc hitecture: The
VA..,'<-11. Hank holds a l3.S. degree (
1974 ) from Carnegie-Me l l on Un iversity and an M . S . degree
( 19R 1) from University of Washington .
Richard J. Merewood
Richard Merewood is the software development manager for the DECnet-VAX , Local Area VAXcluster, and VAXcluster soft· ware projects. In Reading, England, he managed the development of Digital's
X . 25 networking products, performed advance development on the ISDN project, and supervised a modem development project. Before joining Digi tal in 1 980, Richard was an international consultant in data communications and transaction processing. He studied electrical engineering at the I mperial
Col lege of Science & Technology, London .
Daeil Park As a principal software engineer in the Systems Performance
Group, Dale Park executes and ana lyses tests to determine VAXcluster per· formance. He is particularly involved with measuring the performance of application programs on these systems. Dale joined Digital in 1 983 after receiving his M.S. degree in computer engineering ( 1 983) from Case West ern Reserve University. Earlier, he was a system design engineer at Samsung
Electronics Co . Ltd , in Korea . Dale earned his B.S. degree in electrical engi neering (1977) from Seoul National University in Korea .
Nii N. Quaynor
After earning his B . E . degree from Dartmouth Col lege i n
1973 and his Ph . D . from S . U . N . Y . at Stony Brook in 1 97 7 , Nii Quaynor joined
Digital in 1 978. He first worked in corporate research on multimicro sys tems. In 1 98 2 , Nii joined the VAX 8600 project as a consu lting software engineer and created models for large-scale CAD applications using a register transfer language . Later, he worked on the verification of the VAX 8600 design . Nii is now the manager of the System Performance Group in High Per formance Systems.
William E. Snaman, Jr.
Sandy Snaman is a principal software engineer in the VMS Development Group, currently working on software for VAXcluster systems and the distributed lock manager. Sandy has also developed and taught VAXcluster courses in Educational Services and was a software main tainab i l ity engineer for Customer Services Systems Engineering. He joined
Digital in 1 980 after eight years in the U.S. Navy. Sandy holds a B.S. degree
( 1 985, Magna Cum Laude) from the University of Lowe l l , where he is now completing his M.S. degree in computer science.
William D. Strecker
Bill Strecker, vice president for Product Strategy and
Architecture, joined Digital after receiving h is B.S. , M . S . , and Ph . D . degrees from Carnegie-Mellon University. Bil l's work on cache memories led to the
PDP- 1 1 /7 0 system , and he also led the team that developed the VAX archi tecture . Bill guided Digital's interconnect strategy, which lead to the com puter interconnect (CI) and the Systems Communication Architecture . He holds several patents on CPU designs and computer interconnects. Bill and was elected to the National Academy of Engineering in 1 986.
5
6
Biographies
David W. Thiel
Dave Thie l , a consulting software engineer, is currently studying future directions for VAXcluster systems in the VMS Development
Group . He was project leader for the i nitial VAXcluster support i n VMS ver sion 4 . 0 . Dave a lso worked on the executive and data compression areas of the VMS software. Dave joined Digital i n 1 980 from GenRad, I nc . , where he was a principal software engineer. He earned his B . S . E . E . , M . S. E . E . , and Elec trical Engineer degrees from M . I .T. in 1 97 2 . He is a member of Tau Beta Pi ,
Eta Kappa Nu, ACM , a nd IEEE .
Rekha D. Von Ehren
As a senior software engi neer i n the Systems Perfor mance Group , Rekha Von Ehren works on performance measurements and ana lyses for VAXcl uster systems. Previously, she analyzed the performance of
VAX 8600 and 86'50 CPUs. Rekha joi ned Digital in 1 983 after receiving her
M . S . degree in i ndustrial engineering from the University of Wisconsin . She also earned an M . S . degree ( 1 98 1 ) i n operations research from the London
School of Economics and a B . S . degree i n statistics and computing from North
London Polytechnic. Rekha has just given birth to her first chi ld, a baby boy, named Samuel.
Tzyh-Jong Wang
As a pri ncipal engineer i n the Systems Performance
Group, Tzyh-Jong Wang conducts modeling stud ies to measure system per formance . He analyzes VAXcluster configurations, on- l i ne transaction pro cessing, and other advanced systems. Before joining Digital in 1 98 7 , Tzyh jong was a lecturer at the University of Wisconsin at Madison, where he received h is M . S . and P h . D . degress ( 1 98 7 ) in information systems. He a lso earned a B.S. I . E . degree ( 1 978) from the Nationa l Tsi ng-Hua University, Tai wan . Tzyh-jong is a member o f ACM , I E E E , ORSA, a nd TIMS.
Thomas F. Weyant
Tom Weyan t is the manager of the Systems Rel iabi l i ty
Engineering Group i n Computer Systems Manufacturi ng. As a consulting engineer, he worked on systems reliab i l i ty a nd avai labi l i ty modeling, com puter- i nterconnect reliab i l i ty, infant-morta.lity and long-term failure-rate model ing, and was the manager of advanced development . Before joining
Digital i n 1 98 5 , Tom worked for ten years at AT&T Bel l Laboratories and
Hughes Aircraft Company. He earned his 13.S. M . E . degree ( 1 9 7'5) from the
University of Ca liforn ia at Santa Barbara , and his M .S. and Ph . D . degrees
( 1 98 1 ) i n operations research from UCLA.
John A. Ywoskus john Ywoskus is a principal software engineer with the
VAXjVMS Development Group . He is currently project leader of the
Local
Area VAXcluster development effort and was lead technical contributor i n the development of the first release of this product. Before joining the VMS group in 1 98'5, john worked as a developer on the LAT- 1 1 terminal server project and as project leader of the LATplus Vl. O application terminal pro ject . john came to Digital in 1 98 1 from the Charles Stark Draper Laboratory, where he worked on CAD system software deve lopment . He earned a B.S. degree i n Appl ied Mathematics from Harvard Col lege in 19 7 9 .
Nancy P. Kronenberg
Henry M. Levy
William D. Strecker
Richard J. Merewood
The VAXcluster Concept:
An Overview of a Distributed System
A VAXcluster system is a highly available and extensible configuration of
VAX computers that operate as a single system. To achieve high perfor mance in a multicomputer environment, a new communications architec ture, communications hardware, and distributed software had to be jointly designed. The software is the VAXjVMS operating system, using a distributed lock manager to synchronize access to shared resources. The communications hardware includes a 70-megabit per second message-ori ented interconnect, and an interconnect port that performs communica tions tasks traditionally handled IJy software. The Local Area VAXcluster system, an implementation of the VAXcluster architecture, uses a standard
Ethernet as its interconnect. This development provides VAXcluster func tions for the Micro VAX family.
Contemporary mul ticomputer systems typically lie at the ends of the spectrum delimi ted by tightly coupled multiprocessors and loosely cou pled distributed systems. H istorically, loosely coupled systems have been characterized by the physical separation of processors, low-bandwidth message-oriented interprocessor communication , and independen t operating systems . 1·2··H Con versely, tightly coupled systems have been char acterized by close physical proxi mity of proces sors, high-bandwidth communication through shared memory, and a single copy of t he operat ing system .'·6·7
An intermediate approach taken at Digital
Equipment Corporation was ro build a "c losely coupled" structure of standard VAX computers,R caJ i ed a VAXcluster syste m . By closely coupled , we imply that a VAXcluster system has character istics of both loosely and tightly coupled systems .
On one hand, a VAXcl uster system has separate processors and memories connected by a mes sage-oriented i nterconnect, running i nstances of the same copy of the distributed VAXfVMS oper ating system . On the other hand , the initial
The original version o f 1his paper appeared in "VAXclus1ers:
A
Closely·Coupled Dis1ribU1ed Sys1em," by Nancy
P.
Kronen berg, Henry M. Levy, and William D. S1recker, published in
ACM Transactions on Computer .\:ystems,
Vol.
May
4,
No.
2,
1986
Copyrigh1
1987,
Associa1ion for Compming
Machinery, Inc. implementation of the cluster rel ied on close physical proximity, a single (physical and logi cal) security domain, shared physical access to d isk srorage , and high-speed memory-to-memory block transfers between nodes .
The goals of the VAXcluster multicomputer sys tem are high availability (in suitable configura tions) and easy extensibility to a large number of processors and device controllers. In contrast to other h ighly available systems,9 10 1 1 · 1 2 a VAXclus ter system is built from general-purpose, off-t he shelf processors ranging in size from MicroVAX workstationsl.\ to high-performance VAX CPUs, and a general-purpose operating system .
A key concern i n this approach is system performance . Two important factors in the per formance of a multicomputer system are the software overhead of the communications archi tecture and the bandwidth of the computer inter connect . To address these issues, severa l develop ments were undertaken as part of the original
VAXcJuster design , i ncl uding
•
A simple, low-overhead communications architecture whose functions are tai lored to the needs of highly available, extensible sys tems. This architecture is called the System
Communication Arch i tecture (SCA).
•
A very high speed message-oriented Computer
I nterconnect, cal led the CI bus
Digital
Tecbtzical Journal
No.
5
September
1987
7
lbe V AXcluster Concept: A n Overview of a Distributed System
•
An intelligent hardware interface to the CI bus, called the C I port, that implements part of the SCA in hardware
•
An intelligent, message-oriented mass storage controller that uses both the CI bus and the Cl port i nterface
This combined software and hardware archi tecture supports a high-performance communi cations structure for interconnecting high-perfor mance VAX systems. For low-end VAX CPUs, the
Local Area VAXcluster system has been developed to perm it workstations interconnected by means of the Ethernet to share a common file system, printers, and batch processing. Workstation users can derive the benefits of centra lized timesharing without sharing a CPU and without system man agement overhead. A Local Area VAXcluster sys tem is supported by software that emulates some of the CI functions, thus making the difference between CI-based and Ethernet-based VAXclus ters largely i nvisible to higher l evel software .
Local Area VAXcluster systems can be formed from and coexist with existing Ethernet networks without the need for special-purpose hardware .
This paper describes the communications hard ware developed for VAXcluster systems, the hard ware-software i nterface , the Local Area VAXclus ter system, and the structure of the distributed
VAXjVMS operating system. The developments described in this paper are part of Digital's VAX cluster product; there are, as of mid-1987 , approximately 6 , 0 0 0 VAXcluster and Local Area
VAXcl uster systems in operation .
VAXcluster Hardware Structure
The CI- based VAXcluster System
Figure
1 shows the topology of a typical CI-based
VAXcluster system . The components i nclude the
Cl bus, VAX hosts, Cl ports, and Hierarchical
Storage Controllers (HSC) for mass storage ( i . e . , disk and tape) . For high-reliabil i ty applications, a cluster must contain a minimu m of two VAX pro cessors and two mass storage controllers w ith dual-ported devices. The preferred method of attaching terminals is through a Local Area Trans port (LAT) server (not shown) , which allows a terminal to connect to any host in a VAXcluster system.
The Cl bus is a dual-path serial intercon nect with each path supporting a transfer rate of
70-megabits per second . The primary purpose of
CI PORT
VAX
Cl PORT
HSC
DISK
SYSTEM
VAX
Cl PORT
Cl PORT
VAX
Cl PORT
HSC
DISK
SYSTEM
STORAGE
CONTROLLERS
Figure
1
VAXcluster Hardware Topology the dual paths is tO provide redundancy i n the case of path failure; when both paths are avail able, they are usable concurrently. Each path is implemented in two coaxial cables; one for trans m itted and one for received signals. Baseband sig naling with Manchester encoding is employed .
Wh i le the C I bus is logically a bus, it is physi cally organized as a star topology. A central hub called the Star Coupler connects all of the nodes through radial CI paths of u p to 4 5 meters.
The current coupler is a passive device that sup ports a maximum of 1 6 nodes; node addresses are 8 bits, providing an archi tectural limit of
2 5 6 nodes.
The selection of a star topology was chosen over a conventional linear topology for several reasons. First, the efficiency of a seria l bus is related to the longest transi t t i me between nodes .
The star permits nodes tO be located within a
4 5 - meter radius (an area of about 6400 square meters) with a maximum node separation of
90 meter radius (an area of about 6 4 00 square meters) with a maximum node separation of
8
Digital Technicaljournal
No. 5
September
1987
90 meters. Typically, a linear bus threaded through 1 6 nodes in the same area would greatly exceed 90 meters. Second, the central coupler provides simple, electrically and mechanically safe addition and removal of nodes.
The CI port is responsible for arbitration , path selection , and data transmission . Arbitration uses carrier sense multiple access (CSMA) but is dif ferent from the arbitration used by the Ether net .14·15 Each CI port has a node-specific delay time. When wishing to transmit, a port waits until the CI bus is quiet and then waits its specific delay time. If the CI bus is sti ll quiet, the node has won its arbitration and may send i ts packet. This scheme gives priority to nodes with short delay times. To ensure fairness, nodes actu ally have two delay times- one relatively short and one relatively long. Under heavy loading, nodes alternate between short and long delays.
Thus the bus is contention driven under light loading and round robin under heavy loading.
Upon winning an arbitration , a port sends a data packet and waits for receipt of an acknowl edgment . If the data packet is correctly received , the receiving port immediately returns an acknowledgment packet without re-arbitrating the CI bus. This action is possible because the Cl port can generate an acknowledgment in less time than the smallest node-specific delay.
Retries are performed if the sending CI port does not receive an acknowledgment.
To distribute transmissions across both paths of the dual-path CI bus, the CI port maintains a path status table indicating which paths to each node are currently good or bad . Assu ming that both paths are marked good, the CI port chooses one randomly. This provides statistical load sharing and early detection of failures. Should repeated retries fail on a path, it is marked bad in the status table and the other path is tried.
The Ethernet-based VA.Xcluster System
Figure 2 shows an example of a Local Area VAX cluster system. The CI bus of Figure 1 has been replaced by an Ethernet, and the VAX hosts
(referred to as satel lite nodes) are MicroVAX computers and workstations. Satellite nodes may be diskless, in which case one or more VAX hosts act as storage servers, serving a function analogous to the HSC controllers in CI-based con figurations. One or more storage servers, cal led boot nodes, are responsible for load ing satellite nodes with the VMS operating system and for storing crash dumps from those nodes. Satellite nodes may use remote disks for process swapping and virtual memory backing storage .
The important difference between the Cl based and the Local Area VAXcluster systems is t hat the communication functions performed by the CI hardware are emu lated in the latter by soft ware within the VMS operating system. The Eth ernet is an industry-standard, 1 0-megabit per sec ond baseband local area network'
5 that uses the carrier sense multiple access with collision detection (CSMA/CD) technique for arbitration .
Unlike the CI bus, an Ethernet may be used to carry multiple protOcols simultaneously. (Note that this allows a cluster to share the Ethernet with other protocols, such as the lAT and DECnet protocols.)
A new Ethernet protOcol , which is an extension of SCA, was designed for Local Area VAXcluster system . Using this protocol , a VMS software com ponent emulates the Cl port interface, which is to say that the higher level software interface is identical to that of the Cl bus, but the Ethernet is used to carry data . This approach eliminated the need for any special hardware and allowed the software modifications needed to be mostly lim ited to a single VMS component.
Exactly the same approach was used for load ing the VMS system into satel lite nodes. Here, a special port emulator was developed to operate in the booting and system-initialization environ ment . This boot driver forms part of a vestigial
VMS environment whose function is to read , initialize, and start the VMS system i mage from the remote disk. These modules are themselves loaded by means of the Digital Network Architec ture maintenance operations protocol (MOP)
.16
The CI
Port
Architecture
Each VAXcluster host and mass storage controller connects either to the CI bus through a CI port or to the Ethernet by means of a standard Ethernet adapter. CI ports have been implemented for the
HSC50 and HSC70 mass storage controllers, and the VAX- 1 1 /7 5 0, 1 1 /780, 1 1 /7 8 2 , 1 1 /785 , and
VAX 8000 series hosts . Ethernet adapters have been implemented for all VAX processors. VAX
CI pons i mplement a common architecture, whose goals are to
•
Off load much of the communications over head typically performed by nodes in dis tributed systems
VAXcluster
Systems
Digital Technical journal
No.
5
September
1987
9
The VAXcluster Concept: A n Overview of a Distributed System
•
Provide a standard, message-oriented sofnvare i nterface for both interprocessor communica tion and device control
The design of the CI port is based on the needs of the VMS System Communications Architecture.
SCA is a software layer t hat provides efficient communications services to low- level distributed applications (e.g . , device drivers , file services, and network managers) . SCA supports three com mu nications services: datagrams, messages, and block data transfers . In a Local Area VA.,'(cluster system, the SCA functions performed by the CI port are performed by software in the port emula tor modul e .
SCA datagrams and messages are information u n i ts of less than 4,000 bytes sent over a connection. They differ only i n reliability . The delivery of datagrams is not guaranreed; they can be lost, duplicated, or delivered out of order. The deliv ery of messages is guaranteed, as is their order of arrival . Datagrams are used for status and infor mation messages whose loss is not critical, and by appl ications Jike the DECnet software that have their own high-level reliabil i ty protOcols. Mes sages are used , for example, to carry disk read and write requests.
To simplify buffer a llocation, hosts must agree on the maximum size of messages and data grams that they will transmit. VA.Xcluster hosts use standard sizes of
5 76 bytes for data grams and
112 bytes for messages.
To ensure t he del ivery of messages without duplication or loss, each CI port maintains a vir-
CPU
PORT
PORT
--------�--------------�------------�---------------r------------- ETHERNET
PORT
PORT
CPU
10
Figure 2 Local Area VAX cluster Topologv
Digital Technical journal
No. 5
September
I ')87
tual circuit with every other remote CI port. A virtual circuit descriptor table in each port indi cates the status of its port-to-port virtual circuits.
Included in each virtual circuit descriptor are sending and receiving sequence numbers. Each transmitted message carries a sequence number enabl ing duplicate packets to be discarded .
Block data is any contiguous data in a process' virtual address space . There is no size l imit except that i mposed by the physical memory constraints of the host. The CI port hardware is capable of copying block data directly from the process virtual memory on one node to the pro cess virtual memory on another node. For the Eth ernet, this function is performed in software by the port emulator.
The delivery of block data is guaranteed . The sending and receiving ports and the port emula tors cooperate in breaking up the transfer into data packets and ensuring that all packets are cor rectly transmitted, received, and placed in the appropriate destination buffer. Virtual circuit sequence numbers are used on the individual packets, as with messages. Thus the major differ ences between block data and messages are the size of the transfer, and the fact that block data need not be copied by the host operating system .
Block data transfers are used, for example, by disk subsystems and disk servers to move data associated with disk read and write requests.
CI Port Interface
The VAX CI port i nterface is shown in Figure 3 .
The interface consists of a set of seven queues: four command queues, a response queue, a data gram free queue, and a message free queue. The queues and queue headers are located in host memory. When the port is initialized , the host software loads a port register with the address of a descriptor for the queue headers.
Host software and the port communicate through queued command and response packets.
To issue a port command , the port driver software queues a command packet to one of the four com mand queues. These four queues accommodate four priority levels; servicing is FIFO within each queue.
An opcode within the packet specifies the command to be executed . The response queue is used by the port to enqueue incoming messages and datagrams, while the free queues are a source of empty packets for incoming messages and a sink for transmitted message packets.
For example, to send a datagram, software queues a SEND DATAGRAM packet onto one of
VAX
PORT
SOFTWARE
COMMAND
QUEUE
0
I I l
COMMAN D QU EUE
3
I I I
R ESPONSE QU EUE
I I I
I
DATAGRAM FREE QUEUE
I I I t
I
MESSAGE FREE QUEUE
I I I
t
BU FFER
DESCRIPTOR
TABLE
VAX MEMORY
Figure
3 The CJ Port Interface
Cl
PORT
VAXcluster
Systems
Digital Technical journal
No. 5
September
1987 l l
The VAXcluster Concept: An Overview of a Distributed System
OPCODE
FORWA R D L I N K
BACKWA R D L I N K
I
PORT
I
STATU S
DATAGRAM LENGTH
DATAGRAM TEXT
Figure 4 CI Port Command jacket the command queues. The packet contains an opcode field specifying SEND DATAGRAM , a port field with the destination port number, the data gram size , and the text of the datagram. The packet is doubly l inked through its first two fields. This structure is shown in Figure 4 .
If the host software needs confirmation when the packet is sent, it sets a response queue bit in the flags field. This bit causes the port to place the packet in the response queue and interrupts the host after the packet has been transmitted.
The response packet is identical to the SEN D
DATAGRAM packet, except that the status field indicates whether or not the send was successful.
Had the response queue flag bit been clear in t he
SEND DATAGRAM command (as it typically is), the port would instead place the transmitted command packer on the datagram free queue without causing a host interrupt.
Upon receiving a datagram , a CI port takes a packet from its datagram free queue . Should the queue be empty, the datagram is discarded . Oth erwise, the port constructs a DATAGRAM
RECEIVED packet that contains the datagram and the port number of the sending port. This packet is then queued on the response queue.
Messages operate in a similar fashion, except that they have a different opcode, and the mes sage buffers are dequeued from the message free queue . If the message free queue is empty when a message arrives, the port generates an error interrupt to the host. The high-level SCA flow control ensures that the message free queue does not become empty.
Block transfer operations are somewhat more complicated . Each port has a data structure calle<J a buffer descriptor table. Before perform ing a block transfer, host software creates a buffer descriptor that defines the virtual memory buffer to be used . The descriptor contains a pointer to the first VAX page table entry mapping t he virtu ally contiguous buffer. In addition, the descrip tor contains the offset (within the first page) of the first byte of the buffer, the length of the buffer, and a 16-bit key. The data structures for a block transfer are illustrated in Figure 5 .
Each buffer has a 32-bit name, consisting of a 16-bit buffer descriptor table index and the
16-bit buffer key. The key is used tO prevent dan gling references and is modified whenever a descriptor is released. To transfer block data , the initiating software must have the buffer names of the source and destination buffers. The buffer names are exchanged through a high level mes sage protocol . A host can cause data to be moved either ro another node (SEND DATA) or from another node (REQUEST DATA) . A SEND DATA or
REQUEST DATA command packet contains the names of both buffers and the length of the trans fer. In either case (send or request) , a single command packet causes the source and destina tion ports to perform the block transfer. When the last packet has been successfully received, the initiating port places a response packet on its response queue, indicating that the transfer is complete .
The goal of reducing VAX host interrupts is met through several strategies and mechanisms. First, the block transfer mechanism minimizes the number of interrupts necessary to transfer large amounts of data . Second, at t he sending port,
DATAGRAJVI SENT/MESSAGE SENT confirmation packets are typically generated only when a fail ure occurs . Third, a receiving port interrupts the
VAX host only when the port queues a received packet on an empty response queue . Thus when software dequeues a packet in response to an interrupt, it always checks for more packets before dismissing t he interrupt.
Port Emulation for the Ethernet
Figure 6 shows the relationship of the port emu lator to the VMS operating system functions that usc that emulator. For comparison, the CI port interface is a lso shown in this diagram . The port emulator implements the same functions as the
12
Digital Technical journal
No.
5 September J'J87
Cl BUS
COMM'""
QUEUE
I
1\
I � I I I
�
QU EUE LINKS
FLAGS
PORT NO.
I
OPCODE
I
STATUS
TRANS. SIZE
SOU RCE BUFFER NAME
DEST. BUFFER NAME
\ '"""'
I
DESCRIPTOR
Cl PORT
BLOCK TRANSFER
PORT COMMAND
HOST MEMORY
PAGE
TABLE
Figure
5
CI Port Block Data Memory Mapping
VMS
SYSTEM
COM M U N ICATION
SERVICES
ROUTINES
COM MAND
INTERFACE
TRANSPORT
PROTOCOL
DATALINK
CONTROL
C l BUS
PAGES
VAXcluster
Systems
ETHERNET
Figure 6 CI Port Emulation Using Ethernet
1
:�
Digital Technical journal
No. 5 September 1 987
The VAXcluster Concept: A n Overview of a
Distributed .�)'stem emulator implcmt:nts the same functions as the
CI port and its associated driver. The emulator also operates the SCA protocol across the Ether net and manages i ts interface with the Ethernet datalink driver. Thus the emulatOr is responsible for
•
•
The provision of a compatible command int er face tO the system communication services
(SCS) module
The operation of a transport protOcol that imi tates CI behavior
•
•
Node authentication and topology control functions
Propagation of Ethernet datagrams and data link control
The port emu latOr must deal with an underly ing datal ink layer whose characteristics are some what different than those of the CI bus. The Eth ernet datalink can transmit datagrams between
64 and I , 5 36 bytes in length in either a point-to point, multicast, or broadcast fashion . The Ether net provides neither automatic acknowledgment nor flow control , and Ethernet adapters do not handle either buffer segmentation or d i fferent message types. The CI functions of datagram transmission , sequenced messages, and block transfers must be implemented by the e mu lator and translated into requests that can be p ro cessed by the standard VMS Ethernet device drivers .
Port emu lation can be viewed conceptually as three separate layers . The highest layer provides a command interface for the higher level SCS rou tines . That interface is compatible with that used for CI ports . This layer is also responsible for tht: fragmentation and re-assembly of block transfer buffers that are larger than the maximum Ether net message size.
The transport layer provides a sequenced nlCs sage and datagram service ro the correspond i ng layer in the remote node. Its hand l ing of data grams amounts to l i trle more than a pass-through function ; the hand! ing of sequenced messages and block transfers, however, is more complex. ln the latter case, the transport layer must t:nsure that messages are transmi tted and received in the correct order, ensure that acknowledgments arc sent and received, and retransmit messages that have been lost. The transport layer operato a simp k pipeline flow control scheme that a l l ows a fixed window of unacknowledged messages.
Acknowledgments can be " piggybacked" on rt:turning messages.
Last. the datali n k control layer is responsible for passi ng messages between the Ethernet device clrivns and the transport layer and control of the
Ethernet data link service . The datal ink control layer also maintains a record of t he cluster's topol ogy by exchanging mu lticast messages with other cl uster members .
Below the port emulator module is tht: stan dard VMS Ethernet device driver, which can also be ust:cl simultaneously by other applications l i ke the DECnet, LAT. and ISO transport protocols.
These protocol<> are mu ltiplexed and demu lti plcxed by the Ethernet device driver using the
Ethernet standard p rotocol type.
The
C l port emulation function for the Local
Area VAXcluster system has a h igher system ovnhcad than the equivalent CI connection si nct: the operations i nvolved are performed by tht: host VAX processor. Since the Ethernet has lower bandwidth and longer response ti mes. however, the demand for host system resources is moderated . The Local Area VAXcluster per formance is acceptable for typical customer work loads in which most nodes are single-user workstations. The CPU t i me overheads are most noticeable on nodes that serve disks to multiple users; those nodes are typical ly dedicated proct:ssors .
Mass Storage Control
·rhe move from control- and status register activatt:cl storage devices to message-oriented storage devices offers several advantages:
•
•
Sharing is simplified si nce several hosts can queue messages to a single controller. In add i tion , device control messages can be transmit ted to and executed by hosts with local disks.
Extension to new devices is easier. In contrast to conwntional systems where there is a differ ent drivn for every type of d isk and disk i nterface, a single d isk class driver si mply bui l ds mt:ssage packets and transmits them using a communications i nterface. The disk class driver is independent of drive specifics
( e . g , cy l inders and sectors) . New d isk and tape devices ancl controllers can be added with
I i ttk or no modi fication to the host software .
•
Performance is improve d . The controller can mainta i n a q u eue of requests from mu ltiple hosts and can opt i m i ze disk performance in rt:a l time. The control ler can also hand le error rt:covt:ry and bad-block replacement.
1 4
Digital Technicaljournal
No. 5 September 1 ')87
The HSC fam i ly, shown i n Figu re 1 , is a CI based controller for both disks and tapes . A single
HSC70 control ler can hand le up to 32 disk drives. Multiple HSC control lers with dual ported d isks provide redundancy in case of fail ures. Further redundancy can be provided by grouping disk volumes rogether in shadow sets to form a single virtual vol u me in which a l l mem bers contain exactly the same data . If one mem ber of the shadow set fai ls, the virtual disk vol ume continues t O b e available.
The protocol interpreted by the HSC controller is called t he Mass Storage Control Protocol
(MSCP) , which provides accl:ss to mass storagl" vol uml:s at the logical block kvd . The MSCP
VAX 1
PROCESS
FILE
MGMT.
RECORD
MGMT.
PROCESS
FILE
MGMT.
RECORD
MGMT. model sl:paratl"S the tlow of control and status information from the tlow of data . This d istinc tion has been used in other systems to achieve efficient fi le access
1 7 and corresponds to the
CI port's message and block data mechanisms; mes sages arc used for device control commands whi l e block transfers are used for data .
The same control protocol is used tO provide clusterwide access tO
CI-based controllers l i ke the HSC devices, and tO disks connected directly to a VAX processor (See Figure 7 ) . In a Loca l Area
V�'Cclustcr system , all mass storage is connected directly to the boot node and to zero or more other storage server nodes. Messages arc routed from the d isk class driver in the requesting node
VAX 2
VAXcluster
Systems
DISK
CLASS
DR IVER
LOCK
MANAGER
DISK
CLASS
DRIVER
LOCK
MANAGER
MSCP
DISK
SERVER
DISK
PORT
DRIVER
LOCAL
DISKS
SCA SOFTWARE
SCA SOFTWA R E
----------�--------------�--------------�----------------------------
Cl BUS
OR
ETH ERNET
HSC
(CI BUS ONLY)
SCA SOFTWARE
DISKS
Figure
7
VAXcluster
Software Structure
Digital Technical journal
No.
5 September 1 987
1 ')
The VAXcluster Concept: A n Overview of a
Distributed System to an MSCP server on the node with the local disk. This server then parses the MSCP message, issues requests to its disk, and initiates the block transfer through i ts SCA interface . Thus in either a CI-based or a Local Area VAXcluster system , all locally attached disks can be made transparently available to all other VAX hosts in the cluster.
VAXcluster Software
From a user's point of view, a VAXcluster system is a set of nodes cooperating through the VAX/
VMS distributed operating system software to provide sharing of resources among users on all nodes . Shared resources include certain devices, fi les, records within files, and system batch and print queues. Typically, user account and pass word information resides i n a single file shared by all cluster nodes. A user obtains the same environment (files, default directory, privileges, etc.) regardless of the node to which he or she is logged into. In many respects, the VAXcluster system " feels" like a single system to the user.
This sense of a single system results from the fact that the VAXcluster system is symmetrical with respect to the participating VAX processors .
In other words, there is no specialization of func tion designed into the sofrware (although an installation may choose to configure certain
CPUs differently according tO the special needs of that installation) . The VMS and VAXcluster file system architecture is based on the concept of clusterwide and uniform logical block access to the mass storage managed by a distributed file system . This concept contrasts with fil e server based distributed systems.
Figure 7 shows an example of a small VAXclus ter system and some of its major sofrware compo nents. Note that the operation of the VMS soft ware in the VAXcluster environment is exactly the same for both Local Area and CI-based VAX cluster systems. The diagram shows an underly ing interconnect that may be either the cr bus or the Ethernet, both of which use the port interface methods described above. HSC disk control l ers connect only to the Cl bus.
At the highest leve l , multiple user processes on each node execute in separate address spaces .
File and record management services are imple mented as procedure-based code within each process . The fil e and record services rely on lower level primitives, such as the lock man ager18 and disk class driver. The lock manager is the foundation of all resource sharing in both clustered and single-node VMS systems. I t pro vides services for naming, locking, and unlocking clusterwide resources. The disk class driver, mentioned earlier, uses the MSCP to communi cate with disk servers. The disk class driver runs in both clustered and nonclustered environments and contains no knowledge of the VAXcluster configuration . SCA software below the driver is responsible for routing driver messages to the correct device control ler.
A distributed connection manager is responsi ble for coordinating the cluster. Connection managers on all cluster nodes collectively decide upon cluster membership, which varies as nodes leave and join the cluster. Connection managers recognize recoverable failures in remote nodes; they a lso provide data transfer services that han dle such failures transparent to higher software levels.
Form ing a Cluster
A VAXcluster system is formed when a suffi cient set of VAX nodes and mass stOrage resources becomes available . New nodes may boot and join the cluster, and members may fai l or shut down and leave the cluster. When a node leaves or joins, the process of reforming the cluster is cal led a cluster transition . Cluster transitions are managed by the connection managers.
In an operating cluster, each connection man ager has a list of all member nodes. The l ist must be agreed upon by all members. A single node can be a member of only one VAXcluster system ; in particular, the same resource (such as a disk controller) cannot be shared by two clusters or the integrity of the resources coul d not be guar anteed. Therefore , connection managers must prevent the partitioning of a cluster into two or more clusters attempting co share the same resources.
To prevent partitioning, the VMS system uses a quorum voting scheme . Each cl uster node con tributes a number of votes, and the connection managers dynamically compute the total votes of all members. The connection managers also maintain a quorum value. As transitions occur, the cluster continues to run as 1long as the total number of votes present equals or exceeds the quorum . Should the total number of votes fal l below the quorum , the connection managers wil l suspend VAXcluscer activity. When a node joins and brings the total votes up to the quorum , clus ter activity will resume .
1 6
Digital Technicaljournal
No. 5
Septem ber
1987
A c l uster member may have a recoverable error in its communications. Such an error leaves the node's memory i ntact and al lows the operating system to continue running after the error cond i tion has disappeared . These errors can cause ter mination of a virtual circuit and a corresponding loss in communication . When cluster members detect the loss of com mu n ication with a node, they wait for a short period (specified by the sys tem manager) for the fai l i ng member to re-estab l ish contact. If the fai li ng member recovers within this period, it rejoins the cluster. Users may experience a brief interruption of service when this happens. If the fai ling member docs not recover in time , the surviving members remove the fai led node from the cluster and con tinue operating (assuming sufficient votes arc present) . A node that recovers after it has been removed from the cluster is told to re-boot by the connection managers.
Shared Files
The VAXcluster system provides a clusterwide shared fi le system to i ts users. 1 9 Cluster accessi ble files can exist on CI-based disk controllers or on disks local to any of the cluster nodes. Each cl uster disk has a unique and location-indepen dent name . A complete cluster fi le name includes the disk device name , the d irectory name, and the fi le name. Using the device name for a fi le, the cluster software can locate the node (either a
CPU or a d isk controller) on which the fi le resides .
Cluster fi le activity requ i res synchronization ; exclusive-write fi le opens, coord i nation of fde system data structures, and management of fi le system caches arc a few examples. However, despite the fact that files can be shared cluster wide, the fi le management services are largely unaware of whether they are executing in a clus tered environment. These fi le managers synchro nize through the VMS lock manager, described later. The lock manager hand les the locking and unlocking of resources across the cluster. At the l eve! of the fi le manager, then, cluster fi le sharing i·s simi Jar to single-node file sharing. lower levels hand le the clustcrwidc synchroniza tion and routing of physical-level disk requests to the correct device .
Distributed Lock Manager
As previously described , the VMS lock manager is the basis for clusterwide synchronization . Severa l goa ls influenced the design of the lock manager for a distributed environment. First , programs using the lock manager must run in both single node and cluster configurations. Second , lock services must be efficient to support system- level software that makes frequent short-duration accesses. Therefore , i n a VAXcluster system , the lock manager must minimize the number of SCA messages needed to manage locks. In a single node configuration , the ,Jock manager must rec ognize the simpler environment and bypass any c l uster-specific overhead. Finally, the lock man ager must recover from fai lu res of nodes holding l ocks so that surviving nodes can continue to access shared data i n a consistent manner.
The VMS lock manager services al low cooper ating processes to define shared resources and synchron ize access to those resources. A resource can be any object an application cares to define.
Each resource has a user-defined name by which it is referenced . The lock manager provides basic synchronization services to request and release locks. Each lock request specifics a locking mode, such as exclusive access, protected read, concurrent read, and concurrent write . If a pro cess requests a lock that is incompatible with existing locks, the request is queued unt i l the resource becomes avai lable. In many applica tions, resources may be subdivided into a resource tree, as i l lustrated in Figure 8 .
DISK VOL U M E
FILE
1
F I L E
2
FILE
3
�
RECORD
1
RECORD
2
Figure 8 VAXcluster Locking Structure
In this example, the resource Disk Vol u me contains resources Fi le
File
3 change ( c .g . , a leaf) can do so whi lc stiH per mitting concurrent access to other parts of the tree. 20
I through Fi le 3 ; resource contains resources Record
1 ,
Record 2 , and so on . The first locking request for a resource can specify the parent of that resource, thereby defin ing its relationship in a tree. A process making several global changes can hold a h igh-level l ock
(e.g . , the root) a nd can make them all very effi cient ly. A process making a smal l , low-level
VAXcluster
Systems
Digital Technical journal
No. 5
September
I
987
1 7
The VAXcluster Concept: An Overview of a
Distributed System
The lock manager's implementation is in tended to d istribute the overhead of lock man agement throughout the c luster while s t i l l mini mizing the internode traffic needed to perform l ock services. The database is therefore divided into two parts: the resource lock descriptions, and the resource lock directory syste m , both of which arc d istributed. Each resource has a master node responsible for granting locks on the resource ; the master mainta i ns a l ist of granted locks and a queue of waiting requests for that resource . The master for a l l operations for a sin gle tree is the node on which the l ock request for the root was made . While the master mainta i ns the l ock data for i ts resource tree , any node hold i ng a lock on a resource mastered by another node keeps its own copy of the resource and lock descri ptions.
The second part of the database , the resource d irectory system , maps a resource name i nto the name of the master node for that resource . The directory database is d istributed among nodes wil l ing to share this overhead . Given a resource name, a node can trivia ll y compute the responsi ble directory as a function of the name string a nd the number of d irectory nodes.
To l ock a resource in a YA.Xcluster system , the lock manager sends a lock req uest message through the SCA to the d i rectory for t he resource .
The directory responds i n one o f three ways:
1 .
If l ocated on the master node for the
� ource
, the d irectory performs the lock
J,y
6 request and sends a confirmation response to the requesting system .
2 . I f the directory i s not o n the master node hut finds the resource defined , it returns a response containing the identity of the mas ter node.
3.
I f the d i rectOry finds the resource to be undefi ned, it returns a response tel ling the requesting node to master the resource i tsel f.
In the best cases
( l and 5 ) , two messages arc required to request a lock; case
2 takes four mes sages. An u n l ock is executed with one message . lf the lock request is for a subrcsource in a resource tree, the requesting process wi l l e ither be located on the master node ( i . e . , the request is local) or will know who the master for its parent is, allowing it to bypass the directory looku p . In a l I cases the number of messages required is independent of the number of nodes in the VAX cluster system .
I n addition t o standard .locking services, the lock manager supports data caching in a dis tri buted environment. Depend ing on the fre quency of modifications, caching of shared data in a distributed system can substantially reduce the I/0 and commun ications workload .
A I 6-byte block of information , cal led a va lue block, can be associated with a resource when the resource is defined to the lock manager. The val u e in the value block can he mod i fied by a process releasing a lock on the resource and can be read by a process when it acq u i res ownership.
Thus this information can be passed a long with the resource ownership.
In the case of a fil e buffer, for example, a ver sion nu mber is mainta i ned i n the value block.
When caching a buffer, a process saves the cur rent version number. To mod i fy the buffer , the process obtains an exclusive l ock and receives the current version number. If the current ver sion number equals the version number of the cached data , the cache is val i d . Several u pdates can then be made on the cached data before it is written back tO disk. When the modified data is written , the process incre ments the version num ber and rel eases its lock.
Another mechanism used in buffer caching is a software i nterrupt mechanis m . When requesting an exc lusive lock, a process can specify that it shou ld be notified if a nother lock request on the resource is forced ro block. A process can then hold a modified copy of the data without writing it back. When another process wants access, the owner wri tes the modified data and releases irs l ock.
In the case of cl uster transi tions (e.g . , fai l u re of a node ) , the connection manager notifies the lock manager that a transition has started . Each lock manager performs recovery action , and a l l lock managers must complete this activity before cl uster operation can continue.
As the first step i n hand l i ng transitions, a lock manager dcall ocates a l l locks acquired on beha lf of other systems. Only local l ock and resource information is retai ned . Temporari ly, t here are no re-source masters or d i rectory nodes . In the sec ond step, each lock manager re-acqu i res each lock it had when the cluster transition began .
This step establishes new di rectory nodes based on a new set of e ligible c luster members and rear ranges the assignment of master nodes. If a node
1 8
Digital Technicaljom-nal
No. 5
September
1987
has left the cluster, the net result is to re lease locks held by that node. If no node has left the cluster but nodes have joined , this recovery is not necessary from an integrity point of view. It is performed , however , to keep the directory and lock mastering overhead evenly distributed .
Some resources, depending on how they are modi fied. might be left i n an inconsistent state by a c luster transition. To ensur e the proper han d l i ng of such resou rces, users can defi ne a class of l ocks that are nor released on a c luster transi tion . In this case a special process can search for such locks and perform needed consistency checks before releasi ng them .
Batch and Print Services
In a VA.Xcluster system, users may either submit a batch job to a qu eue on a particu lar node (not necessarily their own node) , or submit a job to a c lusterwide batch queue . Jobs on the c lusterwide queue arc routed to queues attached to specific nodes for execution . The algorithm for assigning jobs to specific nodes is a simple one based on the ratio of executing jobs compared to the job l imit of the queue.
The management of batch jobs i s the responsi bil ity of a VMS process cal led the job control ler.
Each VMS node runs a job controller process, which acqu ires work from one or more batch queues. Batch queues are stored in a d isk file that may be shared by a l l nodes. The synchron i zation of queue man i pulation is handled with lock man ager services.
Pri nt queues are similar tO batch queues. Users may queue a request for a specific printer ( not necessari ly physically attached to their own node) or may let the operating system choose an avai lable printer from those i n the cluster .
Both batch and print jobs can be declared restartable. If a node fails, restartable jobs are either requeued to complete on another node in the c luster or executed w hen the fa iled node reboots (for jobs that must execute on a speci fic node) .
DECnet Communications
Each member of a VA.Xcluster system can also participate i n a DECoct network as an i ndividual node. Simu ltaneously, the cl uster as a whole may participate in the network as a s i ngle node. The cl uster's system manager may select an additiona l
DECnet node name and address, known as the c luster's alias, ro be assigned to the cl uster. DEC net connections originating from a cluster mem ber can be made to appear as if they came from the a lias node, regardl ess of the true originator.
Connections add ressed to the al ias w i l l be d irected to any cluster member that has declared itsel f wi l l i ng ro receive them . This concept i s particu larly usefu l for send ing and receiving net work mai I . Al l mail sent from the cluster wi l l appear to have come from a si ogle node . AJ I repl ies w i l l be delivered to the cluster's mail fi les even when the node from which the first message was sent is u navai lable (provided that the disk rema i ns available) .
The VA.Xcl uster DECnet al ias address requ i res the presence of at least one routing node i n the cluster. DECnet routing nodes mai ntai n rabies describi ng the topology of the network and com mun icate this information to other nodes. The existence of the cluster's alias address is thus propagated in control messages tO other nodes i n the network. Although the a lias node does not actually exist, a path to it v i a the cluster's router is apparent. The router mai ntains a table of con nections ro the a l i as node by means of the dis tributed lock manager. When a connect request for the a l i as arrives at the router, it passes the request tO another node in the cl uster, distribut ing the connections in a round -robin fashion .
Connect requests originating from the cl uster members arc simply set up as if they came from the a l ias.
Terminal Support
The optimum method for connecting users' ter m i nals to a VA.Xc luster system is through the LAT server. Termi nals are connt:cted to the LAT server, which is attached to the VAX systems by the Eth ernet. In a Loca l Area VA.Xcl ustcr system , this connection can be the same Ethernet used to i nterconnect the members of the cluster . Users command tht: LAT server to connect them e ither to a specific node or ro any node i n the cluster.
The case of sw itching nodes leads users to find and usc the least busy node . The server a lso a l l ows usns to quickly move from a fai led nodt: tO one that is sti l l ru nning. If the LAT server is d irected to select a node , it attempts tO fi nd the least busy one . Its choice is based on node CPU type (a mt:asurc of proct:ssi ng power) and recen t i d l e time .
VAXcluster
Systems
Digital Technical journal
No.
5 September J 987
1 9
The VAXcluster Concept: An Overview of a Distributed
Syst(!m
Performance
Performance measurements using a CI-baseu
VAXclusrer system of two VAX- 1 1 /780 systems have shown i t is possible to achieve 3 ,000 mes sage rou nd-trips per second
2 1
A rou nd-tri p i s defined a s the transmission of a message and the receipt of its acknowledgment from the remote system . This performance provides a basis for efficient execution of h igher l evel d istributed services, such as the VMS distributed lock man ager and the MSCP logica l block service used for access to mass storage . The performance characteristics of CI-based VAXcl uster systems vary almost l i nearly in relation to the n u mber of CI nodes i n t he system . From this it can he concl uded that the u nderlyi ng communications archi tecture upon which the VAXcluster system is based sca les wel l w i th an i ncreasing nwn ber of nodes . Measurements with up to twelve
VAX- l l j780 nodes showed nearly l i near perfor mance in cluster round trips per second .
The performance characterist ics of a Local Area
VAXcluster system are somewhat d i fferent for the fol lowing reasons:
•
The i nterconnect speed is l i m ited to I 0 mega bits per second , as opposed to 70 megabits per second for the CI bus.
•
The delay ( i . e . , latency) for message round trips in the Ethernet network is somewhat greater.
Because VMS VAXc luster systems attac hed to the Ethernet are optim ized as single-user work stations, the l i mits of throughput and latency do not present a problem . Wor kload stud ies have shown that the l i m i ti ng factor i n Local Area VAX cluster performance i s the rate at which the boot node can service the sate l l i tes' mass storage 1/0 requests . These studies fu rther indicate that this l imit in turn depends upon the CPU speed of the boot node wh i le executing both the C I porr emu l ation code and the MSCP server code. For a fast
VAX system (e .g. , a VAX 8700 CPU) , the next l i m it is imposed by the through put of the Ether net adapter used by the boot node. The fi nal l im i t to be encou n tered is the saturation of the
Ethernet network i tself. This l i m i t is reached at approxi mately 1 00 typical VMS 1/0 requests per second and is largel y i ndependent of the n u m ber of sate l l i te and boot nodes accommodated by the network. Note that the factors l i miting the n u m ber and size o f Loca l Area VAXcl uster systems that can be sustained by a single Ethernet segment is heavi ly dependent upon the nature of the app l i cations being ru n .
Summary
A principal goal of VAXcluster systems was the development of an ava ilable and extensi ble mu l ticomputer system bui It from standard processors and a genera l -purpose operating system . Much was ga ined by the joi n t design of distribured sofl warc, com mu nications protocols, and hardware a i med to meet this goa l . For example:
•
The C l interconnect su pports t he fast message transfer needed by the system software.
•
The CI port i m p lements many of the functions needed by the SCA software .
•
The HSC conrroJiers, with the i r message-pro tocol and request-queuing opti m i zation logic, support a l a rge pool of d isks for multiple hosts .
Designing hardware and software together a l l ows for system-level trade-offs; the software in terface and protocols can be tuned to t he hard ware devices.
An i m portant simpl i fying aspect of the VAX cl uster design is the usc of a d istr i buted lock manager for resource sync hron ization . I n this way, higher level services such as the fi le system do nor req u i re special code to hand le sharing i n a distributed e nvironment . However, the perfor mance of the lock manager becomes a crucial fac tor. The performance of the distributed lock manager has been attacked with the design of a locking prorocol requiri ng a fixed n u m ber of messages, independent of the nu mber of cooper ating nodes.
The system design of the original VAXc luster i mplementation a lso a l lowed i ts straightforward m igration to the Ethernet without the need for extensive hardware and software modification .
The Loca l Area VAXcJuster prod uct a l lows work station users ro e njoy the benefits of a large, cen tra l l y managed timesharing system on the i r indi vidual office system withour having to dea l with the various system management tasks.
Fi nally, we believe that performance measure ments show the extent to which t he VAXcluster system has succeeded in i m p lementing an effi cient com mun ications architecture that is appli cable to both a high-speed ded icated LAN (the
CJ bus) and a general -purpose shared LAN (the
20
Digital Technical jounwl
No. 5 Septem ber J 'J87
Ethernet) . This feat is particu larly impressive when consideri ng that t he VMS software is a large , general -purpose operati ng syste m.
Acknowledgments
VAXcluster systems are the result of work done by many ind ividuals i n severa l engineeri ng groups at
D igita l Equipment Corporat ion . We would partic ularly l i ke to acknowledge the contributions of
Richard
I.
Hustvedt to the VAXcluster design .
References
1 .
G . Almes et a l . , "The EDEN System : A Tech nical Review ,"
IEEE Transactio ns on Soft ware Engin eering SE- I f
(January 1 9 8'5 ) :
4 3 -'59
2.
Apollo Domain A rchitecture
( North Bil l er ica : Apo l lo Computer Corporation , 1 9 8 1 ) .
3 . A. Brownbirdge, A. Marsha l l . and A. Rande l l ,
"The Newcastle Connection or UNIXES of the World Unite ' , "
Software - Practical
Experiments 1 2
( 1 98 2 ) 1 1 4 7- 1 1 62 .
4 . G . Popek et a l . , .. LOCUS: A Network Trans parent, H igh Rel iabi l i ty Distributed Sys tem ,"
Proceedings of the 8th Symposium on Operating Systems Principles, A CM
( 1 9 8 1 ) : 1 69- 1 7 7 .
5 . G . FieiJ and and D . Rodgers, " 3 2-bit Com puter System Shares Load Equa l ly Among Up tO I 2 Processors,"
Electrical Design
(Sep tember 1 9 84 ) : 1 5 3- 1 68 .
6 . K . Hwang and F. Briggs ,
Computer Architec ture a nd Parallel Processing ,
( New York :
McG raw-H i l l , 1 984 ) .
7 . M . Satya narayanan,
Multiprocessors: A Com parative Study ,
(Engl ewood Cliffs : Pren tice-Hal l . 1 9 80) .
H . W .
Strecker. "VAX- I 1 /780 : A Vi rtua l
Address Extension to the DEC PDP- 1 1 Fam i l y,"
Proceedings of A FIPS NCC
( 1 978 ) :
967-980 .
9 . J. Bart lett, "A Nonstop Kerne l , "
Proceedings of the 8th Symposium on Operating -�)s tems Principles, A CM
( 1 9 8 1 ) : 2 2 - 29
1 0 . A . Borg, ). Baumbach, and S. G lazer, "A Mes sage System Supporti ng Fault Tolerance."
Proceedings of the 9th Symposium on
Operating .�ystems Principles, A CM
( 1 98:) ) : 90-99.
1 1 . D. Katsuki et al . . " PLURI BUS An Opera tional Fault-tOlerant Multiprocessor, "
Pro ceedings of the IEEE 66
1 1 4 6- 1 1 '59 .
(October 1 97 8 ) :
1 2 . .1 - Katzman, "The Tandem 1 6 : A Fault-toler ant Computing System ,"
Computer Struc tures: Principles and Exa mples , ed. D.
Siewiorck (New York: McGraw-H i l l , I 98 2 ) .
1 3 . M .
Fox and j . Ywoskus . "Local Area VAXclus ter Systems,"
Digital Techn ical journal
(September 1 9 87, this issue ) : '56-68 .
1 4 . R . Metca lfe and D . Boggs. "Ethe rnet: Dis tributed Packet Switching for Loca l Com puter Networks,"
Com munications of the
A CM / 9 (July 1 97 6 ) : 3 9 '5 - 4 04 .
1 '5 .
The l:."thernet: A Local A rea Network, Data
Link Layer and Physical Layer Specifica tion, Version 2. 0
( Digita l Equi pment Cor porat ion , lnte l Corporation, and Xerox Cor porat ion , Order No. AA- K75 9B-TK. 1 98 2 ) .
1 6 .
DECnet Digital Network Architecture
(Phase IV) Maintenance Operations Func tional Specification
( Bed ford : Digital
Equ i pment Corporation, Order No.
AA
X4 ,16A-TK, 1 98 3 ) .
1 7 . D . Cheriton and W . Zwaenepoel , "The Dis tri hu ted V Keene I and 1 ts Performance for
Diskl ess Workstations,"
Proceedings of the
9th Symposiu m on Operating Systems
Principles, A CM
( 1 98 3 ) : I 2 9- 1 4 0 .
1 8 .
\Xl
Snaman. Jr. and D. Thie l "The VAXjYMS
Distributed Lock Manager,"
Digital Techni cal journal
2 9 - 4 4
(September 1 9 87, this issue) :
1 9 . A. Goldstein, "The Design and lmplementa r ion of a Distributed F i l e System , "
Digital
Technical journal
(September I 987 , this issue ) : 4 '5-5 5 .
2 0 . ] . Gray e t a l . . " Granu larity of Locks and
Degrees of Consistency in a Shared Data
Base . "
Modelling in Data Base Ma nage ment Systems , ed . G . Ni jssen (Amsterda m:
North Hol land , 1 9 76) .
2 I .
1
Kronenberg. H . Levy, and W . Strecker,
"VAXclusters: A Closely-Coupled Dis tributed System . "
A CM Transactions on
Computer Systems ,
1 986 ) : 1 30- 1 4 6 . vol . 4 . no . 2 ( May
VAXcluster
Systems
Digital Technical journal
No 5 Septemhl'r 1 ')8 7
2 1
DarrellJ. Duffy
I
The System Communication
Architecture
The System Communication Architecture defines how data traffic is han dled among host systems and their disk systems over the CI interconnect in a VAXcluster configuration. Low CPU overhead was a key design goal. The
SCA supports the management of cluster configurations, buffers, and con nections. It also supports directory services, datagram and sequenced message services, and named-buffer transfer services. The SCA can be extended to connections between hosts and locally attached storage con trollers, and to Local Area VAXcluster systems, which use the Ethernet.
Each CI port is capable of sustaining about two megabytes per second of bandwidth with minimal overhead required from a CPU.
The System Communication Architecture (SCA) defines the network architecture for VAXd ustcr systems , much l i ke the Digital Network Architec ture ( DNA) defines the network protocols for
Digita l 's wide area networks. 1
In 1 98 1 , as the Computer I nterconnect (Cl) hardware was being developed , i t became clear that some type of network archi tecture was needed to bind the CI subsystems together. This architecture required a relatively simple struc ture so that litrle overhead wou ld be needed in either the VAX host computers or the H ierarch i cal Storage Controllers (HSC) . Many o f the sys tem processes within the systems and contro l l ers wou ld have to communicate i n , at that time. unforeseen ways. Therefore , the SCA architecture had to support all the features and performance of the CI hardware so they cou ld be used by the system processes.
The Cl lnterconnect
The CI i nterconnect provides the foll owing basic scrvices:2
•
Sendi ng datagrams, which arc not guaranteed against loss and duplication
•
Sending sequenced messages, which are guar anteed aga i nst loss and dupl ication (If an error occu rs, the sendi ng node on the CI i n tercon nect wi l l be noti fied . )
•
Named-buffer transfers, which are potentially large c.Jata transfers between process buffers in virtual memory (These transfers arc also guar antcccl against l oss and duplication . )
These services are very useful to the operating system software when VA.Xc luster and other dis tributed systems are bui l t . However, in the form that the C I port provided those services, they cou ld not be shared conve n iently by the many parts of the operating system need ing them.
The SCA arch i tecture provides a simple and efficient means for the various parts of the operat ing system and the d isk-control ler software to use these services.
SCA Goals
SCA was developed from the beginning with the fol l ow i ng set of goals:
•
To provide a high-performance means of accessing and d i recting mass-stOrage control lers . and of transferring data
•
To faci l i tate access to and sharing of a l l the capabili ties of the CI ports among many pro cesses within the operating systems of the host computers
•
'l(> pro v ide a way for each system on the C l interconnect ( e . g .. V�X host systems. d isk and rape con trol lers) to obtain configuration i n for-
2 2
Digital T(•chnical journal
No.
5 Se{Jielllber I 'JH�
marion about every other system and which functions each system performs
•
To establish a means of binding together sys tem appl ications (SYSAPs) in two different systems over the CI interconnect so that the
SYSAPs can com municate using the i r names
SYSAPs arc fu nctions within the operating systems of hosts and within the firmware of disk and tape comrol lers. I n host systems, those functions include disk and tape class drivers, DECnet soft ware, and the VAXcluster connection manager, among others . '
I n single computer systems, command status registers are used to d i rect the mass-storage con trollers and other devices. In YAXcluster systems, however, the SCA network archi tecture wou ld now direct the traffic between host systems and disk systems One imporram design goal of SCA was to n1ake i t operate as efficiently as possible. that is, with low overhead on the systems.
SCA Services
The SCA architecture supports the performance of six different functions.
1.
Cluster configuration management
2. Buffer management
3.
Connection management
4 . Directory services
5.
Datagram and sequenced- message services
6. Named-buffer transfer services
The fo.llowing sections descri be each of these functions and show how they i nteroperate to provide a coherent scheme for system communi cation .
Cluster Configuration Management
A node on the CI interconnect is either a VAX computer system or an HSC control ler su pporr ing disc or tape devices. Within the cluster, a node cannot communicate with another node until it has established that node's location on the
CI i nterconnect. At present, 1 6 nodes is the maxi mum nu mber the C I interconilect can support , a lthough the architecture can support 2 2 4 . Since this current number is smal l , pol ling is an effi cient method for each node tO determi ne which of the potential nodes are present. There is an
" instance " of the SCA software within each of the hardware components connected to the C I inter connect . Using the ID requestjresponse feature of the Cl ports, SCA software period ically pol ls each of the other nodes on the CI and keeps a l ist of the active members in the hardware cluster.
Using the information in this list, the SCA soft ware keeps a port-to-port virtual circuit open to every other node on the interconnect.
SCA software opens this port-to-port virtual cir cuit by using a series of messages, ca l l ed a hand shake, between itself and another SCA software instance in a partner node. The handshake allows the two SCA instances to first synchronize and then exchange i nformation . At the end of the handshake each node w i l l di rect its local CI port to enable the virrual circu it state with the other node's Cl port . This enabl i ng allows the guaran teed excha nge of sequenced messages and named -buffer transfers between the two pons.
The information exchanged in the handshake gives to each node the software type and SCA ver sion running on the other node. That a l lows nodes with different SCA versions to interoperate .
Other information , such as the t i me of day and the ti me the node last booted, is also exchanged.
A node with multiple Cl ports will use all its ports to form port-to-port virtual circuits tO all the other remote nodes . Each node will stare informat ion about each of the remote nodes in a system block for that remote node . Each port-tO port virrual circuit is cal led a path. The informa tion blocks representing these paths, called path blocks. are chained together to the system block for a particular remote node. ln that way, SCA can maintain the exact relationships among the paths and nodes .
The tOtal number of paths between two nodes is equal to the nu mber of Cl ports on the local node ti mes the nu mber of Cl pons on the remote node. SYSAPs i n both the local and remote nodes can determine the topology of the CI intercon nect by making spec ial calls to SCA software . Fig ure 1 depicts an exa mple of the relations hip between system blocks and path blocks for a net work.
Buffer Management
One of SCA's most important properties is i ts close control over how the communications buffers arc used within the nodes. This control is i mportant because node activity norma l ly occurs at very high data rates. The buffers cou ld be
VAXcluster
Systems
Digital Technical journal
No. 5
September
1 ')87
2 :1
The System Communication Architecture
SYSTEM
BLOCK
A r--
PATH
BLOCK
--..
PATH
BLOCK
-
PATH
BLOCK
!
SYSTEM
BLOCK B
--
PATH
BLOCK
1----
PATH
BLOCK l
SYSTEM
BLOCK C
--
PATH
BLOCK
--
PATH
BLOCK r---
PATH
BLOCK
Figure
1
Connections between .�ystem and Path Blocks qu ickly overrun if data transmission were not strict ly conrrolkd from t he source . Recovery from buffer exhaustion is not a ra pid process.
During periods of high load within the node, these delays yield further de lays and t hus i ncrease the requirements for buffering.
SCA softwa re conrrols the buffers for two types of traffic : SCA control messages, and SYSAP data messages. SCA control messages are used ro estab l ish and re move SYSAP-to-SYSAP connections and to control bu ffe r usage on t hose con nections . The
SCA control- message protocol i s structu red so as to s i m p l i fy the control of buffer usage .
Control messages come in pa i rs , a com mand and i ts response . A response is expected for each com mand se n t , and a buffe r must be ava i lable to receive ir. The SCA arc h i tecture specifies that a response w i l l be received for each command sent. Therefore. a command bu ffe r is made ava i lable on the free queue of the CI porr to receive the response . Thus each SCA path has two buffers avai table for control messages , one for send ing a com mand and rece iving its response . t he other for receivi ng a command and sendi n g i ts response .
Allocating buffers for SYSAP dialogues is not as simple as the commandjresponse al loca tion . I n this case , the buffer a l location must be based on the needs of the protocol used by the
SYSAPs. Some protocols are commandjresponse in nature , such as t he Mass Storage Control ProtO col (MSC P) used for the HSC and other storage control lers. Ot hers are not, such as the VMS con nection -manager prorocol used for VAXc!uster systems.
SCA a rc h i tecture enables the SYSAPs on a node to a l l ocate as many rece i ve bu ffers as arc needed for each connec ti on . Each SYSAP provides t hese buffers ro SCA. which then keeps track of them .
Each receive buffer acts as a " c redit" to a l l ow the ot her node to send one message over that con nect ion . The nod e's SCA software i nforms t he remote SCA softwa re of the number of cred i ts avai l a ble for each con nection . If a credit is not avai lable. t he remote SYSAP wi l l suspend sending its message . This style of buffer m anagement is ca l l ed " pess i mistic flow control . " It i s norma l l y u ns u i table for use i n ge neral networks i nvol ving routin g messages between nodes . However, s i nce routing is not done i n the SCA environment, this style has the advantage of bei ng completely pre di cta b l e . If a node mome n tarily l ags in satisfying commun ication requests made u pon it, the other nodes simply wai t u nr i l the lagging node recov e rs . Thus no add i t ional bu ffer management is requ i red .
The cost of these tight controls on buffe r man agement is some addi t iona l overhead to com mu n icate the credits to t he sendi n g node. These cred i ts are " p i ggybac ked " onto messages goi ng tO the correct node by inc l u d i ng a cred it fie ld i n a l l
SCA messages . When the SYSAP protocol docs not conta i n returning traffi c , however, addi tional control messages are req u i red.
The com mancljresponse nature of SCA control messages and the pessim istic flow control for
SYSAP messages remove much of the time-related behavior from the SCA a rc h i tecture . That means the SCA operation is re latively i ndependent of t he exact t i m i ng of t he arrival of messages and t he speed of response of the nodes involved i n the com m u n ication . These fac tors make it relat ive l y easy t o i m plement a n d ve rify the SCA softwa re.
Connection Management
A connection between two SYSAPs in d i fferent nodes is a correspondence between two connec tion ide n tifiers, one from each SCA i nstance .
These connection identifiers a l low the SCA soft ware ro mu ltiplex i ts services onto the underly ing vi rtu al circuit by d ispatching t he messages to the correct connection based on t he connection iden t i fiers . Each SCA message has a header con tain ing these con nection identifiers. Figure 2 shows the layout of an SCA message wi th t he for mar of t he protocol header.
When a node rece ives a message , SCA w i l l dis patch it based on the message type . For SYSAP-
2 4
Digital Technical journal
No. 5
S.-ptemb<'r
I
<JI:J7
Cl PORT H E A D E R
C R E DITS
I
MSG TYPE
DESTINATION CONN ECTION I D
S O U R C E CONN ECTION ID
SYSAP MESSAGE CONTENTS
1
SCA P
J
_j
OTOCOL
HEA
D E R
Figure
2 SCA
Message witb Protocol
Header related messages, SCA uses the I D of the desti na tion connection to dispatch further to the correct
SYSAP. As mentioned earlier, the cred i t field in each message header allows credi ts to be p iggy backed in message traffic.
A SYSAP signals its wi l l ingness to receive con nections from other SYSAPs by i nitiati ng a " l is ten" ca ll to its own SCA so ftware instance. This call establishes the name of that SYSAP in a l ist of names of wai t i ng processes . SYSAP names are defined by the architecture as strings of u p t o 1 6 characters. Some of the currently defined names are MSCP S DISK and MSCP STAPE for the disk and tape servers, YMS SVA.Xcluster for the YA.Xcl uster connection manager, and
SCS SDI RECTORY for the SCA directory server.
A SYSAP from another node, the source node. can establ ish a connection to a l istening SYSAP in a destination node by issu i ng a connect call to
SCA, giving the node address of the desti nation node and the name of the listening SYSAP. Two
SCA control-message pairs are required to estab lish a connection . The first commandjresponse pair from the source establishes the connection at the destination end ; the second pair from the des tination to the source e i ther accepts or rejects the
SYSTEM A
SENDING SYSAP STARTS
CALL CONNECT (system B.· SYSAPname·)
CONN ECT R EQUEST M ESSAGE
CONNECT R ESPONSE M ESSAGE connection . This separation i nto pairs al lows the destination SYSAP to decide , based on t he i nfor mation passed with the connect request from the source and on its current resources. whether or not to accept the connection .
Figure
:1 i J l ustrates the events required to es tablish a connection between two SYSAPs . The sequence of messages is as fol l ows :
1 .
A connect-request message is sent from the source node to the destination node . This message contai ns the source and destination
SYSAP names and
1 6 bytes of additional in formation from the source SYSAP.
2 . A connect-response message is sent from the desti nation node to the source node. This message indicates that a SYSAP w i th the requested name exists and that enough re sources arc present for SCA to honor a con nection . If there arc not enough resources, then the connection is refused .
3 .
Later. the destination SYSAP performs either an accept or a reject ca l l , and i ts SCA soft ware responds by sending either an accept request message or a reject-request message to the source node .
4 .
If the message was accept request, the source wi II respond with an accept-response message and noti fy i ts SYSAP that the con nection is open. If the message was a reject request , the source SCA software w i l l re spond with its own reject response, and the connection w i l l not be opene d .
The accept and reject responses by the receiv ing SYSAP arc separated from the connect-request and connect-response message pai r . That separa tion allows the SYSAP to ini tiate a potentially
SYSTEM B
CALL LISTEN ( ' SYSAPname·)
CONNECT R EQU EST SIGNALLED
SYSAPname SYSAP.
CALL ACCEPT (CONNECT ID)
ACC E PT SIGNALLED TO S E N D I N G SYSAP.
CONN ECTION OPEN TO SYSTEM B
ACC E PT R EQUEST M ESSAGE
ACC E PT R E SPONSE M ESSAGE
CONN ECTION OPEN TO SYSTEM A
Figur e
3
Events to
Open a Connection
VAXcluster
Systems
2 5
Digital Technical journal
No.
5 September I ')87
The .�ystem Communication A rchitecture time-consuming operation without tyi ng up t he
SCA control -message buffer of the send i ng SCA i nstance .
When either member of a pair of SYSAPs hold ing an open connection wishes to break that con nection , that member performs a disconnl'ct ca ll to its SCA software . The SCA software will i n form the SYSAP in the other node . which must then perform its own disconnect call to synchron ize t he dismantl i ng of the connection . Each side informs the other of the disconnect ca ll by exchanging a disconnect-request and d isconnect response message pair.
Directory Services
To accompl ish their tasks , the various SYSAPs runn ing within a node need the help of SYSAPs in other nodes . These SYSAPs operate either i n a peer-to-peer relationsh i p , such as the VA.Xcl ustcr connection manager,1 or in a c l ient-to-server rela tionship, such as the disk class driver and t he
MSCP disk server. The method by which SYSAPs find those other SYSAPs within the context of SCA is cal led the d irectory service. This snvicc is itself implemented as a SYSAP that l i stens for i ncomi ng connections. The service responds to a simple protocol of requests for information about which SYSAPs on this node are listening for con nections from other nodes.
To query the d irectory service , a SYSAP must request an SCA connection to another node with a destination process name of SCS S DIRECTORY
This specia
I process name is reserved for usc by the d i rectory serv ices. The requesting SYSAP can t hen i nqui re if a SYSAP with a particular name is l isten ing for a connection and also ask for a list of a l l SYSAPs currently listening for connections. figure 4 shows two VAX systems and an HSC device in a cluster, with the SYSAP processes lis ten ing i n each node.
VAX A
SCS$D I R ECTORY
VMS$VAXcluster
MSCP$01SK
HSC
SCS$ D I R ECTORY
MSCP$01SK
MSCP$TAPE
SCS$01 R ECTORY
VMS$VAXcluster
Figure
4
SBAP Processes among Three Nodes
Every i mplementation of a SYSAP has t he prob lem of fi nd ing partner SYSAPs of thl' same name to communicate with in the clustn. To central ize the software performi ng t h is function , the VAX/
VMS software impleme nts a general faci l i ty for
SYSAPs to find other SYSAPs . This faci l i ty periodi cally polls other nodes through t he d irectory ser vice to determi ne which l isten ing SYSAPs are present. 'fhis process poll er is a powerfu l tool that simplifies the design of the SYSAPs and the operating system software by allowing various
SYSAPs to start in one node wi thout depend ing on whether or not other nodes arc working yet .
When new nodes - and the SYSAPs within those nodes - arc added to t he cluster, a l l the SYSAPs currently runn i ng w i l l fi nd each other and com mun icate automatica lly.
Datagram and
Sequenced- message Services
The CI port and the C I interconnect provide the capab i l i ty to exchange datagrams and sequenced messages between ports . Datagram and seq uenced-message services arc both pro vided by SCA in the context of a connection . A
SYSAP establishes a connection wi th another
SYSAP and then sends datagrams or messages over t hat connection. l n the context of SCA, datagrams and messages, by convention . d i fft:r in size as wel l as i n the i r deli very mechanisms. Datagrams are
5 7 6 brtes i n length so that they arc suitable for use by the DECnet protocol as dt�talink buffers . Messages are
1 1 2 bytes in lengt h to accommodate MSCP control messages and VAX/
VMS lock manager messages.
Control l i ng the tlow of credits for datagrams and messages is done separately by SCA. Data gram credi t controls operate at the rl'cciver. The receiving of datagrams is not guaranteed . U pon receivi ng a datagra m , a SYSAP must have ava i l able a datagra m-receive credit; otherwise. the datagram is d iscarded. The receiving of messages. however. is guaranteed . Message-cred it controls arc institu ted at the send ing nod e . When a SYSAP wants ro send a message, the rec<:iving node must have a credi t avai lable . If nor , the scnd ing SYSAP waits and docs not send the message until i n formed that the cred it is ava i lahl c .
A-; mentioned earl ier, the port-to-port virtua l circuit provided by the Cl port hardware controls t he loss of sequenced messages buwccn nodes.
The c i rcuit retra nsm its these messages as neces sary to guarantee the ir deli very .
I n fact, rhc hard-
26 Digital Techn ical journal
.Vo. 'i
.
\ e p tem l> er
I 'JR �
ware performs this task for CI datagrams as wel l . but higher layers o f software do not take advan tage of this fact.
Datagrams are used to log events and other commun ications, such as from DECnet nodes, that control the Joss of datagrams i n other ways. I t is useful i n these applications t o d iscard informa t ion when buffering becomes a problem so that too many buffers are not consumed . In the case of event logging, the lost messages are l i kely to be duplicates anyway. I n t he case of the DECnet soft ware, h igher layers of DECnet protocol control the loss, and d iscarding the datagrams prevents congestion at i n termedi ate nodes.
SCA and the CI port work together to make message transfer more efficient by e l i minating transmit-done i nterrupts. When a node expects a response to a message, SCA and the C I port coop erate to queue the buffer sendi ng the message to the free queue. That buffer can then be used to receive the response . Thus i n a command/ response exchange of two messages, the sendi ng and receiving nodes each experience on ly one receive interrupt.
Named- buffer Transfer Services
One striking feature of t he C I port hardware is i ts ability to transfer large amounts of data between named buffers i n the virtual address space of pro cesses within a node 2 This feature is the most useful one for d isk and tape transfers .
SCA provides services for the two named-buffer transfer commands ava ilable i n the C I port: the send-data command , and the request-data com mand . The send-data command transmits the con tents of a segment of a local named buffer i nt o a segment of a named buffer i n a remote node. The parameters for the send-data command are the transfer length in bytes, and the names and byte offsets of the sendi ng and receiving buffers. The request-data command asks the remote port to transmi t data from a remote named buffer ro a local named buffer. The send-data command per formed by a d isk controller corresponds to a disk read function , and the req uest-data command to a d isk write function .
Of course, named-buffer transfers can be used by any SYSAP, not just the ones communicating with d isk controllers. Using named-buffer trans fers, i t is poss i ble for two VAX systems in a clus ter to exchange memory data at a transfer rate of over
2 megabytes per second at the Cl ports.
Extensions to Other Interconnects
To this poin t , only the C I i m plementation of SCA has been discussed . However, the util i ty of SCA is not l i mited solely to the Cl i nterconnect. SCA i s a general network communication architec ture that can serve a number of i nterconnects .
For example, i t is currently used i n local l y con nected storage controllers a nd on Ethernet for low-end VAXc luster systems.
Locally Connected Storage Controllers
The UDA50 UNIB US and KDB ') O B l d isk con trollers an: locally connected storage controllers t hat connect Digital Storage Architecture ( DSA) d isk drives to VAX computers without an i nter ven ing Cl i nterconnect. These devices are i nte l l i gent control lers that i ncorporate the SCA and
MSCP protocols, just as does the HSC 5 0 Cl-based disk controller. The use of SCA in these con trollers has proven to be an efficient means to commun icate with disk controllers i n which a d irect bus interface has tradi tionally been used.
I n control lers, there is no i nterconnect be tween the host adapter and the d isk controller; both functions are performed by the same con trol ler. Although the port header has been simpli fied because it does not have to address multi pie ports on an i n terconnect, the basic SCA functions sti l l operate . The use of SCA allows multiple functions to be placed i n a controller and used separately by having them appear as SYSAPs with d i fferent names. For example, d i sk a nd tape con troller functions can both co-reside in a con troller but are accessed via d i fferent SYSAPs.
Adapting SCA to Ethernet
Digital decided to extend the VAXcluster archi tecture to the Ethernet i n order to support work stations and other Ethernet-based systems . The most obvious way to accompl ish that extension was to build a port emularor for the CI capab i l i ties o n top o f the datagram capabi l ities o f the Eth ernet adapters . Such a port emulator performs t he functions of a CI port in software written as a driver running under the VMS system . SCA extends naturally i n this way since the Ethernet has the fu ndamental properties expected of a net work to be used by SCA That is, Ethernet is a mu l tiaccess media i n which the nodes need not be concerned with how packets are routed to the i r final destinations .
VAXcluster
Systems
Digital Technical journal
No. 5 September
I
'}87
2 7
The System Communication A rchitecture
SCA Performance
VAXcluster performance greatly depends on the performance of SCA, in terms of messages and bytes transferred per second . and on the overhead on the system software performing the transfer.
Not only does SCA perform srorage access. it also sends the lock manager messages that al low VAX cluster systems ro share devices and fi les. SCA. together with the CI port design, is i ndeed a h igh-performance and low-overhead intercon nect. For example, on a VAX- 1 1 /780 system, over
3 000 sequenced-message round trips per second can be exchanged with another VAX syste m . Yet , only about 3 0 0 microseconds of C P U overhead are required to send and receive each message pair. Each C I port wi ll sustai n approxi mate ly
2 megabytes per second of named-buffer trans fer bandwidth with no overhead on the part of the CPU . Each mass srorage operation requ ires a sequenced-message pair and a named-buffer transfer i nitiated by the HSC'50 d isk control ler. Therefore, the CPU overhead of SCA soft ware alone for these functions is only about
3 00 m icroseconds . The storage transfer itsel f can proceed at the rate of about 2 megabytes per second for long transfers to d isk or between host systems.
Summary
SCA is a high-performance network architecture developed to allow the C I i nterconnect ro be shared among the various functions requ ired i n
VAXcluster systems. Among these functions are mass-storage and tape-storage access, which had traditionally been done using direct control over a bus instead of a network message-passi ng proto col . SCA has proven ro be a highly efficient means both to control storage access and to a l low VAX host systems to commu n icate :; I ts flexibi lity per mits its use ro be extended to direct local-storage controllers and to other int erconnects such as
Ethernet.
Acknowledgments
A large number of people contributed to the SCA architecture and i ts impleme ntations. Without their efforts, SCA cou ld not have met i ts goals and wou ld not be so widely used . Thanks also tO all the fol ks who have reviewed and contributed helpful suggestions ro t h is paper.
References
1 . A . Lauck, D . Oran, and R . Perlman, " Digital
Network Architecture Overview,"
Technical journal
1 0- 2 4 .
(September
Digital
1 986) :
2 . N .
Kronenberg, H . Levy. W. Strecker, and R.
Merewood , 'The VAXcluster Concept:
An
Overview of a D istri buted System , "
Digital
Technical journal
(September 1 98 7 , this i ssue ) : 7-2 1 .
3 . W . Snaman , Jr. and D. Thie l , "The VA.."XjVMS
Distri bmed Lock Manager, "
Digital Techni cal journal
(September 1 98 7 , this issue) :
2 9 -4 4
4 .
N . Kronenberg, H . Levy, and W . Strecker,
" VAXcl usters: A C losely-Cou pled Distribu tion System , "
A CM Transactions on Com puter Systems , vol . 4 , no. 2 (May 1 986) :
1 5 0- 1 4 6 .
28
Di?,ilal Technical Journal
No. 5
September I ')87
William E. Snaman, Jr.
David W. Thiel
I
The VAX/VMS Distributed
Lock Manager
The VMS distributed lock manager provides the synchronization mecha· nism needed to ensure transparent and reliable data sharing between nodes in a VAXcluster system. The lock manager provides services for mutual exclusion and event notification, and achieves high performance by minimizing the number of messages sent between nodes. The lock man ager also handles deadlock situations with a minimum of messages exchanged. Since processors systems can join or leave a cluster at any time, a connection manager was developed to handle reconfigurations in a dynamic, efficient manner.
Development Background
A<> people and organizations came to depend heav i ly on computer systems to perform t heir daily activities, i t became i ncreasingly obvious that they needed continuous access to the vital data stored i n those computer systems. Moreover, growing organizations were faced with a need to i ncremental ly i ncrease the amoun t of computing power available to them over an extended period of time . I n the past, their options were usua lly l i m i ted to either buying more than needed i ni tial ly o r facing pai nfu l upgrades a n d application conversions as the systems were outgrown . The emergence of bus technologies, such as Digi tal 's
Computer Interconnect (CI) and the Ethernet, provided an opportunity to combine multiple processors and storage controllers into close ly coupled d istributed systems. Such systems could provide the needed clara ava il a b i l i ty and i ncre mental growth characteristics. The VA.Xclustcr system was developed to answer those needs. 1
To encompass the VA.Xcluster concept, the VMS operating system was extended to provide trans parent data sharing and dynamic adjustment to changes in the u nderlying hardware configura tion . These extensions make it possible for multi ple processors, storage control lers , cl isks, and tapes to be dynamically added to a VA.Xcluster system configuration . Thus a sma l l system can be purchased i nitially and expanded as needed by adding computing and storage resources with no software modifications or application convcrsions. New devices can even be added without shutting down operations. The abil i ty to use redu ndant processors and storage controllers vir tually elimi nates single points of failure .
The VMS software running on each processor node in a VA.Xcluster system provides a high level of transparent data s haring and i ndependent fai l ure characteristics. Each processor runs its own copy of the operating system a nd i nteracts with the other processors tO form a cooperating dis tributed operating system . In this system, a l l d isks and the fi les residing o n them arc accessible from any processor in exactly t he same fash ion as if those fi les were connected to a single proces sor. They can be transparently shared at t he record level by appl ication software .
One of t he chall enges of putting together such a system is to provide both max i mu m perfor mance and a very high level of rel iabi l ity. A data sharing model was chosen as the design center rather than a cl ient-server mode l . In the data sharing mode l , data resources are made direct ly ava ilable tO a l l processors, which must coordi nate t heir accesses to t hose resources. This model contrasts w i th t hat of the c l ient-server, i n w hich t he server mediates access to the data . The data sharing model e l i minates potential bottlenecks that develop around heavily uti l i zed servers, pro vides better opportunities for para l l e lism, and avoids the server as a single point of fa i l ure .
In 1 9 8 2 , the first lock manager was provided in version 3 0 of the VA.XjVMS operating system.
DiJ:ita/ Technical journal
No. 5
September
I
<JR7
29
The VAXjVMS Distributed Lock Manager
The lock manager provided synchroni zation ser vices for multiple processes resid ing on a single processor, as well as deadlock derection 2 Con currently, design work was u nder way for a dis tributed version of rhis lock manager . The d is tri buted lock manager was released in
1 9H4 wirh version 4 . 0 of r he VAXfVMS operating system; the
C I bus was used as the com mun ications med i u m .
I n 1 986, the Local Area VAXc lusrer sysrem was rel eased 5 This system has the same locking and other algorithms as the CI-based VAXcluster sys tem, but uses the Ethernet as the communications interconnect .
Lock Manager Description
This paper describes the d istributed lock man ager, which is the basic synchroni zation mecha nism for VAXcluster systems. The lock manager perm i ts the high degree of transparent data sharing attained by t he VMS system by provid i ng a set of services used by cooperating processes to synchronize access to shared resources. These processes can reside on any or all of the VAX pro cessors that com prise a VAXcl usrer sysrem . I n this paper, the terms " node " and " processor" arc used i nterchangeably to refer to VAX processors.
Each resource in a VAXc luster system is repre sented by a unique abstract name that is agreed upon by all the cooperat i ng processes . This name is entered i nto a d istributed global namespace that is maintai ned by the distributed lock man ager. Cooperating processes can use the lock manager as a mechanism to mediate access to a resource by requesti ng loc ks on t he abstract rep resentation before accessing the acrual resource .
The lock manager does not actua l ly a! locate or control the resource , and there is no req ui re ment that t he name represent an actual physical resource . This permi ts the lock manager services to be used for event notification and other com munication functions, in addition ro mutual exclusion functions. Deadlock detection is a lso provided .
To per m i t maximum concurrency, resource names can be rree structured , and locks Gin he requested at modes thar perm i t varying degrees of sharing. Many resources have an inherent hierarchical structu re rhat perm i ts d i fferent pans to be accessed by different processes ar rhe same rime. For example, a d isk can cont a i n various fi les, each in turn containing records This struc ture allows different records of the same fi l e . and different fil es to b e updated concurr(.'n t ly .
Providing tree-structured resource names per m i ts locks to be requested at d i fferen t levels of the h ierarchy.·• ln the lock manager, six lock modes are repre sented by an abstract matrix t hat defines whether or not a given mode is compat i ble with another mode.
An application designer can i n terpret t hese modes as setting l im i ts on how a resource can be accessed (e .g. , no access, read, or wri te) .
The modes can also be i nterpreted as setting l i m i ts on how a resource is shared (i .e . , permi t read access. write access, or no access to others) . Lock requests that arc granted at one mode can be con verted ro a more or less restrictive mode . Table 1 descri bes the compatibi l i ty of each lock mode;
Table 2 conta ins the suggested interpretation of each mode.
The services provided by the distributed lock manager arc flexible enough to be used by coop erating processes for mutual exclusion , syn chron ization . and event notification . These ser vices arc known as the S ENQ ( lock) and S DEQ
( u n l ock) syst(.'m services. The S ENQ system ser vice allows a process to request a lock on a resource . The lock request is then e ither granted or denied by the lock manager, based on the mode of other locks that are granted on the resource . The S ENQ service al lows a caller to queue a lock request and e ither wait for the request ro be granted or continue execution . The caller can also sign i fy that the request shou ld not be queued . In this case the status is returned i n the cvenr that the request cannot be granted i mmediately.
I f a caller chooses to queue a lock request and continue execution , the S ENQ service provides asynchronous notification when the lock request is granted . Tht: ca ller can specify a rourint: to be cal led when tht: lock req u est is granted . This
Ta ble 1 Compatibility of Lock Modes
Mode of
Requ ested
Lock
N L
C R cw
PR
PW
EX
Mode of Currently Granted Lock
N L CR cw
PR PW EX
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes Yes Yes
Yes Yes Yes
No
Yes No
No
No
No
No
No
Yes
No
No
No No
N o No
No No
N L - Null lock
CW - Concurrent write
PW - Protected write
C R - Concurrent read
PR - Protected read
EX - Exclusive lock
50 Digital Technical journal
No. 5 September 1 987
abi l ity to specify a routine perm i ts queuing a request i n a way that leaves t he process free to carry on other fu nctions u n t i l t he request is granted . The notification mechanism used is called a completion asynchronous system trap
(AST) .
The S ENQ service a lso provides a notificat ion mechanism whereby a process that has been grantee! a lock on a resource can be notified when another process is wa i ting for it to release the lock. This mechanism , known as a blocking AST, can provide an i mportan t performance optimiza tion when a resource is shared infrequently. After acquiring a lock. the holder can access the resource multiple t imes wi thout fu rther lock i ng unti l notified by a block i ng AST that another process is wa iting for it to rel ease the lock. The
Table
2
Modes at which Locks Can Be
Requested
Mode Suggested Interpretation of Mode
N L
CR cw
PR
PW
EX
N u l l mode g rants n o access t o the resource; it i s typically used either as an ind icator of interest i n the resource or as a place holder for future lock conversions.
Concurrent read mode g rants read access to the resou rce and allows its sharing with other readers. The concurrent read mode is generally used either when additional lock ing is being performed at a finer granularity with sublocks or to read data from a resource in an " u n protected" fashion
(allowing simultaneous writes to the resource).
Concurrent write mode grants write access to the resou rce and allows its sharing with other writers. The concurrent write mode is typically used either to perform additional locking at a finer g ran ularity, or to write in an " u n protected" fash ion.
Protected read mode grants read access to the resource and allows its sharing with other readers. No writers are al lowed access to the resource. This mode is the trad itional " share lock."
Protected write mode g rants write access to the resource and al lows its sharing with concurrent read-mode readers. No other writers are al lowed access to the resource.
This mode is the trad itional " u pdate lock."
Exclusive mode grants write access to the resource and prevents it sharing with any other readers or writers. This mode is the traditional "exclusive lock." holder then stops accessing the resource and releases the lock. thus perm itting the lock re quest of the other process to be granted.
Appl icat ions can be designed that dynamical ly change their locking protocol from blocking
ASTs (during periods of low contention) to a request- release protocol (during periods of h igh conte n t ion ) .
Another usc for the blocking AST is to imple ment a "door-be l l " notification mechanism in which a process takes out a l ock and speci fies a blocking AST. When another process wants to get the first process's attention , it makes an i ncom patible lock request that resu l ts in the del ivery of a blocking AST to the first process.
A 1 6-byte va l u e block associ ated with each resource fu nctions as a sma I I piece of global memory that is atomical l y updated . The contents of a va lue block are optionally returned when a lock is granted, and u pdated when a n exc l usive
(EX) or protected wri te ( PW) mode lock is re leased . Parameters on the lock and u n lock requests control the use of a va lue block.
A val u e block can be used to help i m p lement l ocal cac h i ng of d isk data . The resource repre sents the data be ing accessed and l ocks arc used to provide mutu a l exclusion . A val u e b lock asso ci ated with the resource is used to maintain a sequence nu mber representing the current ver sion of data stored on the disk. Whenever data is i n i t i a l ly read from the d isk i nto a local buffer, a lock is first obtained, and the version number con ta i ned in the val u e block is saved with t he data that is read. Whenever the data is to be mod i fied, a lock is first obta i ned, t hen the buffer i s updated and written back t o the d isk. When t he lock is rcl casccL an updated version n u m ber is stored in the va l u e block representing the new version of the data on the disk. Upon subsequent reads by this or any other node i n the VAXcluster system. a lock is first obtai ned . a nd the sequence number conta i ned in the value block is com pared to the sequence nu mber stored with the loca l l y cached data . Whenever the seq uence n u mbers match, the cache is va lid and no disk read is req u i red . '
Va lue bl ocks c a n a lso b e used for communica tion between processes.
The S DEQ system service is used to indicate that a process no longer wan ts to maintain a lock on the resou rce . Part of i ts function is to option a l ly u pdate the va lue block when the mode of the lock being re l eased is ei ther J>W or EX .
VAXcluster
Systems
Digital Technical journal
No.
5 Sept em her
I
'J87
5 1
The VAXjVMS Distributed Lock Manager
Design Constraints and Goals
Several constra i nts were placed on the design of the d istributed lock manager, the most i mpor tant one being that i t had to be extreme ly rel i able. Thi s constrai n t was vital since the VMS fi le system, the Record Management Syste m , several database systems, and other critical products would depend on the lock manager to mainta i n the i ntegrity o f their resources. The lock manager had to be general enough so that many d i fferent applications could be b u i l t using i ts services, rhus avoiding the creation of a separate synchro nization tool for each appl ication . Moreover, the lock manager had to have very h igh perform· ance characteristics and be able to tolerate the fai l u re of an arb itrary number of processes or nodes.
For performance reasons, it was essential ro minimize the number of messages exchanged between the various nodes. This was especia l l y i m portant as t h e number o f nodes i ncreased .
Add itional ly, minimum penalties shou ld be imposed when a l l the cooperat i ng processes reside on a single processor. The goal was to have the cost i ncrease no more than l i nearly as the number of nodes i ncreased . In fact, what was atta i ned was a cost bounded by a sma l l constant that is i ndependent of the number of nodes that exist in a VAXcluster system .
Relationship between the Distributed
Lock Manager and the Connection
Manager
As the lock manager was being developed , it became clear that a need existed to separate the function of managing a dynamic configuration of processors from that of managing the resource namespace. This separation requ ired the creation of a new entity , the connection manager. The d is tri buted lock manager rei ies on the connection manager for several viral services.
The connection manager maintai ns a globa l l y consistent l ist of a l l processors that are i n the
VAXcluster system at any given i nstant. To main tain this consistency, the connection manager uti lizes a very strong notion of c l uster membersh ip and orchestrates the addition and remova l of nodes . Part of that orchestration process is the coordi nation of the distributed lock manager's task of rebui l ding a database describing the lock· ing namespace and stare whenever the configura t ion changes.
Another function o f the connection manager is to prevent the partitioning of the namespace .
This partitioning could happen if the d istributed l ock managers in disjoint subsets of nodes oper ated i ndependently. They cou ld do so in the event of a communications fai lure, or a " rol l i ng" power fai lure a nd recovery cycle. I n these situa· tions, any objects accessible to mu ltiple subsets cou ld be inconsistently accessed and therefore corrupted . The connection manager uses a voting algorithm to ensure that the set of available pro cessors cannot be s p l i t into two or more function i ng groups if communications fai l . This approach requ i res that only a very l i m i ted amount of global information ( i .e . , the number of votes hel d by a node and the total nu mber of votes ava i lable to the entire set of member nodes) be known by each system . Furthermore , protection is given against a very wide set of fai lures because there are no additional underlying assumptions about fai lure mechanisms.
The fi nal function is a commun ications ser vice that provides a virtual circui t between each member node of a VAXcluster system . This ser vice ensures the rel iable delivery of sequenced messages . If messages cannot be delivered in sequence, the virtual circui t wi l l break . The most sign i ficant characteristic of this service is that cl uster membershi p and the existence of the v i rtual circu i t are t ightly coupled . The virtual circuit must exist for a pair of nodes tO become or rema in part of a VAXcluster system .
A fa i l ur e of the virtual c i rcuit, therefore, requ ires the removal from the cluster of at least one of the nodes termi nating that circ u i t . This approach greatly simp l i fied the design of the distributed lock manager because only one type of com muni cations fai l u re is visible to i t . The requ i red action upon the occasion of such a fai lure is made sim pler because ir is certain to be fol lowed by a change in the c l uster's membership. Such a change involves rebu i ld i ng the distributed lock manager's database .
The Operation of the Distributed
Lock Manager
The fol lowing section descri bes the operation of the distribmed lock manager when a l l lock requests can be granted i mmediately. A l ater sec tion discusses its operation u nder condi tions of conten t ion . Table
3 gives defi n it ions of the terms used in describing these operations.
3 2
Digital Technical journal
No 5 September J ')87
Table
3
Terms and Definitions
Term
Resource tree
Definition
The lock manager allows names to be structured in a hierarchical fashion. For example, the root resource can represent a device; its child, referred to as a sub-resource , can represent a file on that device; and another su bresou rce beneath it can represent a record .
Lock req uest
Root-lock
Sublock
The request by a process for a lock on a resource.
The lock request for a resource at the root of a resource tree.
The lock request for a resource below the root of a resource tree .
Resource manager The node that controls the granting of lock requests on a g iven resource tree for which it maintains i nformation about all g ranted and waiting lock requests. All nodes are potentially resource managers, each handling a particular su bset of the set of resource trees .
Di rectory service The directory service provides a mechanism to locate the cu rrent resource manager. This service is needed because lock requests must be directed to the resou rce manager, which may change over time. The di rectory fu nction i s distributed among the various nodes in a VAXcluster system, each node providing the function for a subset of the resou rce trees. This d istribution elim inates potential performance bottlenecks .
Lock mode The mode of a lock request indicates the type of lock being requested, such as NL, P R , or EX. B y convention, t h e mode represents t h e type of access t o t h e resource that is being requested, such as read , write, or no access. It also indicates a will ingness to permit others to share the resource.
VAXcluster
Systems
An Initial Lock Request on a
Root Resource
When a process somewhere i n a VA.Xcluster sys tem requests a root-lock, the distributed lock manager must first identify which node is cur rently managing the resource tree. The resource name specified by the lock request is hashed , and the resultant value is applied to a vector contain i ng zero or more entries for every node currently in the cluster. The selected vector entry identi fies the directory node for the resource specified .
A message is then sent to this node requesting a lock on the resource. The bui ld i ng and send i ng of a message can be avoided if the node making the request is also t he directory node .
The vector is maintained by the connection manager, which ensures that the vector is updated whenever a node enters or leaves the cluster. The connection manager also ensures that the vector is identical on all nodes. Each node can request that it be entered zero or more times in the directory vector, depending on the extent to which the node wants to participate in the d istributed directory function .
Upon receiving the message , the d i rectory node can respond in any of three ways . First, it can indicate t hat the node making the request should manage the resource i tself. Second , it can indicate that the request shoul d be re-sent to another node that is already managing the resource . Finally, it can respond to the request d i rectly, since the d i rectory node itself may already be managing the resource. If this lock request is the first one on the resource , the d irec tory node wi l l instruct the requestor to manage the resource itself. It wi l l also create a directory entry for the resource , thus ensuring that subse quent requests from other nodes will be di rected to the new resource manager. Figure I i llustrates this case .
All subsequent lock requests for additional root-locks or sublocks on t h is resource from t he node that originated t he i n i tial request will now be processed without further message traffic, since the node is now managing the resource i tself. This action , called local locking, was developed to minimize the cost of locking should all the processes sharing a resource reside on one node. Figure 2 provides an i llustration of local locking.
At this poin t , if a process residing on another node makes an initial root-lock request , t he resource name is again hashed and the d irectOry node identified in the same fashion as before . The request is sent to the directory node, which responds by identifying the node currently man-
Digital Technical journal
No.
5
September
1987
33
Tbe V AXjVMS Distributed Lock Manager
RESOURCE MANAGER
CD
NODE C
NODE
B
0
0
DIRECTORY NODE o: CD
CD
0
CD
0
When a lock request is received, a resource block and a lock block are created.
A message requesting a lock is then sent to the directory node.
A directory entry is created listing node A as the resource mana ge r
.
The response message directs node A to become the resource manager.
KEY:
D
RESOURCE BLOCK
D
0
DIR ECTORY ENTRY FOR RESOURCE
( I MPLEMENTED AS A R ESOURCE BLOCK)
LOCK BLOCK
Figure
1
A Root-lock Request When No Resource Manager Exists aging the resource. Upon receiving the response, the requestor re-sends the lock request to that node.
This case is potentially the worst with regard to messages since one round trip is required to the directory node (assum i ng t hat i t is another node in the VAXcluster system) and another round trip to the resource manager. Note that t h is cost is bounded by a sma l l constant with respect to the number of nodes in a VAXcluster system. Figure 3 i l lustrates this case.
Subsequent Root-Lock and Sublock
Requests
Once a lock on a root-level resource has been established , the identity of the resource-manager node is known. After that point n o further mes sages are sent to the directory node by that pro cessor; all requests are sent d i rectly to the resource manager. If the lock request is made on a node that is not t he resource manager, rwo mes sages are required for every lock request after the first : a request, and a response. This process is called remote locking. Figure 4 i l l ustrates the remote locking concept .
Releasing Lock Requests
When a process residing on the node managing the resource decides to release a lock, no mes sages are sent u nless the lock is the last remain ing one on the resource . I n that event a message is sent to the directory node ind icating that this
3 4
Digital TechnicaiJournal
No. 5
September
I 987
NODE A
NODE C
DIRECTORY NODE
D
As root-lock requests are received. lock blocks are created and linked to the existing resource block.
When a sublock request is received, a sub resource block is created if this is the first request for a lock on the subresource. A lock block is then linked to the subresource block.
KEY:
RESOU RCE BLOCK
D
0
DIRECTORY ENTRY FOR R ESOURCE
(IMPLEMENTED AS A RESOU RCE BLOCK)
LOCK BLOCK
Figure 2 Root and Sublock Requests Made on the Resource Manager node is no longer managing t he resource . The directory node then deletes the d irectory entry for the resource. This deletion allows the next node requesting a lock on the resource to become the resource manager. No response is necessary because the message delivery is guaran teed by the connection manager.
For the case in which a process releasing a lock does not reside on the node that manages the resource, a message is sent to t he resource man ager. Again, if this is the last remaining lock on the resource, the resource manager sends a mes sage to the directory node i ndicating that this node is no longer the resource manager. Figure 5 i l lustrates the concept of u nlocking.
Converting Lock Requests
The lock manager also permits the mode of a granted lock to be altered . This action is called a conversion . Conversion requests can be pro cessed more efficiently than new lock requests because all the data structures are already i n place, and the resource manager has already been identified. If a conversion request is made on the node managing the resource, no messages need be exchanged. If the resource manager is not the node on which the request is being made, either one or two messages are required . For example, i n some cases in which the requested mode is compatible with the granted mode, the request
Digital Tecbnicaljournal
No.
5
September
1987
3 5
VAXcluster
Systems
The V AXjVMS Distributed Lock Manager
NODE A
RESOURCE MANAGER
NODE C
NODE 8
DIR ECTORY NODE
KEY:
Dl
R ESOU RCE BLOCK
D
DIRECTORY ENTRY FOR RESOU RCE
0 LOCK BLOCK
When a new root-lock request is received, local copies of the resource block and lock block are created.
CD
CD
A message requesting a lock is then sent to the directory node.
The response i ndicates that node A is currently the resou rce manager.
0
0
The lock request i s again sent to node A .
A master-copy lock block i s created o n the resou rce manager and linked to the resource block.
(D
A granted response is returned.
Figure 3 New Root-lock Request When a Resource Manager Exists can be unilaterally granted , and a single message sent ro noti fy the resource manager of the change . In others, the resource manager must make a dec ision based on the other requests t ha t are granted . A request i s then sent to the resource manager, who must respond . Jn all cases, no com munications are req uired with the di rectory node. Figure 6 il lustrates a conversion request .
Operation During Periods of
Resource Contention
The operation is slightly more complicated dur ing periods of contention . When a resource man ager receives a lock request t hat cannot be granted because an incompatible lock exists, two actions are required . First, all holders of incom patible locks that have indicated a desire ro receive blocking ASTs must be noti fied that a pro cess is waiting. To accomplish this, a message is sent to each node where a lock holder resides.
The process holding the lock is notified only once , even though it may be blocking multiple lock requests. Second , the requester of the lock must be told to wait; this is accomplished by sending a response to the lock request. When the blocking lock is later released, a message is sent to each waiting requestor indicating that the lock is now granted . Table 4 su mmari zes the numbers of messages used for different types of lock requests .
Digital Tecb�Jical journal
No.
5
September
1987
NODE A
RESOURCE MANAGER
NODE C
NODE
B
DIR ECTORY NODE
D
When a sublock request is received. a lock block is created. If this is the first lock on the sub· resource, a resource block is also created.
CD
The request is sent to the resou rce manager. No d irectory lookup is requ ired.
0
If locks already exist on the subresource . only a lock block is created. Otherwise, both a lock block and a resource block are created.
(D
A granted response is returned.
K EY:
RESOURCE BLOCK
D
DIRECTORY ENTRY FOR RESOURCE
(IMPLEMENTED AS A RESOURCE BLOCK)
0
LOCK BLOCK
Figure 4 A Subtock Request on a Node that Is Not the Resource Manager
Scaling Behavior of the Distributed
Lock Manager
It can be shown that the number of messages requ ired for any locking operation is bounded by a small constant that is independent of the num· ber of nodes, or cl uster size, i n a VAXcluster sys· re m .
This section addresses how the size of the dara representing the locking state and the total number of locking messages vary with a cluster's size .
The distributed lock manager uses a fixed-size control block to represent both a lock and a lock request . An instance of this control block exists on the node requesting the lock. If the resource manager is a different node, another instance exists on the resource manager. A resource is rep· resented by another fixed-size control block. An instance of this control block exists on each node requesti ng the lock, on the resource manager, and on the directory node . Whenever any of these categories overlap ( i . e . , requestor, resource man· ager, and directory node) , only one instance of the control block is present. The control blocks for Jocks and resources are dynamicaLly allocated and deal located.
At l east one lock is represented for every resource represented. Conversely, a resource is represented for every lock represented . For each lock, the u pper bound on the storage require· ments is two lock control blocks and three resource control blocks . This upper bound is usually quite loose and depends on a c luster's size.
Digital Technical journal
No. 5
September I
Y87
37
VAXcluster
Systems
The VAX/VMS Distributed Lock Manager
NODE A
RESOURCE MANAGER
NODE B
DIR ECTORY NODE
When an unlock request is received for a root lock, the lock block is deallocated. If this is the last lock on the resource. the resou rce block is also deallocated.
0
A message is sent to the resource manager. No response is required.
CD
The resource manager deallocates the lock block.
If this i s the last lock on the resource. the resource block i s also deallocated.
0
A message is sent to the directory node.
(I)
The directory entry is removed.
KEY:
RESOURCE BLOCK
D
D
D I RECTORY ENTRY FOR RESOURCE
(IMPLEMENTED AS A RESOURCE BLOCK)
LOCK BLOCK
Figure 5 Unlock Request for the Last Remaining Lock on a Root Resource
VAXcluster applications are typically designed so that their algorithms do not change as the size of the cluster changes. Therefore, an i nstance of a typical application running on one node exhibits a behavior with respect to the number of ou t standing locks and the frequency of locking oper ations that is independent of the number of addi tional instances of that application running on the same or other nodes. If multiple instances of the application are running, the number of out standing locks and the frequency of locking oper ations increase i n proportion to the number of copies of the application , independent of the cJ uster size.
Both the number of messages per locking oper ation and the storage requirements for a lock are bou nded b y constants that are independent of the cl uster si ze. Therefore, the rate at which mes sages must be exchanged and the total storage required to represent the locking state are pro portional to the nu mber of instances of the appli cation that are running, which is also i ndepen dent of the cluster's size . If the nu mber of instances of the application i s proportional to the cluster size, the rate of message exchange and the total storage required to represent the locking state are both bounded by a constant ti mes t he cJ uster size.
This argument is also val id when multiple instances of each of several appl ications are present .
3 8
Digital Technical journal
No.
5
Septem ber
198 7
NODE A
RESOURCE MANAGER
NODE C
NODE
8
DIR ECTORY NODE
D
(D
A conversion request is received.
(D
The request is sent to the resource manager.
G)
The request is granted.
(D
A granted response is returned.
Note: Conversion requests on the resource manager require no messages.
KEY:
RESOURCE BLOCK
D
0
DIRECTORY ENTRY FOR RESOURCE
(IMPLEMENTED AS A RESOURCE BLOCK)
LOCK BLOCK
Figure 6
Conversion Request on a Node that Is Not the Resource Manager
These characteristics of the distributed lock manager ( i . e . , total space and message traffic behavior that is subject to a linear bound i n the
"workload ") are a significant fact or in allowing
VA.Xcluster systems to act as distributed operat ing systems . These characteristics suggest that, from the distributed lock manager's viewpoint, additional growth in the size of a VAXcluster con figurations is certainly viable.
Performance Aspects of the Distribu ted
Lock Manager
Table 5 sum marizes the performance of the d is tributed lock manager. The measurements reflect operations that are norma lly done in pairs . Such operations include an SENQ fol lowed by a S DEQ, and a conversion to a more restrictive mode (up) foUowed by a conversion to a less restrictive mode (down) . The operations reported in the table are performed on sublocks.
When Processors join or Leave the
VAXcluster System
The connection manager plays a major role in the lock manager's abi lity to deal with configuration changes when one or more nodes join or leave the
VAXcluster system. When the membership of the cluster must be altered, a coordinator node is e lected to lead the other nodes through the state transition . Any node can become the coordinator
Digital TeciJnical journal
No. 5
Sepl<!mher
1987
3 9
VAXcluster
Systems
The VAXjVMS Distributed Lock Manager
Table 4 Summary of Number of Messages Used for Lock Requests
Request Type
Messages Comments
I n itial root-lock request from a system for a previously unknown resource (i.e., no manager exists)
2 or
0
Zero messages if node making the request is the di rectory node. Otherwise two messages; a di rectory lookup req uest followed by a "do local" response.
0
Subsequent root-lock requests on resource manager
Sublock request on resource manager
U nlock request on resource manager with locks remaining
U nlock of last lock on resource by resource manager
0
1 or
0
Initial root-lock request from a system for a resource that is known (i.e., a manager exists)
Sublock requests and subsequent root-lock requests from a system that is not resource manager
U n lock request from a system that is not the resource manager
2 or 4 (1 )
2 ( 1 )
1 or 2
Remove directory entry message sent to d i rectory node. No message sent if manager is also di rectory node.
If requester is the d irectory node, two messages consisting of a lock request followed by a response from the manager. If requester is not directory node, do a d i rectory lookup, a resend to manager response, a lock request to the manager, and a response back.
Lock request to manager and a response back.
Dequeue message to manager. Manager may then send a remove directory message to d i rectory node if this lock is the last one.
NOTE: I f the lock request cannot b e granted immediately, add one message. I f the lock is granted, blocking another request, and a blocking
AST was requested, add one message. In all cases the number of messages is independent of the number of nodes in the VAX cluster system.
Table
5
Performance Summary of the
Distributed Lock Manager
VAX- 1 1 /780 VAXcluster System Locking
Using the Computer Interconnect (CI780)
ENO + DEO
CVT
(up+down)
Local
Locking
Local
CPU
Remote Locking
Local
CPU
Remote Elapsed
CPU Time
0 .
6
0.4
2.7
2.4
1 .5
1 .3
3.9
3.3
MicroVAX I I Locking Using the Ethernet
ENO
CVT
+
DEQ
(up+down)
Local
Locking
Local
CPU
Remote Locking
Local
CPU
Remote Elapsed
CPU Time
0.7
0.5
6.0
5.6
4.8
4.6
8 . 1
7 . 8
•
All numbers are in milliseconds
•
For Local Locking, Local CPU
�
Elapsed Time
•
ENO refers to a lock operation, DEO refers to an unlock, and
CVT to a mode conversion and i t i s usual ly the first to d iscover that a mem bership change is requ i red . The need for a mem bership change can resu l t from t i ming out a bro ken connection , or upon d iscovering a new node.
All configuration changes arc made using a two phase com m i t protocol to ensure consistency on a l l nodes. To add or remove a node, the coord ina tor descri bes a proposed configuration to the other members . They have the option of agreeing or d isagreeing with the proposed con figu ration .
They w i l l disagree i f they can construct a more opt i m a l configuration based on the nu mber of nodes they can com mu nicate with and on the assignment of votes to those nodes. The resulting
VAXc l ustcr system can only consist of a strongly connected group of nodes where every node has a connection ro each of the others .
I n case o f d isagreement , t h e coordinator backs out of the operation , wai ts a random amount of t i me , and t hen initi ates the election protocol aga i n . During this interval other nodes can attempt to become the coordi nator. D isagree-
4 0
Digital Technical journal
No. 5
September
I
')87
ments are quickly resolved so that the node that can put together the most optimal configuration becomes the coordinator. At this point, the new configuration has been described to all nodes and they have agreed ; therefore, commit messages are sent.
Thus the connection manager is able to provide the distributed lock manager with a consistent view of the processors that are members of the
VAXcluster system. The connection manager can a lso ensure that the vectors used to identify the directory node for a given resource are identical on all nodes. In addition, the manager assigns a unique identifier, called the cluster system I D
(CSID) , to each processor admitted into the VAX cluster system.
At the completion of any change in member ship, the connection manager leads the other nodes through a lock database rebuild. The node that was the coordinator now takes on the role of a synchronizer. Each node begins to execute a series of action routines that control how the lock database is to be rebuilt. Each action routine describes a particular step in the rebuild pro cess, and all nodes execute the action routines in parallel.
One or more action routines are separated by synchronization steps. Upon reaching a synchro n ization step, a node sends a message to the syn chronizer i ndicating that that node has com pleted a step and is waiting for notification to proceed with the next one. After receiving this message from each processor in the VAXcluster system , the synchronizer sends a message to each node telling it to proceed with the next step.
This process continues until all action routines have been executed and the lock database has been rebuilt on all nodes.
From the viewpoint of the distributed lock manager, the actions taken are identical when nodes are added or removed. This redistrib utes the management of resource trees to pre vent the management of most of them from migrating to the "oldest" member of the VAX cluster system .
Upon discovering a broken connection to a remote node, the connection manager initially assumes that this condition is temporary and attempts to restore the connection for a speci fied interval that depends on the installation .
During this interval, normal activity can gener ally proceed. Lock-request and other messages addressed to the remote node and sent using the connection manager's message delivery service are queued pending the re-establishment of the connection . If the connection is re-established, the queued messages are sent in the original order, and the sender remains unaware that a problem existed .
If the connection cannot be re-established within a specified interval, the connection is declared i rrevocably broken , and a cluster recon figuration is required. Locking is disabled on all nodes duri ng a reconfiguration. Lock requests can stil l be made , but the processes making them wil l be blocked pending completion of the state transition.
The lock database is rebuilt in the following fashion by each node. First, new lock requests are disabled . Then, the lock database is scanned and all directory information is removed, since a change in membership redistributes the direc tory functions. Information about locks that are either held or requested by processes on other nodes is also discarded . These actions result in a period of time during which no directory nodes and no resource managers exist. The only infor mation retained concerns the lock requests made by processes actually residing on a node.
At this point the nodes re-acquire all the locks held before the membership changed, using the same algorithm by which the locks were initially acquired. Locks that were waiting to be granted are re-ordered by a sequence number that was assigned when they were queued so that the order in which they wait is preserved. By the pro cess of re-acquiring locks, new directory entries are created and new resource managers chosen.
Since each node re-acquires its own locks, the locks held by nodes that are no longer members of the VAXcluster system are released. Once all locks have been re-acquired, an attempt is made to grant waiting locks since the removal of lock requests contributed by a failed node may permit wai ting requests to be granted . Once these actions have been accompl ished, locking is enabled and activity proceeds normally.
Distributed Deadlock Detection
The requirements for a distributed deadlock algorithm were to minimize the number of mes sages involved in a deadlock search, find all deadlocks, and not find false deadlocks. Since the distributed lock manager was to be a general purpose synchronization tool used by many
VAXcluster
Systems
Digital Technical journal
No. 5
September
1 987
4 1
The VAXjVMS Distributed Lock Manager applications, simplifications based on assump tions about the way it was used cou ld nor be made.
From the lock manager's perspective, there are two classes of deadlocks: conversion , and multi ple-resource . This distinction is made because conversion deadlocks are easily detected by the resource manager whereas mu ltiple-resource deadlocks are detected by a more complex d is tributed deadlock algorithm.
A conversion deadlock i nvolves multiple con version requests on a single resource so that a l l i nformation will b e readily available for the resource manager to identify them . Let us con sider a request to convert a lock held at one mode to another more restrictive mode (e.g. , from CR mode to EX mode ) . I f another lock i s also held at
CR mode , the conversion request must wait for the second lock to be released or converted tO a compatible mode . If an attempt is then made to convert the second lock from CR mode tO EX mode, a conversion deadlock results. The first conversion request cannot be granted whi le the second lock is sri
11 held at the original mode and the second conversion request cannot be granted because it must wait for the first lock to be granted.
A multiple-resource deadlock can be identified by searching for cycles in a " wait-for" graph of processes . A simple example can be constructed with two processes and two resources. Suppose a process PI , which is already hol d ing a granted lock on resource R 1 , waits for a lock request to be granted on resource R 2 . A deadlock resu lts if a process holding a lock on R 2 that is blocking
P l
's request attempts to acqui re a lock on R 1 t hat is i ncompatible with t he granted lock held by
P l .
Distributed deadlock detection is imple mented with an a lgorithm that searches the cl .
us terwide wait-for graph by sending messages to traverse arcs that cross system boundaries . The algorithm using messages tO traverse arcs between systems was developed i ndependently both at Digital and at I BM Corporation.6·7
One of the assumptions that was made in the design of the lock manager was that deadlock searches wou ld be an infrequent occurrence and relatively costly. This being the case , deadlock searches are initiated only after a process has waited longer than a configuration-specified period. This has the effect of greatly reducing the number of searches that are initiated . For exam ple, if process A on system 1 has a l ock request waiting for longer than the deadlock wai t inter val , then a deadlock search is i n itiated on i ts behalf.
Ti me-outs are detected on the node that is man aging a resource so that information about aU lock requests on the part icu lar resource is avail able for the deadlock search. If a conversion request has timed-out, the queue of conversion requests is searched to identify whether the granted mode of any conversion request made after the t i med-out conversion request is incom patible with the requested mode of the ti med-out conversion request. If one is found , a conversion deadlock exists and a victim is selected. The wai ting lock request of the victim is then com p leted with a error status indicating that a dead lock was fou nd . Granted locks are never affected by victim selection .
If no conversion dead lock is found , a more extensive multi pie-resource deadlock search i s i n i tiated . The wai t-for graph o f processes i s tra versed , beginning with t he process owning the timed-out lock request and searching for a path back to that same process. Beginning with the lock request, each process hold ing a blocking l ock on the resource is tested to determi ne i f the process has waiting Jocks on other resources. For each waiting l ock found , the a lgorithm is app l ied recursively until either no more waiting locks are found or the initial process is found . In the for mer case no dead lock exists because no cycle exists. In the later case a deadlock exists because a cycle was found to i nc l ude the process owning the lock that t i med out .
I f the arcs o f the wai t-for graph traverse proces sor boundaries in the VAXcluster system , mes sages are sent i nd icating t hat the search should be continued on the desti nation processor. The mes sages i ndicate that the search shou ld commence with a certain lock and continue with the u lti mate goal of d iscovering a path to the process owning the t i med-out lock request.
In the i mplementation , two possibilities exist that must be accounted for. In the first, a block ing Jock is found that is owned by a process resid ing on a remote system. I n this case the search must be continued on the remote system by iden tifying a l l locks that the process is waiting for. In the second , a process is wai ti ng for a lock man aged on a remote system . In this case the search must be continued on t he remote system by iden tifying all locks that are blocking the waiting lock.
4 2
Digital Technical journal
No. 5
September 198 7
Let us consider the fol lowing example. A wait ing lock request L 1 owned by process PI on node
N l times out , and a deadlock search is i n it iated .
The search is i nitiated on node N 2 , which man ages the resource tree . A blocking lock L 2 owned by process P2 located on node N3 is d iscovered on the resource . A message is then sent to node
N 3 , i ndicating that a search should be continued there, beginning with the lock L2, with the goal of finding a path to process P l . Upon receiving the message, node N3 determines that process P 2 is wai ting o n lock L3 managed b y node N4 . A message is sent to node N4 to continue the search starting with l ock L3 with a goal of findi ng pro cess P 1 . Lock L3 is discovered to be blocked by lock L4 that is owned by process P l . Since a cycle has been discovered, a victim is selected, and its waiting lock request is completed with deadlock status. Deadlock messages contai n the identity of the best victim found so far, and a message is sent to the node i n which the victi m resides.
An interesting extension to the simi Jar algo rithm described in reference 6 is used in the deadlock search. To prevent looping on cycles that do not i nclude the process with a t i med-out
Jock request and to greatly reduce the worst-case search t i me , a bitmap is used to i nd icate if a pro cess has already been visited i n the search. Each node in the VAXcluster system has a bit map with one bit for every process on that node . When the search is i n itiated, all bits are cleared. If a pro cess has been involved in the deadlock search, its corresponding b i t is set. I f a message then arrives that i ndicates that this process should be involved in the search, the message is ignored since a l l paths from this process have been searched already.
A node never knows when a deadlock search is completed because the messages simply die out when no dead lock is found . Therefore , some way must be provided to determine when the bitmap can be reused for a new search. That is accom plished by assigning a " ti mestamp lifetime" to the deadlock search. I n this scheme, one node is assigned the role of a timestamp server by the connection manager whenever the cluster mem bership changes. To initiate a deadlock search, a node requests a timestamp from the timestamp server. The timestamp represents a time slightly in the future . Once that t i mestamp has been issued, the timestamp server wi l l nor issue another until that time has passed ( i . e . , the times tamp has expired) . The i n i tial val ue of the t i mestamp is
50 m i U iseconds, based on an estimate of a reasonable worst-case search t i me . The t i mes tamp is used in the deadlock messages to indicate a speci fic deadlock search.
Whenever a deadlock message is received , i ts timestamp is compared to a t i mestamp stored with the bitmap. The comparison determines how the bitmap is to be used . There are three possible cases, described as follows :
•
The message value exceeds the bitmap value -
The bitmap was being used by a previous deadlock search and i ts timestamp lifetime has expired . In this case the bitmap is available for use by t he new deadlock search . The bitmap is cleared and the timestamp from the message is saved with it. The new search is then contin ued .
•
The bitmap equals the message value - The bitmap is avai lable and has already been used by an earl ier message i nvolved in this search.
Proceed with the search. If the bit corre sponding to the process requesting the lock is already set, then ignore this message since a l l paths from this process have already been searched .
•
The bitmap value exceeds the message value
The bitmap has been preempted by a sub sequent deadlock search. The timestamp assigned to this message expired before the search completed . Abort this deadlock search for now but reinitiate it later with a new times tamp that is double the last timestamp's l i fe time.
The bitmap optimization provides not only the performance benefits noted above, but a lso pre vents the a l gorithm from looping when it encounters u nsuspected deadlocks. For example, suppose process A is waiting for B which waits for C which waits for B. Processes B and C have a dead lock that will not be discovered when searching on behalf of process A since the u lti mate destination of the search is process A . How ever, the deadlock will be fou nd when searching on beha lf of B or C . The use of bitmap opti miza tion prevents the search from loop i ng when searching on behalf of process A.
Acknowledgments
The authors wou ld l i ke to acknowledge a l l those who worked to make VAXcluster systems a real i ty. We especially want to acknowledge Steve
VAXcluster
Systems
Digital Technical journal
No. 5
September
I ')87
4 3
The VAXjVMS Distributed Lock Manager
Beckhardt , who designed and implemented t he distributed lock manager. Also, our thanks to
Steve Neupauer, who supplied the performance figures, and to all those who reviewed t his paper.
References
1 . N . Kronenberg, H . Levy, and
W .
Strecker,
"VAXclusters :
A
C losely-coupled Dis tributed System," A CM Transactions on
Computer Systems , vol . 4 , no. 2 . (May
1 986) : 1 30 - 1 4 6 .
2 . L . Kenah and S. Bate,
VAX/VMS Internals and Data Structures ,
(Bedford: Digita l
Press, 1 984 ) .
3 . M . Fox and ]. Ywoskus, " Local Area VAXclus ter Systems , "
Digital Technical journal
(September 1 98 7 , t h is issue) : 5 6-68.
4 . J . Gray, R . Lorie, G . Putzo l u , and I . Trai ger,
" G ranu larity Of Locks and Degrees of Con sistency in a Shared Data Base," IBM
Research Report RJ I 654 ( I 9 7 5 ) .
5 .
VAX/VMS System Services Reference Man ual
(Maynard : Digital Equipment Corpora t ion , Order No . AA-Z5 0 I C-TE, I 986) .
6 . R . Obermarck , "Global Deadlock Detec t ion ." I B M Research Report RJ284 5 ( 3 6 1 3 1 )
Oune 1 980) .
7 . S. Beckhardt, Digital Equipment Corpora tion Internal Memorandum describing the dead lock detection a lgorit h m used by the
VMS operating syste m .
4 4
Digital Tet:hnica/ jourual
No.
5
Seplembf!r I 'J87
Andrew C. Goldstein
I
The Design and Implementation of a Distributed File System
The advent of VAXcluster systems, with their simultaneous requests for storage data, altered the requirements of the file functions in the VMS soft ware. To replace the single-system process, an extended QIO processor was developed to synchronize file accesses. The locks in the VMS lock man ager provide that synchronization by arbitrating and blocking requests.
Deadlock is prevented by taking out locks in a consistent order. Proper cache manag emen t is ensured by locks with sequence counters and a set of synchronization queues. This total scheme works so weU that, in addition to VAXcluster hosts, it is used for single systems as weU.
The VMS file system provides basic file-manage ment faci lities to all VMS users and to many other components of the VMS system itself. From a raw d isk, which consists simply of a series of data blocks, this fi le system provides files and fi le management, d irectories, security enforcement , and a variety o f functions related t o the intrica cies of manag i ng a fi le structure. The VMS i nter face to the file system is the SQIO system ser vice . 1 The SQIO read and write functions provide block-level access to file data . Other SQIO func tions specific to the fi le system create, access, modify, and delete fi les .
The SQIO service norma l ly leads to the VMS driver context. This context consists of initial kernel-mode execution i n the process context, with few system services allowed, followed later by interru pt-level execution . The complexity of the file system makes it impractical to execute i n the normal driver context. Therefore , the VMS system provides two methods for extending the operating context of the fi le system to provide the richness needed to support i ts complexity.
The Ancillary Control Process
I n VMS releases 1 through 3 , a technique cal l ed the ancillary control process (ACP) extended the file system's context. An ACP is a separate VMS system process that executes in a privi leged con text . All the VMS services normal ly available ro processes are available to the ACP, thus making feasible the implementation of complex code.
The ljO processing routines (the FDT routines) i n a process context send SQIO functions for the fi le system to the ACP. In turn , the ACP executes t he functions in i ts own context, returning com pletion data and status to the caller by using the ljO completion routines in t he VMS kernel .
An extension of the VMS buffered-I/O mechanism copies both the caller's arguments to the ACP and the return parameters back to the caller.
In addition to the extended execution environ ment, the ACP concept provides an i mportant fac i lity to the fi le system : synchronization . The
VMS fi le system ACP executes user functions in a single stream, completing each function before starting the next one . Thus a l l file functions are i n herently synchronized because only one ACP performs fi le management on a vol u me . More over, the i mplementation of a fi le system cache becomes quite simple and straightforward when operating in the single-process context . Figure 1 depicts the ACP-based fi le system .
D I S K
VOL U M E
Figure
1
A CP-based File System
Digital Technicaljournal
No. 5
September
1987
4 5
The VAx'jVMS Distributed Lock Manager
Cluster Alternatives
Many of the attributes that made the ACP concept attractive were i nval idated when the VMS soft ware had to su pport the VAXcluster concept.
VAXcl uster systems req u i re that each d isk vo l ume be accessible to a l l host systems in t he clus ter. Therefore , a disk volume can no longer be served by a single process. We exa m i ned other concepts, i ncluding having a single " master ACP " for a vol u me o n one member o f the cluster . That
ACP wou ld then execute a l l fil e fu nctions for a l l cluster members. We rejected t h i s approach , however, because of the high avai labi l i ty requ i re ments of VA.'Xcluster systems. Transferring the fi le system context to another cl uster member in the event of a fai lu re woul d have been very d i fficult.
Based on those considerations, we chose an approach that uses a symmetrica l file- manage ment design in which t he fi le functions execute on the cluster mem ber on which they origi nate .
No longer having the i m p l icit synchronization and cache management of a single ACP, we were now forced to address those issues expl ici t ly i n the distributed system .
The Extended QIO Processor
Our need for an expl icit synchron ization scheme e l i m i nated one of the major attractions of the
ACP : i ts implicit synchron i zation . In addition, it seemed redundant to have two schemes - one i mpl icit (ACP) and one expl ici t - to manage fil e operations. Therefore, rather than using exp l icit synchroni zation only between cluster members, we chose to usc it for a l l operations, i ncluding those local tO one processor. As a resu lt, we developed the second opera t i ng con text for t he file system now ava i lable i n the VMS softwa re : t he extended QIO processor, or XQP.
The XQP executes as an asynchronous system trap (AST) thread at the kernel level in the con-
USER USER USER
XOP XOP l
DISK
VOLUME
'-
Figure
2
XQP. hased File .�vstem
XOP text of the ca l l i ng process . An extended kernel stack and a data area located in the process's
P l region provide the necessary execution con text. S i nce execution occurs a t i nterrupt priority leve l ( l PL) 0, a l l the basic system services can be used . Figure 2 depicts the XQ P-bascd fi le syste m .
The X Q P design for the fi le system has several advantages over the d istri buted master-ACP design :
•
Consistency - Al l fi le operations are synchro nized in the same way, whether the vol u me is accessi b l e cl usterwide or not. This technique simpl ifies the synchron ization design and pro vides fewer opportu n i ties for bugs .
•
Performance - We e l i m i nated the process con text switch associated with an ACP ca 1 l by ru n n i ng the file system i n the context of the ca l ler.
•
Concurrency - Multi p l e fi le operations can proceed concurrently, in many cases, by i m p l ementing exp licit synchroni zation where it is needed , t hus i mprov i ng system perfor mance.
The remainder of this paper concentrates on the prob lems u n ique tO the VAXcluster dis tributed-fi le system : synchron ization , and cache management.
Synch ronization
The fi le system requires synchroni zation for two basic reasons:
I. F i l e structure integrity - Multiple users must be prevented from s i m u l taneously modifying the same pans of the fi le structure ( e . g . , atte mpting to fi nd and a l l ocate the same piece of free disk space to different fil es) .
2 . File system semantics - Certai n file opera tions provide user- level sync hron ization
( e.g. , preventing two users from simul ta neously accessing the same fi lc in a conflict ing manner) .
Synchron ization is achieved first by orga n i zi ng the fi le structure i nto units that can be synchro nized. then by using an underlying fac i l ity to control concu rrency. The VMS l ock-management services provide an idea l synchron ization fac i l i ty for VAXcluster systems 2 The VMS file structure readi ly decomposes i nto managea ble units . In fact, a l l u n i ts are files. Natura l ly. a file i tsel f is a
4 6
Digital Technical journal
No. 5
Seplernber
1987
fi le.
A d i rectory is a fi l e . Even the vol u mewide management structures (e.g . , the quota fi le and the srorage bitmap) arc fi les. Thus the fi le is the natural u n i t of synchron ization for most aspects of fi le operations .
1
Each fi le has a 4 H-bit fi le l D that un iquely identifies the fi le within a vol u me or vol u me set.
Removing the sequence nu mber from the fi le lD leaves a 3 2 -bit integer that u n iquely identifies the fi l e at any i nstant of time . This i nteger, the fi le nu mber , forms the resource name that syn chron i zes operations on the fi l e .
A fil e consists of i ts contents plus a fi le header, both of wh ich are synchron ized by a si ngle lock. Not all Jocks arc based on individual fi les. For example, for conve nience and efficiency , a si ngl e-vol ume synchro ni zation Jock controls the al location and deal lo catioo of all free space and fi le headers .
Armed with this introduction , we can now examine in deta i l how each lock is used ro syn chronize the operations of rhe file system .
Device Lock
'J'he device lock manages the states of devices accessed by the c l uster. The resource name of the lock is derived from the device name, prefixed with the text string SYS S . The fol lowing lock modes represent the device state:
Lock Mode
( No lock)
CR
PW
EX
Device State
Idle
Vol u me has channels assigned andjor is moun ted for shared access
Mount in progress
Volume al located or mounted privately
These lock modes provide the same device arbi tration that is avai lable on singl e-CPU VMS systems. The va lue block of the device lock con tains addi tional deta i l s about the device state
(device ownership and protection , whether moun ted or nor , whether mou nted on a fore ign system or not, etc . ) .
Mount Lock
'J'he device arbi tration semant ics in the VMS sys tem dictate that the device lock may not be wa ited upon ; any attempt at a confl icting access to a device yields a lock error. Therefore . an add i tional moun t lock w i ll serial ize concurrent attem pts to mount the sa me device. The resource name of the moun t lock is again derived from the device name. prefixed with the text string
MOU S . The mount lock is hel d i n EX mode while a user mou nts a device . thus a l lowi ng others i n the cl uster t o q u eue beh i nd the current mount operation .
Volume Synchronization Lock
Mounti ng a volume creates the vol u me synchro n i zation l ock in CR mode . This lock represents the mounted vol u me and associates one for one with the device on which the volu me is mounted.
The Jock's resource name is derived for shareabl e volu mes from the vol u me labe l . prefi xed with the text string F 1 1 B Sv . Thi s derivation guaran tees that all shareable vol u mes mounted in the cluster w i l l have un ique vol u me labe l s . Non shareable vol u mes usc the system address of the u n i t control block (UCB, the VMS data structure represe nting the device) as the vol u me lock name, thus a l low i ng vol u mes with dupl icate na mes to be mou nted. The value b lock of the vol u me lock conta ins add itiona l flags to describe the state of the vol u me as wel l as the al location and buffer-management states.
Both the device lock and the vol u me lock must he held by a cl uster member for the tota l length of t i me a vol u me is mou nted . This period wi l l usua l ly exceed the l i fetime o f any process i n the syste m . Therefore , normal locks. which arc asso ciated with an owner process. cannot bc used .
I nstead . thc fi le system uses system-owned locks. which arc he ld by the system as a whole , not by any particular process. As a resu l t . they survive the l i fe of any ancl a l l processes in the system .
These locks arc released only when explici tly commanded by thc system software or when the system lcavcs the cJ uster ( e . g . , i t crashes ) .
The volume synchronizat ion lock also synchro ni zes the a l location and cleal location of a l l space on the vol ume. When the XQP w ishes to a l locate space ( e . g .
. to create a fi le) , it takes a scparate copy of the vol u me lock in PW mode. ( Note that
PW mode is compatible with the CR-mode Jock representing the mou nt, but i ncompatible with i tse lf. That ensu res that on ly one process w i l l attempt t o al locate o r deal locate space at the same r i me . ) This form of the vol u me lock is held as a process lock. but on ly for short periods of time (the duration of a single fi le function or less) . Pan of the va l ue b lock for the volu me lock controls the a l l ocation of space and contains the current count of free blocks as we l l as poi nters
VAXcluster
Systems
Digital Technical journal
No.
5 September I ')87
4 7
The VAXjVMS Distributed Lock Manager i n tO the space-allocation bitmaps. Upon rai s i ng the volume lock to PW mode, the XQP reads this value block and wri tes it back to the lock man ager upon release .
File Serialization Lock
The fi le seriali zation lock synchronizes aU opera tions t hat affect the state of an ind ividual fil e . The resource name of the fi le serialization lock is sim ply the file nu mber, prefixed with the text string
F I I B Ss. The resource name is qualified by the volume name by virtue of being a sublock of the volume synchronization lock. By holding the file serialization Jock at PW mode, the XQP ensures that only one operation (opening, clos ing, extendi ng , deleting, etc .) is performed at a time on any one fi le. The serialization lock also ensures that only one operation i s performed at a time on any one directOry. The file serialization lock, a process lock, is held only for the duration of a single file operation .
The locks described so far are sufficient tO assure the i ntegrity of t he fi lc structure in the face of concurrent operations. However , two addi tional locks are required to support the syn chronization semant ics that the fi le system pro vides tO i ts users .
Arbitration Lock
The fi le system provides access arbitra tion for fi les; that is, users may open fi les for read or write operations and can specify whether other users may open the file concurrently .
An arbi tra tion lock is used to arbitrate fi le access across a cluster. The resource name of the arbitration lock is the fi le number, prefixed by the text string
F I I B Sa and the volume lock name (the resource name of the volume lock) . The arbi tration lock is held as a system-owned lock i n any of the avail able lock modes, depend ing on the state of access of the fil e . These states of access are
N L - No-Jock fi le access
CR - Open for read , al lowing other readsjwri tes
CW - Open for readjwrite , allowing other readsjwrites
PR - Open for read, al lowing other readers
PW - Open for readjwrite, allowing other readers
EX - Open for exclusive access
S i nce the arbitration lock is held for the entire time that a file is ope n , its usc is optimized. One system -owned lock represents the state of all accesses to t he fil e on each cluster node . The loc k mode represen ts the " h ighest" mode of access to the fi le on that cluster member .
Blocking Lock
Certain ma intenance operations on the fil e struc ture require it to be held stable for a period of time . For example, the ANALYZE/D ISK utility wi l l lock out all fi le operations during a disk rebu ild operation by using privileged fil e func tions to lock t he volume. To implement the lock i ng function c lusterwide requi res another vol ume-specific lock, the blocking lock. The resource name of the blocking lock is the volume l ock name, prefixed by the text string F l l
Since performance degrades if the lock man agt:r checks on the blocking lock as each file function starts, this lock is managed in a n opti mi zed fashion . Under normal condi tions , each cluster member holds t he block i ng lock as a sys tem -owned loc k i n CR mode . This state is noted in the vol ume control block (VCB) . Thus the start of every fi le fu nction requi res on ly a local state check. When a lock-vol ume function executes , it attempts ro raise t he blocking lock ro EX mode .
Since t he EX lock is incompatible with the CR locks , a system-blocking AST routine will be exe cuted on each cl uster member hold ing the lock at the CR mode . This AST rout i ne executes as a su brou tine cal led at IPL
8 using the JSB sub routine cal l instruction . The routine acqui res process context by " borrowing" the swapper process .
A kernel A..I)T is then queued ro the swap per, causing another rout i ne lO execute in the swappcr's process context . This other routine releases the CR-mode blocking lock and up d ates the VCB context accord ingly. When all the
CR-mode locks have been released, t he EX lock wi l l be granted and the lock-volume function comp letes
The volume w i l l rema i n locked because the blocking- lock check at the start of every fi le fu nc tion wi ll now fai l . When that happens, the XQP wi ll attempt to reacquire the blocking Jock. This attem pt causes the process to stal l because the blocking lock is sti l l held elsewhere in EX mode .
When a n unlock-volume funct ion fi nally releases the blocking lock, a l l processes waiting for the lock wi
I I also be released and the CR mode lock is re-establ ished . Normal fi le operations can then proceed .
4 8
Digital Technical journal
No. 5
Septernbe1·
I 987
Deadlock Prevention and
Locking Order
The execution of a si ngle file function can involve taking out several locks. Holding more than one lock at a time always presents the poten tial for deadlock. The XQP avoids dead locks, however, by taking out locks i n a consistent order, as follows:
I .
Blocking lock
2. Directory serializa tion lock
3. File seri alization lock
4 .
Volume lock
5 . Other special locks
Note that the ordering of the directory and fi le locks assumes a truly hierarchical directory struc ture . The VMS fi le structure a llows the creation of arbitrary links; thus directory li n.ks can point
"upward " in the directory hierarchy. Any artempt to traverse an upward l ink while another process is traversing the correspondi ng downward I ink can result in a dead lock error. The VMS system views such deadlocks as an exceptional circum stance and returns them to the caller.
Caching
The file structure of the VMS fil e structure is complex.-� Typical fi le operations require the examination or mod ification of several separate components of the fi le structure. To achieve acceptable performance, the VMS fil e system has always maintained extensive caches of compo nents of the fi le structure . These caches include the fol lowing:
•
A genera l-pu rpose block-buffer cache holds recently read disk blocks containing file struc ture components.
•
A fi le control block (FCB) list describes the attributes and states of all open fi les and recently referenced directories.
•
An extent cache holds a portion of the disk's free space for fast allocation and deal locat ion .
Space held in t he extent cache is marked " i n use" in the disk's storage bitmap ( the primary structure that controls space al locat ion) to ensure safety if the system crashes. Shou ld the system crash, the space in the extent cache will be temporarily lost. Because this space has been marked " i n use," there is no possibil ity of space that was a !located to fi les before the crash being again a l located to other files after the crash . Lost space is usually recovered with a disk rebui ld operation after the volume is mounted .
•
A fi le- I D cache holds a set of free fi le nu mbers for fast allocation and deallocation of fi le headers. Similar to those in the extent cache, fil e n u mbers held in this cache are marked " i n use" in the d isk's fi le-number bitmap.
•
When quota management is in effect, a quota cache holds quota records for currently active users.
Together, these caches absorb over 75 percent of the disk 1/0 that the fi le system wou ld other wise incur in perform ing fi le management func tions.
I mplemcming these cac hes in the single-sys tem ACP context was relatively straightforward.
The block-buffer cache was located in the ACP's process context; the rema ining caches occu pied sma ll portions of the system non paged poo l .
The advem o f clusters and t h e X Q P introduced the traditional prob lems of maintain ing cache coherency in a d istributed envi ronment . These problems were solved by using tradi tional cache consistency techniques and both trad itional and nontraditional appl ication of the VMS lock manager. Many of the synchroni za tion locks described so far also play a second role in manag ing the caches.
To put the block-buffer cache into a shared context , we moved this cache from the ACP pro cess context tO the system paged pool . The other caches remai ned in their exist ing locations Since each CPU in a cl uster has i ts own set of caches, a l l were synchron ized with locks using a combi nat ion of sequence counters and blocking ASTs .
Because major changes were involved , we wok the opportunity w examine some of the design decisions made in VMS version
I.
Based on this exam ination, we made some a lterations to reflect the changes in scal e that have taken p lace i n the
VMS software since i rs initial release . For exam p l e , the original block-buffer cache bad used lin ear searching on its descriptor tables. The new block-buffer cache uses descri ptors based on a hash tabl e to a l low faster access ro a la rge cache.
Previous versions of the VMS system used a si m ple directory-i ndex mechanism built into the directory's fi le control block . I n effect. this mechan ism kept a sma l l table of corm:nts that al lowed faster access to the entries of a d i rectory
Digital Technical journal
No. 5
September
1 98 7
4 9
VAXcluster
Systems
The VAXjVMS Distributed Lock Manager fi le. In the XQP conversion , this i ndex was moved into the block-buffer cache to i ncrease the space avai lable to each d irectOry index, thus i mproving i ts effecti veness .
Block Buffer Cache
The block-buffer cache consists of a col lection of
5 1 2 -byte buffers for disk blocks , plus the neces sary collection of descriptors and hash tab les.
Cache coherency is maintained using the trad i tional lock and sequence-number technique.
Every fi le structure block processed by the
XQP is governed by some synchroni zation lock.
The value block of the lock contains a sequence number representing the last update to blocks governed by that lock. Upon reading a block , the file system associates the current sequence num ber with the copy of the block held in the cache.
Upon mod ifying a block, the file system incre ments the sequence number and, at the end of the file operation, releases the lock with the updated sequence number. The corresponding locks are not ful ly released i f any data blocks remain in the cache . Instead, the locks are demoted to NL mode to preserve the continuity of the value block.
If another system's XQP su bsequently refer ences this file structure block and finds an old copy of it in its own block-buffer cache , that sys tem will find that the sequence numbers in the cache descriptor and in the value block of the lock do not match. This mismatch indicates that the block has been modified , and that the cache contents are invalid and must be refreshed from the disk.
We observed earl ier that the volu me synchro nization lock and the file serialization lock are the only ones strictly necessary tO ensure the integrity of the fi le structure . Consequently, all file structure data is read and written u nder these two classes of locks , which govern cache coherency. Blocks related tO space allocation on the volume, such as the storage and file-number bitmaps, are processed under the volume lock.
All other blocks, such as fi le headers and direc tory contents, are processed under the fi le serial i zation lock of the file to which they belong. The file serialization lock carries two sequence num bers to discrimi nate berween updates to fi le data
(e.g. , directory contents) and u pdates to fi le headers (e.g. , the directory file header) .
Detailed Cache Organization
The bu ffers of the cache are partitioned into fou r buffer pools . These pools contain
•
•
File headers and file-nu mber bitmap blocks
Storage bitmap blocks
•
Di rectOry, quota fi le, and m iscellaneous data blocks
•
Directory i ndex blocks
This partitioning is needed because one or two buffers of each type may have to be ava ilable con currently. For example, creating a fi le might
LOGICAL
BLOCK
N UMBER
HASH
TABLE
BUFFER
CACHE
HEADER
LOCK
HASH
TABLE
10
Figure 3 Buffer Cache Structure
50 Digital Technical journal
No.
5
Septem be,.
I ')87
requ ire concurrent access to the file header, the srorage bitmap for space allocation , and the d irectory to create the directory entry. Each buffer pool is managed using a variant of least recently used ( LRU) rep lacement. Consequently, the buffer manager can guarantee concu rrent access to one or rwo buffers of each type withou t any explicit buffer lock and release mechanism.
(Cerrain fi le and di rectory operations require concurrent access to two fi le headers or two d irectory blocks . ) The structure of the buffer cache is shown in Figure 3 .
Each buffer has a buffer descriptor (BFRD) , which contains the information needed to iden tify and manage the current buffer contents, as shown i n Figure
4 .
The BFRD contai ns the follow i ng information :
•
An logical block nu mber (LBN) and a u n i t con trol block (UCB) to identify t he disk address and the volume of the block contained in the buffer
•
The lock basis ( i . e . , the root of the resource name for the lock govern i ng t he buffer)
-
QU EUE
LINKAGE
LOGICAL BLOCK NU MBER
UNIT CONTROL BLOCK
LOCK BASIS
BFRL
NEXT
SEQUENCE NUMBER
TYPE
I
FLAGS
PROCESS ID
Figure 4 Buffer Descriptor Block
REFERENCE COUNT
LOCK
I
NEXT
1 0
LOCK BASIS
PARENT I D
Figure 5 Buffer Lock Block
-
•
The buffer sequence number from the va lue block of the lock
•
A poi nter to t he lock block of t he buffer
•
Flags, including valid and mod ified
•
A process ID of the buffer's owner
•
Queue pointers for state queue l inkage
•
A hash-chain l in k pointer
In add ition , a buffer lock block (BFRL) , shown in Figure 5 , is associated with each buffer, sev eral of which may be processed under the same lock. Thus the BFRL identifies the lock under which some set of buffers is managed and con tains the follow i ng i n formation:
•
The loc k ID of the lock
•
The lock ID of t he parent lock
•
The lock basis
•
A reference cou nt
•
A hash-chai n l ink pointer
Buffers and locks are fou nd using two hash tables, one each for BFRDs and BFRLs. The disk block LBN is used tO hash i n to the BFRD hash table; the lock basis is used to hash into the BFRL hash table. Each entry in the table forms the head of the hash cha i n for a set of BFRDs or BFRLs.
The cache header t ies together the compo nents in the block-buffer cache . The cache header contains
•
Base pointers for the hash tables
•
The BFRD and BFRL lists
•
Availabi l i ty counts and descriptors to form the four parti tions of the buffer cache
•
Performance counters
•
Several synchron ization queues
Each synchroni zation queue is described as follows:
•
Cache synchroni zation queue - Changes to the cache descriptors (e.g. , signing a buffer out of the cache for process use or changing the contents of a buffer) must be serialized.
•
Pool wa it queues - I f i nsuffic ient buffers are left i n the buffer pools, the XQP must wait before processing a fi le function .
VAXcluster
Systems
Digital Technical journal
No. 5
September
1 987
5 1
The VAXjVMS Distributed Lock Manager
•
Ambigu i ty queue - The lock name used to synchronize a file header someti mes changes.
For example, a l l headers of a multi header file are synchronized u nder the serialization lock of the primary fi le header. Therefore, the lock name for an extension header will change when the file is deleted and the header reused for another fi le. The ambiguity queue is used when the VMS software finds that a fi le header buffer is owned by another process under a different lock. Thus the queue allows the cur· rently executing XQP to wa it until the state of the header buffer stabil i zes.
Since each host CPU has a buffer cache, access to it is not synchroni zed by the lock manager.
Rather, an informal queuing mechanism, which saves considerable overhead, is used . When an
XQP must wait on one of the buffer header queues, it si mply sends the 1/0 packet represent· ing the current fil e operation into the appropri· ate queue and suspends execution . Some time later, another process i n the system w i l l rectify whatever condition the first process was waiting for (e .g. , making buffers ava ilable) . Having done so, the other process checks the appropriate queue to detect that the first process is wai t ing.
The first process is then restarted by removing its
1/0 packet from the synchron ization queue and using the 1/0 packet to queue an AST.
Buffer Management
In the block-buffer cache, each buffer is in one of two states: either it is avai lable for use (and may or may not contain va lid disk data) , or i t is owned by a process (and only one process) . The cache is carefu lly managed to avoid resource dead loc ks and to prevent individual processes from "hog· ging" i t .
A resource deadlock happens when a process partially executes a fi le function , then d iscovers t he need for an additional 1/0 buffer. Being par· tially complete, the process probably holds some locks. If no more buffers were avai lable, the pro· cess would have to wai t , holding its locks. In the meantime , some other process, also holding some
1/0 buffers, might attempt to acquire a lock that the first process is holding. In this case, that other process wi ll stall . This situation is the clas· sic deadlock of
"A is holding X and waiting for Y,
B is holding Y and waiting for X . " Yet the VMS lock manager wou ld not detect this dead lock because some of the entities involved are not
Jocks.
Resource deadlocks are avoided by reserving sufficient buffers before starting a file function .
Thus the fi le system i s designed so that all file functions can be completed using a known mini· mum number of buffers. If this m i n i mum num· ber is not available, the XQP must wait on the pool wait queue. Therefore , deadlocks cannot occur because the XQP is not yet holding any locks.
Bu ffers are reserved by simply decrementing the pool availabi l ity counters in the cache header. Individual buffers are not actually taken by the process until needed. The state queue linkage and the owner process 1 0 (PID) repre· sent the state of a buffer . An available buffer is linked i nto the LRU l ist corresponding to the buffer pool; this buffer has a zero-owner PID . A process takes a buffer when the process wishes to read a part icular disk block. The process selects an appropriate buffer either by finding the desired disk block i n the LBN hash table, or, i f the block i s not found , by removing the oldest buffer from the front of the LRU list. Tak i ng a bu ffer for process use involves first removing it from the LRU list and entering it into the pro· cess's in-process l ist, then entering the process
1 0 into the buffer's owner P I O field.
A bu ffer is never taken if marked with a differ· ent owner PID (i . e . , owned by another process) .
If the b u ffer is for a file header, the lock basis for the header could be changing; therefore , the
XQP must wait on the ambiguity queue. The lock basis for other types of buffers never changes while the buffer is owned . Therefore, find ing a buffer owned by another process i ndicates that fil e synchronization has been violated, which causes a system crash.
In many cases, more buffers than the necessary minimum may be usefu l in processing a fi le func· t ion ( e . g . , when a file has many headers or a large directory must be searched) . If more buffers are avai lable i n the cache, the XQP will continue to reserve and take them for process use . Once the cache avai labil i ty cou nters fa ll below a minimum threshol d, however , the XQP wi l l stop reserving addi tional buffers. I n this case, the XQP must return a buffer from its i n -process J ist for each new buffer taken. This swap prevents one very complex fi le operation from hogging a l l avai lable bu ffers and guarantees a minimum level of opera· t iona! concurrency.
At the end of a fi le operation , a l l buffers held on the in-process list must be returned tO the cache. Si nce modi fied buffers are not held in the
5 2
Digital Technicaljournal
No. 5
September
1987
cache , any on the in-process I ist are written back to the disk as they are returned . A'> the bu ffers are returned , the XQP ensures that each one is asso ciated with a BFRL correspond i ng to the synchro ni zation lock under which the bu ffer was read .
The XQP will release a l l synchroni zation locks when a l l buffers have returned . Locks corre spond ing to buffers remaining in the cache are not re leased but are demoted to NL mode to pre serve the buffer sequ ence nu mber.
The i nabi l i ty to hold mod ified buffers in the general cache is a small regression from the ACP based fi le system. VMS versions
2 and 3 cou l d hold modi fied fi le headers o f files currently open for write in the cache . That abi l i ty saved a wri te operation when such a fi le was mod i fied ( e . g . , extended ) . Now, the technique o f hold i ng modi fied buffers and flush i ng them under a blocking
AST is we l l understood . It is possible to add the necessary mechanism to the new bu ffer manager.
However, deve lopment ti me constrai nts pre vented us from including this capabil i ty in VMS version 4 .
User Interference
The fi le system is designed to tolerate the mod ifi cation of the file structure components by user level software (such as the disk-rebu i l d u t i l i ty) .
Therefore , when a user process opens the storage bitmap fi l e for a write operation , for example, any updates to that fi l e must be accounted for i n the block-buffer cache . This task i s accomplished by first recogn izing fi les that constitute compo nents of the file structure when they are opened for write , then routing a l l wri tes through the
XQP. The X Q P checks a l l blocks written aga inst the cache and i nval idates matching cache buffers.
File Control Blocks
Like the block-buffer cache , the fi le control blocks for open fi les and di rectories represent repl icated cache data that must be kept coherent.
The blocking AST mechanism in the lock man ager solves this coherency prob l e m . Recal l that each cl uster member holds an arbitration lock for each open file on a c luster-accessible volume .
Associated with t he arbitration lock is a system blocking AST routine. Fi l e access arbitration never invokes this rout i ne because arbitration does not wai t for fi l e accessibi l i ty. (File access con fl icts are returned as errors to t he caller.)
When a user modifies the attri butes of a fi le ( i ts size, protection , etc . ) , the various fi le control blocks across t he c l uster must be updated . This task is done by queu i ng an EX-lock request for the arbitration lock, thus causing the blocking
AST rou t i ne to execute . The AST routine simply marks the loca l file control block "stale." Once queued , the EX-lock request w i l l be i m mediately canceled since it w i l l norma l ly never be granted .
On the other cluster nodes, the next operation on t he fi le will update the fi le control block . The
XQP, fi nd ing the fi le control block marked stale, wi 1 1 refresh i t with file data read from the disk and rearm the blocking AST by re-establishing the arbitration loc k .
Quota Cache
The q uota cache presents a unique cluster-syn chron ization prob l e m . The quota cache contains a sma l l nu mber of currently active quota records, each representing a fi le owner to whom file space has been charged . Now , users normal l y mod ify fi les owned on ly b y themse lves. There fore, a sma l l cross section of the quota fi le, repre senting the set of users currently logged i n to the system , can be cached with exce llent local i ty.
The quota cache is especially effective because quota changes are reflected on ly in the cache ent ries. These cha nges are written back to the quota fi le only when replacement removes them from the cache . As a resu lt, a properly sized cache e l i m i nates a l most a l l the overhead of q uota management. Figure 6 i l l ustrates the access to the quota cache, and Figure
7 the entry to that cache .
Preserving the performance characteristics of the quota cache presented us with a u nique problem. The loca l i ty of use of fi le owners does not in any way reflect back into local ity of use of quota fi l e b locks. Thus the cache entries must be hand led on an i nd ividual basis. Quota-cache coherency across the cluster is maintained by using a separate lock for each quota-cache entry.
The dynamic part of a quota record (q uota, over draft, and usage , plus some flags) just fi ts i nto the
1 6-byte va lue block of the lock. The resource name of the lock is the fi le owner, plus the vol u me name and the text string F I I B S q .
A lock held a t PW mode backs u p each va lid entry to the quota cache . When another XQP in the cluster wishes to usc the same quota record . that XQP must find a suitable cache entry (by fi nd ing the fi le owner in i ts cache or ra king the
LRU cache entry) and then enqueue for the lock at PW mode . This action triggers a blocking A�T
VAXcluster
Systems
Digital Technica/Journal
No. 5 September
I ')87
5 .3
The VAXjVMS Distributed Lock Manager
,----
-
QUOCACHE
FLAGS
UNIT
CONTROL
BLOCK
VCB
VOLUME
CONTROL
BLOCK
LRU I N DEX
I
CACHE I N D EX
LOCK I D
I
RECORD NUMBER
USAGE
PERM QUOTA
OVERDRAFT
USER ID CODE
QUOTA BLOCK
AST CONTR. BLK.
QUOTA FLUSH
AST CONTR. BLK.
Figure
6
Quota Cache
QUOTA
BLOCK
QUOTA CACHE
ENTRIES
VAL U E
BLO C K
File Number and Extent Caches
During normal operation , the fi le-nu mber and extent cac hes, shown in Figure
8, do not present any synchronization or coherency problems in the c l uster. Since the cache contents are marked
" i n use" in the appropriate bitmap, each cache in each c l uster member simply conta i ns a d i fferent col lection of free disk space or free fi le numbers .
The cache m ay have to be emptied , however, and i ts contents written back to the bitmap.
There are two reasons for these actions. First, the fi l e system w i l l tolerate the mod i fication of the fi le structure components by user-leve l software
(e.g. , the disk-rebu i l d util i ty) . Therefore, when a user process opens the storage bitmap fi l e for a write, for exa mple, all instances of the extent cache must be flushed to the bitmap. That does two t h i ngs :
I .
I t presents the user with a correct view of the bitmap.
2.
It prevents the cache from contai n i ng stale data in the event the user mod i fies the bitmap.
Note , by the way, that the quota cache is affected by a l l these considerations as we l l .
Second , resource exhaustion must be handled as gracefu l l y as possible. With the extent caches in operation , the avai lable free space on t he disk is d istributed i n the various extent caches across t he cl uster. If a user makes an a l l ocation request for a l l the remaining free space on the d isk, that
Figure
7
Quota Cache Entry on the node currently holding the lock at PW mode . Because the quota-cache lock i s system owned , the blocking AST routine will execute at
I PL 8 . Usi ng a n AST control block bu i lt into the quota-cache structure, the rou tine queues an AST to the swapper process to borrow i ts process con text . The swapper AST executes another su brou tine that releases control of the entry to the quota cache . This subroutine marks this entry " invalid" and demotes the PW lock to
CR mode, in the pro cess wri t i ng the entry contents into the va lue block of the lock . Upon re lease , the lock is granted to t he requesting process, which trans fers the lock's value block into i ts cache entry. A..;; a resu lt, the lock manager can transfer quota cac he entries about the cluster without incurring any disk ljO. r--
UNIT
CONTROL
BLOCK
VCB
VOLU M E
CONTROL
BLOCK
�
CACHE
VOLUM E
CACHE
BLOCK
FlOC ACHE
EXTCAC H E
F I L E
N U M BER
CAC H E
EXTENT
CACHE
Figure
8
File Number and Extent Caches
54
Digital Technicaljournal
No. 5
September
1')87
request cannot be satisfied without emptying the extent caches on the other cl uster members .
A cache-flush lock wi l l handle both situations stated above . The quota , file-number, and extent caches are each backed by a cache-flush lock .
The resource name is derived from the fi le number of the related fi le, plus the text string
F l l
B Sc . While a cache is active , the cache-fl ush lock is hel d as a system-owned lock at PR mode.
When wishi ng to cause a cache fl ush for a cer tai n rype of cache across the cl uster, the XQP enqueues for the rel ated lock at CW mode . This action causes the blocking AST associated with the PR lock to execute as a fork IPL 8 routine.
This routine uses an AST control block bui l t into the cache structure t o queue a n AST to the
CACHE_SERVER process of the fi le system. O ne such process runs on each node in a cluster; its sole responsibility is to respond to cache-fl ush requests.
The parameters associated with the AST iden tify which cache is i nvolved and the vol ume for which the cache is to be fl ushed . The
CACHE_SERVER process then executes a privi leged fi le system control function that causes the fi le system to empty the specified cache. Having emptied the cache, the XQP releases the PR lock, thus allowing the process requesting the CW lock to proceed . If a cache fl ush is requested simply to make all free space ava i lable, the CW lock wi l l be immediately rel eased . If the cache flush is associated with opening a piece of the fi le structu re for a write , however, the CW lock wi l l be held as a system-owned lock until the fil<: is closed . Since any attempt tO refi l l the cache must first acquire the PR lock, such attempts w i l l fai l unti l the fi le i s closed and the CW lock released .
Summary
The d istributed file system was one of the most chal lenging aspects in developing VA.Xcluster systems. Starring from a fi le system that was pro cess based and s i ngle threaded, we developed one that is proced u re based and mu l t ithreaded .
The major cha l lenges lay in deve loping the nec essary synchroni zation and in redesigning the caches to work correctly in the distributed envi ronment . We solved these problems by exten sively employi ng the VMS distributed-lock man ager in new and creative ways . The result is a fi le system that works effectively in the cluster envi ronment . What's more. this fi l e system displays better performance and concurrency i n t he sin gle-system environment as wel l .
References
1 . VA.XjVMS ljO User's Reference Manual,
Part 1 (Maynard: Digital Equi pment Corpo ration , Order No. AA-Z600C-TE, 1 986) .
2 . VAXjVMS Systems Services Reference Man ual ( Maynard : Digital Equ i pment Corpora t ion , Order No . AA-Z50
I
B-TE, 1 986) .
3 .
Guide to VAX/VMS Disk and Magnetic
Tape Operations
(Maynard : Digital Equip ment Corporation , Order No . Al-Y506B-TE,
1 986) .
VAXcluster
Systems
Digital Technical journal
No.
5 September 1987
5 5
Michael S. Fox john A. Ywoskus
I
Local Area VAXcluster Systems
Local Area VAXcluster systems use the Ethernet rather than the CI bus as their interconnect between nodes. Ibis makes it possible to include
Micro VAX systems and workstations in a VAX cluster environment. The key technical issues that had to be solved were to provide an Ethernet base equivalent to the CI bus for the cluster's System Communication Architec ture protocols and to allow the VMS software to boot on a diskless system using the Ethernet as a link to a remote system disk. Ibis paper describes the work done to satisfy these two design issues: providing robust cluster communication on the Ethernet as a means of performing remote disk access, and network booting of the VMS system.
The Local Area VAXc luster (rAVe) software is a new product t hat brings VAXcluster functional i ty to the fu l l range of VAX processors . A rAVe uses t he Ethernet i nstead of Digita l 's proprietary Com puter Interconnect cal led the CI bus, t hus mak ing possible the i nc lusion of sma l l systems l i ke t he MicroVAX I I CPU in the VAXc luster configu ration . This paper describes the benefits pro vided by a rAVe, t he concepts on which it was bu i l t , and the techn ica l deta i ls of the two new major i nternal capabilities added to the VMS operating system .
VAXcluster System Definition
A VAXcluster system is a d istributed system made up of VAX computers and t heir associated stor age elements , a l l l i n ked in a c losely coup led arrangemenr. 1 VA.Xcluster members cooperate with each other on a peer-to-peer basis. They a l l share a common fi le system, print and batch queue operations, and comprise a single manage ment domain (the c l uster is managed as a single system entity) enclosed by a single security perimeter.
A VA.Xcluster system d i ffers from a more tightly cou pled mu ltiprocessor arrangement i n severa l ways . First, the VAX systems commun icate over a fast, efficient network l i n k instead of sharing memory. Second, each system has i ts own copy of the VMS system i n memory (possibly loaded from the same shared disk i mage) . Third, t he members may boot and shu t down independently. Fina l ly, the cl usterwide fi le syste m , single security and management domains, and other VAXc l uster fea tures are much closer to those offered by a trad i t ional single ti mesharing system t h a n t o t he capa bil it ies offered by trad itional networks .
Thc first VAXcluster implementation (VMS ver sion
4 . 0) operated on ly on the CI bus, a limited distance LAN connecting up ro sixteen nodes at
70 mega bits per second . C I adapters are highly i nte l l igent, and hence relative ly complex and expensive . They were bu i l t expressly for large systems located i n machine rooms. With thc advent of sma l l desktop VAX processors, some new interconnect was needed for bringing them t he benefits of c l uster fu nction a lity. The CI bus could meet neither the geographical criteria nor the low cost requi red in an office (as opposed ro a computer room) environ ment , nor cou ld it sup port enough nodes .
The VA.Xc l uster support i n VMS version
4 . 4 had matured enough so that extend ing it to another i n terconnect became feasibl e . The Ethernet , a l ready Digita l 's standard for network com muni cation , was the obvious choice for this new i n ter connect . Ethernet's cost, d istance, speed , connec tion capabi l i ties, and existing hardware base a l l owed the VA.Xcluster fu nct ions to move out of t he machine room and effective ly support sma l ler systems.
LAVe Goals, Requirements, and Configurations
The overa ll LAVe goa l was to bring the benefits of VAXcluster systems to low-end and desktOp
DiJ!.ital Technical journal
No. 5 Septern/Jer
I
'J87
systems. The benefits of this goal included the following:
•
A single, clusterwide common fi le system with disks connected to any CPU
•
Fully integrated and synchroni zed fi le sharing at the record level among users on any member in the cluster
•
Clusterwide availabiliry of print and batch queues ( Print and batch execution facilities can be located on any set of members . )
•
A single securiry domai n
•
The simplification (or even elimination) of the end user's system-management responsibi lities
With this goal in mind, we drew up a l ist of requirements for such a product. These requirements included
•
Support the Ethernet instead of the CI bus as a cl uster interconnect, yet a llow simultaneous use by other clusters and nerworks
•
Boot the VMS sofrware over the Ethernet
•
Simplify cluster management and installation by providing tools a nd l i miting configurations
•
Provide clusterwide disk access by means of the sofrware Mass Storage Control Protocol
(MSCP) server instead of the HSC controllers
•
Retain all the existing VA.Xcluster software capabi l ities and as much of the implementa tion as possible
•
Support d iskless systems
The first three requirements had the largest impact on the LAVe deve lopment. In fact, the first rwo required the most engineering effort to deve lop new sofrware . After a brief description of the resulting LAVe product, the remainder of this paper will descri be the techn ical work done to meet those two requirements.
Configurations
The configuration supported by the i n itial releases of LAVe utilizes a single Ethernet as the cl uster interconnect. Conservative restrictions were imposed where necessary to l imit the com plexity and to aLlow thorough testing and perfor mance analysis of almost all supported cluster configurations. The resu lt is the configuration shown in Figure l .
Future extensions to increase the n umber of members, allow both CI and Ether net in the same cluster, and multiple Ethernets are being p lanned. They will not be addressed further in this paper.
The members cooperate with each other in a peer-to-peer relationship. They are managed by a cluster connection manager and synchronized by a distributed lock manager without regard for the roles t hey play in an operating LAVe . 2 That is a key difference between the LAVe and other
"cliem;server" products . Any system in the clus ter can provide or consume resources provided by the other systems. To simplify the resulting supported configurations, however, we chose to assign certain roles to t he systems. The boot member and satell ite roles merely describe the jobs those systems perform ; t he roles are not known by the VA.Xcluster sofrware . The cluster software cares only where the resources are located and which systems have access to t hem.
Each boot member is a management center of the cluster. The VMS system disks connected to each boot member makes them ava i lable to other cluster members by means of the MSCP server software. The initial LAVe releases limit the num ber of boot members and system disks to reduce the complexity of installation and management.
VAX VAX
ETH E R N ET
Figure
1 LAVe
Configuration
VAXcluster
Systems
Digital Technical journal
No. 5
September
I
987
'5 7
Local Area V AXcluster Systems
Boot members may also serve other data disks i n the cluster.
A boot member also functions as a load host during an Ethernet boot operation . This role is discussed further i n the sections on remote booting.
Satellite systems boot off the system disk pro vided by a boot member and general ly depend on that member for other resources as wel l (data d isks, printers, etc.) . On the other hand , satel l ites may serve data disks to the cluster, as well as provide print or batch resources. The satellites are configured by the cluster manager tO best meet the needs of the a pp t ication .
To date, only members of the Micro VAX I I fam i ly of systems and workstations (MicroVAX I I ,
VAXstation I I , VAXstation IljG PX, MicroVAX
2000, and VAXstation
2000 systems) can be satellites. This restriction results from the need for specific code tO be written to support remote booting for the CPU and Ethernet adapters. Satel l ite support for other CPUs (both new and exist ing) will be considered in the future .
Disk Access
I n a CI cluster, the HSC disk controllers connect tO the CI bus in the same manner as do the VAX systems. 1/0 requests originating i n any VAX CPU are passed to the disk c lass driver (DUOR IVER) , which encodes them i nto MSCP packets. These packets are sent over the Cl network to the appro priate HSC controller for execution . All VAX
CPUs i n the cluster therefore have equal access to the HSC controllers and the d isks connected to them. However, an HSC control ler cannot connect to an Ethernet . Therefore, some other method is needed in a lAVe to allow disk access to a l l systems.
In the absence of HSC controllers, each disk must be connected to the system by some con troller, such as a
UDA, KDA, or UNI BUS con troller. Making these disks accessible to other
VAX systems in the cluster requ ires a software emulation of the HSC controller. This need is filled by the MSCP server software .
The VAX CPU originating the I/0 request merely sends an MSCP packet over the network tO the target VAX CPU with the desired disk. The packet is identical to the one DUDRIVER wou ld have sent to an HSC control ler. The MSCP server software on that target CPU receives the packet, performs the operation, and returns the resu lts just as an HSC wou ld do. The class driver on the originating VAX cannot tel l the difference between the MSCP server and an HSC controller.
The result, as shown in Figure 2, is that d isks served by the MSCP server appear to be equally avai lable to a l l systems in the cluster, i ndepen dent of which system they are actually cabled to and the type of interconnect.
System Management
The lAVe configurations described above were designed so that all system management activities would take place on the boot member. Although the cluster can be configured differently, that configuration is the simplest. It is also what most users would want when the sate l l ites are personal workstations.
The VMS, satel lite system, and application soft ware i nstallations are all controlled by command procedures executed on the boot member. Disk backups are done mostly on the boot member, on which the backup device (usually tape) is located. Data disks can be located a nywhere i n the cluster. If the sate l li te i s a single-user work station , we recommend that applications and user data not be put on a ny of i ts disks. Using a work station's local disks on ly for page and swap files eliminates the need for backups, thus freeing the owner of all system-management responsi b i lities.
The overall product simplicity goal is clearly facilitated by configuring the cluster in this man ner. Al l management activity is local to one sys tem and remains u nder the control of a limited number of people. Cl uster users should have no
Figure
2
HSC
Disk Access
5 8
Digital Technical]ournal
No. 5
Septem be•·
1 98 7
more system-management responsibilities than users of dumb ( e . g . , VT2 20) terminals wou ld have.
LAVe's
U
se of the Ethernet
The Ethernet is used as the cluster communica tion mechanism because it is compatible with the LAVe's requirements for cost and system envi ronment (non-computer room ) . There are , how ever, significant tradeoffs i nherent in substituting the Ethernet for the C I bus. For example, commu nication over the Ethernet is slower and more
CPU intensive than over the C I bus. The Ether net's advantages are lower expense, much greater geographic distance, and the abil i ty to connect many more systems.
The VMS port driver that provides reliable cl uster communication utilizing the Ethernet is cal led PEDRIVER. I t provides communication in such a way that the rest of the VMS software is unaffected . This section describes PEDRIVER's role withi n a LAVe, the PEDRIVER protocol , and some technical details about its internal structure.
The PEDRIVER
Communication services within a VAXc lusrer sys tem are described by the System Communication
Architecture, or SCA.3 The SCA model consists of the four layers shown i n Figure
3 .
The system application (SYSAP) layer consists of users of the connection services provided by the systems communication services (SCS) layer.
Examples of SYSAPs are the disk class driver
(DUDRIVER) . the MSCP server , and the cluster connection manager.
The SCS layer provides network resources to the SYSAPs . It multiplexes the u nderlying com munication service , provided by the port-to-port communication layer, into several connections .
These connections Jink a number of enti ties, i ncluding the connection managers between two
SYSAP SYSTEM APPLICATIONS LAYER scs
PPO
SYSTEMS COMM U N ICATION SERVICES LAYER
PORT-TO-PORT COMM U N ICATIONS LAY E R
P I
PHYSICAL I NTERCONNECT LAYER
Figure
3
SCA
Layers members, the class driver to the MSCP server (or
HSC device) , and so forth . The SCS layer a lso pro vides flow control , buffer management, notifica tion of new SYSAPs registering with it, and notifi cation o f connection breakage .
The port-to-port communications ( PPD) layer maintains a single communications pat h , called a virtual circuit, with every other VAX system or
HSC controller in the cluster. On a C I cluster, this layer is the lowest software layer within the
VMS system . I t is implemented by the C I port driver, called PADRIVER. PADRIVE R knows how to interface with the C I adapter and is responsi ble for discovering new nodes, forming virtual circuits with them, detecting communication fai lures, and signal ing these events to the SCS layer.
I n a LAVe, PEDRIVER provides much of the same PPD functiona l i ty as does PADRIVE R . Since the Ethernet hardware offers only a datagram ser vice (instead of the rel iable communication path offered by t he CI bus) , PEDRIVER uses a net working protocol to provide a rel iable com muni cations service. Unlike PADRIVER, PEDRIVER is device i ndependent, utilizing an underlying datal ink driver to control the Ethernet adapter.
The physical interconnect (PI) layer repre sents the medium over which packets are sent and received . A complete specification for this l ayer includes the mechanisms for clocking bits on the wire , the framing of bits into bytes and bytes into messages, el.ectrica l signal require ments, cabl ing, and so forth .
Ports
A port is a software interface between the port driver and a communications entity, usual l y an adapter. A port is i m plemented as a set of queues whose use is rigorously defined . Access to these queues is by means of interlocked i nstructions; thus no other synchronization mechanisms are required. The port d river manages the port . The d river receives requests from the SCS layer, for mats them, then passes them across the port by l inking a packet in a prioritized command queue.
The driver then sets a control bit to inform the port of this action . The entity behind the port dequeues the command packet, executes it, and either returns it to the driver with a status mes sage or p laces it in the appropriate free queue.
Packets being del ivered across the port to the driver are linked into a response queue.
An inter rupt is generated if the queue was previously empty.
VAXcluster
Systems
Digital Tecbnicaljournal
No. 5
September 1987
59
Local Area V AXcluster Systems
In the CI case, this port structure is used to communicate between PADR IVER and the Cl hardware. The hardware guarantees the delivery of sequential messages. It a lso moves user data into or out of the v i rtual address space of a target node during block transfers. Thus the CPU over head is kept to an absolute minimu m . The CI adapter is intel ligent enough to perform these functions on its own and to interrupt the CPU when the operation is finished .
Ethernet adapters do not fit this mode l . They are typica l ly packet-oriented devices that trans mit or receive usi ng discrete , l i mi ted-size buffers. The adapters do not guarantee sequential delivery. Since VAXcluster systems requi re these features , they must be replaced with software, at a corresponding i ncrease in CPU overhead.
To preserve the same port interface, however, we put the software providing these services below the port interface. The port then becomes an i n terface between SCS and a port driver above the port , and a port emulator below. Preserving the same level of fu nctional i ty at the port inter face e l i minated the need for extensive software modifications to the SCS and h igher software lay ers. Figure 4 shows the port structure for both the CI and Ethernet cases.
SYSAP SYSAP
SYSAP
�
I
/
SYSTEMS COMMUNICATION SERVICES (SCS)
C l
PORT DRIVER
(PADRIVER)
ADAPTER
C I CABLE
I
ETHERNET
PORT DRIVER
'---- - - - - -
PORT EMU LATOR
I
DATALINK
DRIVER
I
ADAPTER
PEDRIVER
ETHERNET
CABLE
PEDRJVER Functions
PEDRIVER is used i nstead of PADRIVER as the port driver in a lAVe . PEDRIVER contains two major segments: a port manager that receives packets from SCS and queues them to the port, and a port emu lator that operates below the port interface . This port emulator effectively emu lates the behavior of the CI hardware, uti l izing a sti l l lower l evel datal ink driver for access to the Ether net adapter, as shown in Figure
5 .
Since the port emu lator is the key to the lAVe's use of the Ether net , its design and implementation will now be described in deta i l .
NJ-SCA i s the name of protocol used by the port emu lator tO com municate with its peers on other nodes. This protocol extends the SCA so that sys tems can be connected by the Ethernet (also known as the NI) . This extension is achieved a t the cost o f reduced CPU efficiency, since the soft ware is doing more work, and lower 1/0 band width, si nce the Ethernet is slower than the CI bus . In addition , the publ ic access nature of the
Ethernet introduces security and configuration problems not encountered on the CI bus.
Major Objectives
The goals of the NI-SCA port design are
•
•
Compati bil ity - The i nterface to the N I-SCA port must have a strong resemb lance to that of the Cl port tO minimize the i m pact on the sys tem software directly using the port . In partic u lar, the functions required by the SCS layer and provided by the port shou ld be opera tionally equivalent to their Cl port counter parts so that the SCS layer need not be changed .
Performance - The port arch itecture has tO address rwo performance problems. First, the low Ethernet bandwidth may very we l l be a bottleneck i n some configurations, especially
DU DRIVER scs
MSCP
SERVER scs
PEDRIVER PEDRIVER
DATALINK
'- - - - - - - - - - - - - -
DATALINK
PHYSICAL ETHERNET CABLE
ADAPTER ADAPTER
Figure 4 VAXcluster Software Structure
Figure 5 Protocol Layering
60 Digital Technical Journal
No. 5
September
1 987
as CPU speeds increase . Second , the low band width affects both the aggregate t hroughput and the response rime between a transmitted message and the subsequent response.
•
Securi ty - Provisions for authenticati ng remote nodes are required. (Software data encryption is not currently part of the port design .)
•
Simplicity - The port architecture shou ld be defined so that implementations may substi tute performance for simpl ici ty . Ports imple menting differen t subsets of the architecture must be able to communicate with each other.
Differences between the CI Bus and Ethernet
The NI-SCA architecture must address several areas that result from the fundamental differ ences between the CI and E thernet buses and their existing adapters .
•
Locating other nodes - The CI pol ling for t he existence of other nodes does nor work in the larger Ethernet environment.
•
Data transport - The N I-SCA port emu lator must make t he data transfer l i m i tations of the
Ethernet transparent. Data segmentation and reconstruction must be handled efficiently.
•
Multiple paths - Any given node may i nteract with more than one Ethernet through more than one Ethernet adapter. The port emu lator must allow an i mp lementation to exploit such configurations transparently to achieve the requirements of efficiency and redundancy.
The current i mplementation of PEDRlVER does not support this.
•
Detection of communication fai l u res - The port emu lator must detect node or communi cations fail ures and signal t hem to the SCS layer.
•
Ethernet coexistence - The N I-SCA protocol must allow mul tiple clusters to coexist on the same Ethernet and to share that Ethernet with other network protocols.
•
Security - Secure com munication between nodes must be addressed since the Et hernet spans a wider and less secure environ ment than does t he CI bus, which is typical ly pro tected by the securi ty of t he computer room .
Locating Other Nodes and Virtual
Circuit Formation
The address space on the CI bus is currently implemented as a fou r-bit fiel d . The resu lting maximum of 1 6 possible addresses and t he limi tation of one cluster per Cl bus makes pol ling all possible addresses to locate other nodes an attractive solu tion . Poll ing is clearly not prac tical on the Ethernet, however, where there are
2 4 7 possible addresses, mul tiple cl usters, and nodes totally unrelated to c lusters.
PEDRIVER replaces the CI bus poll ing with a mul ticast scheme to a cluster-specific mul ticast address. A large block of consecutive multicast addresses have been reserved for N I-SCA. The lowest address in the block is hard coded into
PEDRIVER. During instal lation, the user assigns a group nu mber to the cluster. PEDRIVER adds this group number to the base address to generate that cluster's unique multicast address within NI
SCA's reserved block.
PEDRIVE R enables the reception of this mu lti cast address and transmits a HELLO multicast to it every three seconds. PEDRIVER will attempt ro create a circuit u pon receiving a HELLO message from a node with which it does not currently share an open virtual circu i t . HELLO messages received from nodes with a currently open vir tual ci rcu i t indicate that the remote node i s sti l l operationa l .
A standard three-message-exchange handshake is used to create a virtual circu it, as shown in
Figure 6.
The START_VC and START--ACK contain infor mation about the transm itt ing system , and what i t bel ieves the cluster password to be . These parameters are verified at the receiving system, which continues the handshake only if its verifi cation is successfu l . Thus each system authenti cates the other. After t he fi nal ACK message, the virtual circuit is open for use by both systems.
TRANSMITTING
SYSTEM
RECEIVING
SYSTEM
START_VC
START_ACK
ACK
Figure 6 Standard Handshake
VAXcluster
Systems
Digital Technical journal
No. 5
September
1987
6 1
Local Area VAXcluster Systems
Data Transport
PEDRIVER uses the virtual circuit to provide t he three SCA port data transfer services described below. The SCS layer does not need to disti ngu ish between the C I hardware or the NI-SCA port emu lator version of these services.
1 . Datagrams - Packets to be delivered on a
"best effort" basis. No guarantees are made about delivery, sequential ity, or replication.
2 . Sequenced messages - The port guarantees the sequential del ivery of exactly one copy of the packet.
3 . Block transfers - The port moves a large amount of data i n either direction . Segmenta tion, handled below the port, is invisible to the port driver and everyth i ng above i t .
Datagrams are sent as Ethernet packets, which are sufficient since no delivery guarantees are assumed .
PEDRIVER uses a standard networking protocol to provide rel iable communications when neces sary. A sequence number is included in each packet so that lost or out-of-sequence packets can be detected . Each packet requiring reliable del i very must be acknowledged by the receiving port emulator. To improve efficiency, several packets may be sent without waiting for an ACK.
Whenever possible, the recipient wil l also bun dle the ACK i nto a message to be sent back to the original source, thus saving the cost of an explicit
ACK. Timers are used i n both the source and des tination systems to generate a retransm ission if an
ACK does not arrive after a specified time period has elapsed. These ti mers also i n i tiate t he trans mission of an explicit ACK in the absence of any reverse traffic.
To send relat ively smal l amounts of data, SYS
APs use sequenced messages, generally holding up to about 1 2 0 bytes. PEDRIVER sends these messages with a sequence number over the vir tual circui t , and they must be acknowledged by the recipient as described above . PEDRIVER can therefore guarantee reliable message delivery to the desti nation SYSAP.
To send large amounts of data, SYSAPs use block transfers. In a VAXclusrer system, the disk class driver and the MSCP server use block transfers to move data being read from or written to a disk. PEDRIVER's port emu lator i m ple ments block transfers by segmenting the data in
1 3 00-byte chunks. Each chunk is copied out of the source buffer into a datal i n k packet and trans mitted over the virtual circuit as a sequenced message . The receiving port emulator copies the data out of the Ethernet packet into the user's buffer . The virtual circu i t guarantees the sequen tial delivery of these packets, thus mai ntaining data ordering and integrity.
The CI adapter can copy data i nto or out of the virtual address space of a target node by using direct memory access (DMA) . Thus the
CPU is not involved in block transfers . Ether net adapters, however, access data in specific buffers; therefore, PEDRIVER must copy data using a MOVC instruction . This scheme adds a lot of CPU overhead to
Ethernet block transfers.
Detection of Communication and
Node Failures
Communication can be lost between nodes for several reasons: a node shutdown, a system crash, or a hardware fai lure . PEDRIVER must detect these events and signal their occurrences to the
SCS layer.
A system genera lly transm i ts a node-stop (or last gasp) datagram upon learning it will shut down. This shutdown could be a planned event by an operator or a system software crash . The
SCS layer acts upon a received node-stop data gram. SCS breaks all connections with SYSAPs on the originating system and tell s PEDRIVER to break the virtual circuit. Cluster reconfiguration occurs much faster when a last-gasp datagram is received because no time-outs are required.
Communication can be lost, however, without the receipt of a node-stop datagram . Both a hard ware fai l ure and tripping a syste m 's halt swi tch wi l l break contact, or the node-stop datagram coul d be lost on the Ethernet. Therefore, other ways of detecting a breakage are needed. In gen era l , PEDRlVER detects a breakage by checking for the HELLO multicasts being transmitted every three seconds. One eight-second timer checks for the arrival of H ELLO messages for all virtual cir cuits. If two ticks of this timer (eight to sixteen seconds) occur withou t receiving a HELLO mes sage from a system, that system is assumed to have failed. The SCS layer is then notified of this occurrence.
Certain hardware fai lures may cause a node to continue send ing but to be unable to receive
HELLO messages. Therefore, sti ll another fai l ure detection method i s used : t h e counting of retransmission attempts for a sequenced packet.
6 2
Digital Technical journal
No.
5
September
1 987
If a send ing node makes 30 attempts (at one-sec· ond intervals) withou t receiving an ACK , the reci pient node is presumed dead and SCS notified of the fai lu re .
Sharing the Ethernet
The Ethernet is designed as a shared-communica· tions bus. Any NI-SCA architecture that precludes its use by other clusters or networks is un acceptable.
Multiple LAVes coexist on the same Ethernet by using different group numbers. Thus each LAVe uses different multicast addresses to transmi t and receive its HELLO messages.
A."> a result, it does not " hear" messages from other LAVe 's or attempt to form virtual circ u i ts with them. Multicast mes sages on one Ethernet a re not passed to other Eth· ernets that are l inked by means of traffic routers or gateways utilizing other commun ica· tions media. Therefore, group nu mbers must be unique only on each Ethernet. Different clus ters on other Ethernets may use the same group number. The grou p-number space is large enough so that ranges of numbers can be given to different branches of a business organization, thereby reducing the need for networkwide administration.
Nl-SCA is registered as Ethernet protocol type 60-07. This registration allows the datal ink driver to d istinguish NI-SCA packets from those sent by the DECnet, LAT, or other protocols.
PEDRIVER's use of the Ethernet has no effect on any other protocol , regardless of how the packets are mu ltiplexed on the single Ethernet.
Security
The VAXcluster system itself is one VMS security domai n. Al l the security control and alarm fea tures in the VMS system work on a cl usterwide basis. These features can be used with an appro priate degree of physical security (around the systems and Ethernet cable) to achieve a desired level of overall securi ty.
Unauthorized systems are prevented from join· ing the cl uster because a cl uster password is required to establish communications. That pass word is validated by both nodes during the ini tialization handshake to create the virtual circui t .
The password prevents a n unauthorized user from booting off a privately created local d isk with a local au thorization fi le (i nstead of a boot member) and joining the cluster . Satel l ite sys tems booting off the boot member must have been configured i n to a database by the system manager, effectively authorizing t heir entry into the cluster. A means is also provided tO prevent users from performing conversational bootstraps tO alter system parameters.
Ethernet cables are subject to unauthorized taps and eavesdropping. The LAVe assumes the presence of an appropriate level of physical secu rity around the systems and Ethernet cables, as these problems cannot be solved in software.
Encryption hardware is the only tru ly effective counterweapon to these attacks. Exploiting the vulnerabi lities of Ethernet in the absence of encryption could be done, but it wou ld requ ire substantial time , energy, and expertise .
Internal Structure of PEDRIVER
When extending SCA to include the Ethernet, we found the layering of the original model to be somewhat inconvenient. For one thing, the PPD layer performed too many functions to be thought of as a single layer . This problem was fur ther compounded when additional functions, such as node authentication , were included .
Therefore, the approach taken was to adhere gen erally to the original mode l, but to replace the
PPD and PI layers with several layers.
In the NI -SCA model , the PPD layer was replaced with the layers from the port command i nterface (PCI) to the datagram propagation
( OX) layers. The PI layer was replaced with the datalink and physical link ( PL) layers. The resul tant layering may seem a bit excessive - seven layers replacing two - but is nevertheless a nat ural part itioning of the activities below the SCS layer. Increasing the nu mber of layers for NI -SCA does not increase the intrinsic complexity of the port; it merely faci l i tates the port's description .
The new Nl-SCA model is shown in F igure 7 , together with a brief description of each new layer.
The Port Command Interface
(PC!) Layer
The PCI layer effectively i mplements the port by defining the in terface between the port and the port driver. Normal ly, the modu les of a given layer com municate with modu les in the corre· sponding layer on remote nodes. Lacking this characteristic, the PCI is not a layer i n the strict sense of the word but is merely an interface between the SCS and the port-to-port commun i · cations (PPC) layers.
VAXcluster
Systems
Digital Technical journal
No. 5 September 1!)87
63
Local Area VAXcluster Systems
SYSAP SYSTEM APPLICATIONS LAYER scs
PCI
PPC
SYSTEMS COMMUNICATION
SERVICES LAYER
PORT COM MAND
INTERFACE LAYER
PORT-TO-PORT
COM M U N ICATIONS LAYER
TRANSPORT PACKET SEQUENCING AND
ACKing LAYER
CHANNEL
CONTROL
NODE AUTHENTICATION,
TOPOLOGY CONTROL,
UNSEQU ENCED DATAGRAM
SERVICE LAY ER ox
DATAGRAM PROPAGATION LAYER
PPD LAYER
IN
SCA
DATALINK DATALINK CONTROL LAYER
PL PHYSICAL LINK LAYER
PI LAYER
I N SCA
_L
Figure
7
NI-SCA Layers
The PCI layer is the set of queues used to pass command packets down to and response packets up from the port emul awr. Each packet consists of two regions:
•
The port i n terface region i s comprised of com mand and status information between the port and the port driver. The specifics of this region are private tO PEORIVE R .
•
The P P C region is comprised o f t h e i n forma tion used by the l ocal P PC layer to commu n i cate w i t h a remote P P C layer. The specifics of this region are not private to PEORI VER since the region is i n terconnect i ndependent. The
P PC region is the same for the Ethernet as it is for the CI bus.
The Port- to-Port Communication
(PPC) Layer
The P PC layer exists below the port i nterface .
This l ayer provides port services ( datagrams, sequenced messages, and b l ock transfers) co the
PCI layer by translating between PCI packets and a series of PPC messages exchanged with the remote port. The PPC layer a lso segments block transfers into a series of sequenced messages. The datagram and sequenced services provided by the transport layer are used to exchange these mes sages. To be consistent with the CI bus , any errors detected at the PPC layer in a packet sent or recei ved in sequenced mode cause the virtual circuit to be d isconnected .
The Transport (TR) Layer
The transport layer uses one or more paths to the remote node to provide the local PPC layer with a sequenced- message and datagram connection to a remote PPC l ayer. For datagrams, the transport layer is l ittle more than a conduit to the channel control layer. For sequenced messages, the trans port layer handles a l l the sequencing, sending and receiving ACKs, and retransmissions required to provide guaranreed message del ivery and sequcncialicy. Although mu ltiple Ethernecs are not currently supported in a c luster, this layer would be responsible for that functionality.
The Chan nel Control (CC) Layer
A channel is a path that util i zes a single Ethernet to join two ports with an authorized datagram service . To accomp l ish that service , che channel uses the datagram service provided by the OX layer. The channel control layer manages the net work topology and therefore provides such ser vices as node authentication , access control , and virtua l circu it initial ization .
The Datagram Exchange (DX) Layer
The OX layer attempts co transmit packets from the source port to the destination port . On any given system, the OX layer is the interface between the ports and the datal i n ks.
As such, this layer is basica l ly a swi tch; many ports may be above it, many datal. i n ks be low it. Note chat on a si ngk system, the OX layer may be shared among mu ltiple ports and is not owned by any one pore .
The OX layer determi nes which systems are on which Ethernet and t ransmi ts packets correctly to t heir dest inations by managing the group num ber and mu lticast HELLO messages. This layer includes the group nu mber in a l l the packets it transmi ts and checks the numbers on received packets .
The Data/ink Control Layer
'fhe datal i n k l ayer provides access tO the p hysical l i nk ami the functions at the packet leve l . These
64
Digital Technical journal
Nu. 5 St!plem ber l .'.J8 7
functions incl ude the hardware adapter contro l , the minimum and maximum length req u i rements of packet, provisions for data-i n tegrity checki ng, data l i n k header formats , and multicast add ress ing. For NI-SCA, this layer is provided by a sepa rate datal i n k driver. This drive r controls the Eth ernet adapter hardware and is shared by a l l
Et hernet users (LAVe , DECnet, LAT systems, etc.) on the system .
The Physical Link (PL) Layer
The PL layer represents the med i u m over which packets are sent and received. A complete spec i fication for this layer would i nclude the mecha nisms for clocking b i ts on t he wire, the framing of bits into bytes, electrical signal requi rements, cabling, and so forth. For Nl-SCA, this layer is defined by the Ethernet standard .
Network Booting of the VMS Software
Two LAVe requi rements are met by booting the
VMS software over the Ethernet: simplifying sys tem management by requi ri ng only one VMS sys tem disk, and making possible diskless systems.
The software engi neering effort required during
LAVe deve lopment w provide t h i s functional ity was second on ly to that needed to develop
PEDRIVER.
Normal VMS Booting
Booting a system on a VAX processor takes p lace in several stages . Eac h stage is characterized by a loaded program that performs some prescribed fu nction, which i n turn loads and transfers con trol to another program .
The first such program tO ru n i s the console program, which is d i fferent on different proces sor types. Its basic rol e with respect to booting is to retrieve the input parameters, stOre them in the first six general-purpose registers , and then load and transfer control to VMB. VMB , referred to as either t he primary bootstrap or primary loader, is the first program that is more or less common across a l l processor types . Depending on the processor type , VMB is retrieved ei ther from ROM (the M icroVAX 11 class of systems) or the console block-storage device (other VAX systems) .
Although the parti tioni n g of work between the console program and VMB d iffers s lightly with processor type , together they accomp l ish the following:
•
Locate a block of memory tO use during the boot
•
•
Locate and establ ish an access path tO the sys tem disk
Provide a pri mitive r;o system consisting of a boot driver for the system device , a fi le system , and the S Q I O access routi ne
•
Loca te, load , and transfer control to the sec ondary bootstrap, ca1led SYSBOOT. EXE for the VMS system , or DlAGBOOT. EXE for diag nostics
SYSBOOT is the secondary bootstrap selected tO run when VMB is d irected to load the VMS soft ware. SYSBOOT performs the fol l owing actions:
•
Loads the VMS i mages i nto memory
•
Reads the system parameter fi l e , accepts any user speci fied parameter changes if this is a conversation boot , and configures the system accord i ngly
•
Al locates memory for and loads the termi n a l and system d isk drivers
•
Transfers control to t he ! NIT module of the
VMS system
The VMS I N IT module in itializes the now run ning VMS system .
•
•
Loads the processor dependent code (SYS
LOAxxx) and other loadable components i n to memory
Copies the boot I/0 routines ro the nonpaged pool for use duri ng any system crash
•
•
Tries to form a new VAXcluster system or join an existing one i f the parameters are set tO do this
Transfers control to the system scheduler to i n itiate process execution
Remote Booting Requirements
The actions performed during each of the three stages of a network boot are the same as those in a local disk boot. No mod i fications were req u i red i n the fu nctional operation of these programs.
What was needed was the ability to contend with an Ethernet l i nking the booting system with i ts system disk. The Ethernet has tota lly different characteristics than those of the block-structured d isk device previously presen t . The plan , then, was tO load a piece of software that makes the
Ethernet l ook l i ke a disk, rhus enabling t he rest of the VMS boot sequence to proceed normal ly.
VAXcluster
Systems
Digital Technical journal
No. 5
Septembe1·
1987
65
Local Area VAXcluster Systems
The three primary requiremems for the remote booti ng design and implememation were to
•
Change the existing boot p rocess as l irtlc as possible
•
Require no in itial state or comext information on the sate l l i te system
•
Work with the existing M icroVA.t'X II boot
ROMs (Req u i red hardware u pgrades in the fiel d wou ld make a LAVe much more difficu lt tO i nsta l l ) .
The existing boot ROMs on MicroVA.X I I sys tems i nclude an Ethernet device boot driver capa ble of transmi tting and receiving packets , p l us a
VMB program conta i n i ng the DECnet mai nte nance operation protocol ( MOP) . MOP locates a boot host system on the Ethernet network, uses a si mple, synchronous pi ng-pong protocol to copy an i mage from t he host i nto local memory, and then transfers control to that i mage .
The existing SYSBOOT program cou ld not be loaded d i rectly by a MOP exchange. SYSBOOT expects to be able to access the boot device as a block-structured srorage device; i t does not understand the various types of Ethernet adapt· ers that may be present. Moreover , SYSBOOT wou ld not have enough information to locate the system d isk. Therefore, a nother image cal led
NISCS_LOAD is inserted into the boot sequence between VMB and SYSBOOT. NISCS_LOAD pro vides the environment that SYSBOOT needs ro do its job correctly. As a resu lt, mi nimal mod ifica tions ro SYSBOOT and VMS I NIT were necessary.
Remote Booting Operation
The user starts the satell i te boot sequence with the appropriate BOOT command on the system console. From thereon , the process is automatic.
Satellite Operation during the
MOP Exchange
The VMB program i n the satel l i te system's boot
ROM interprets the boot com mand and attempts an Ethernet boot. VMB starts by transm itting a multicast message requesting an operating sys tem load . This message is mul ticast to an archi tectura l ly spec ified address because the ROM cannot have any knowledge of the network con figuration . Th is " please boot me" request is received by host systems on the E thernet that are wi l l i ng ro service network boots. I f t he request ing satel l i te is one that the host is wiJ!i ng to service, it responds to t he request with an "assis tance volu nteer" packet. The satel l ite responds to the first ''assistance volunteer" packet received and ignores any others. That response causes the host to send the N ISCS_LOAD i m age to the sate l l ite .
Boot Member Operation during the
MOP Exchange
The host side of the MOP exchange i s handled by the DECnet-VAX software , which must be ru n ning on t he boot member. Each boot member i n a l l clusters o n the Ethernet wi l l hear the operat i ng system request mu lticasts sent out by every sate l l ite. Other systems that are not boot mem bers will not have enabled reception of t h is mul ticast address.
The DECnet sofrware responds to an i ncoming boot request mu l ticast by extracting the source address of t he m u l t icast from the packet and searchi ng the node database for a matc h . This
48-bit hardware add ress of t he transmitting satel l i te is guaranteed to be u n ique on every Ethernet adapter. This add ress is not norma lly present in the database since it is not used for DEC net (or other) com munication under the VMS system .
Only those nodes that have been configured into the boot member's cluster by the cluster mana ger wi l l have the i r hardware address entered into the database. The request is ignored i f the multicast sou rce does not match a n address in the database. Therefore , sate l l i tes w i l l be booted on ly by a boot member in t he appropriate cluster.
I f the source address does match an address in the database , the DECnet software starts running the maintenance operations module (MOM) . This program handles the host end of the MOP exchange . MOM a lso looks up t he satel l i te in the node database to get o t her i nformation stored there, i nc l u ding the name of a load assist agent
( LAA) progra m . wh ich is used tO custom ize the load procedure for a LAVe . MOM cannot do this custom izing because i t is a general-purpose MOP faci li ty. MOM i nvokes the LAA by merging it i nto
MOM's address space and then cal l i ng i t .
The LAA was written specifical ly t o hand le the loading of N ISCS_LOAD. I.AA custom i zes the
NISCS_tOAD i mage for the booti ng satel l i te by appending necessary i n formation to it, including
•
The name and u n i t nu mber of t he sate l lite's system disk
66 Digital Technical journal
No. 5 September J <)87
•
The name of that sate llite's root direcrory on t hat disk
•
The cluster group number
•
The c luster password
•
A flag allowing or disa llowing conversational bootstraps
The NISCS_LOAD i mage and appended data are then passed to rout i nes within MOM that transmit them ro the sate l l i te using t he MOP protOcol .
When N ISCS_LOAD starts executing on the satel l ite, it can use this information for the next phase of the boot.
Mter NISCS_LOAD has been successfu l ly trans mitted, the MOP phase of the boot (and t he involvement of DECnet-VAX) is complete . The boot member no l onger knows that the sate l l i te is booti ng, and i t does not need to provide the satel lite with addit ional special services .
NISCS_L OAD, L oading SYSBOOT, and
VMS Software
The VMS system will not have been loaded into the sate l l i te when NISCS_LOAD executes. There fore , NISCS_LOAD is designed to run in a bare machine environment ; that is, NISCS_LOAD must be spec ifically programmed to handle any Ether net adapter or CPU i t is to support . To date, only support for the MicroVA.X II CPU has been included , along with the Q-bus adapter and the
MicroVA.X
2 0 00 and VA.Xstation
2 0 00
Ethernet adapters.
The NISCS_LOAD image contains four compo nents:
•
Datalink boot drivers for all supported Ether net adapters
•
A boot driver version of PEDRIVER, cal led
PEBTDRIVER
•
Pri mitive "class driver" MSCP code
•
Parameter values assembled by the load assist agent on the boot member
PEBTDRIVER retrieves the boot member's Eth ernet address, the group nu mber, and the cluster password from the NISCS_LOAD parameter list.
A virtual circuit back to the boor member is set up by transmitting a START_VC packet, which starts the normal initialization sequ ence . The boot member does not know that the system at the other end of this 'virtual circuit is booting since the virtual circuit and 1/0 requests sent over it are identical to those sent by a running
VMS system .
Upon setting u p the virtual circu i t , PEST
DRIVER has a path ro the system disk that
NISCS_LOAD will need to continue the boot.
The prim i tive class driver now issues a normal
MSCP command to read the SYSBOOT.EXE i mage from that disk into memory and transfer control to that image . PEBTDRIVER remains in memory to serve as SYSBOOT's "d river" for accessing the system disk, hiding all knowledge of the E thernet adapter. The presence of the primitive class driver makes SYSBOOT "see" the expected block-structured device interface . SYSBOOT can now load the VMS software normally by issuing a read operation over the virtual circu it set up by
PEBTDRIVER .
After being l oaded by SYSBOOT, the VMS sys tem can initialize normally because the E thernet path to t he system disk is tota lly h idden . No oper ational changes to SYSBOOT or VMS !NIT were necessary. The runtime PEDRIVER takes over from the boor driver during the initial ization of the VMS software, thus breaking the boot driver's virtual circu i t and establishing a new one.
The PEBTDRIVER portion of NISCS-LOAD remains permanently in memory. If the system crashes, that portion is activated again to write the contents of memory into the dump fil e . The runtime driver is not used because the state of the
VMS system, the drivers, and the data structures cannot be trusted in a crashed system . The boot driver is totally ignored while the system is u p ; therefore , its integrity is usually left i n tact by the crash .
As with any other boot driver, the system disk is the only known device. Therefore, the dump fi le must be on that disk.
Summary
We have shown how Local Area VA.Xcluster sys tems are a natural fol low-on ro the original VMS
VA.Xcluster im plementation using the CI bus . The cluster architecture and implementation were generally independent of the interconnect specifics; therefore, the switch to Ethernet was confined to the port driver layer. The replace ment of PAD RIVER with PEDRIVER and the addi t ion of Ethernet booting was all that was required ro make the prod uct work. This combining of
VA.Xcluster functionality with the MicroVA.X sys tems and workstat ions now available, plus the
VAXcluster
Systems
Digital Technical journal
No.
5
Septem ber I
'}87
67
Local Area VAXcluster Systems low cost and flexibil ity of the Ethernet, brings new power to low-end syste ms. These benefits i nclude both the data and resou rce-sharing capa bilities of VAXcluster systems, and the abi lity to isolate workstation users from system-manage ment responsibi lities .
The lAVe has a bright fu ture planned . Work is i n progress to al low both CI and Ethernet inter connects tO coexist in the same cluster. When this work is completed , workstation users wi ll be able to draw upon the power, resources, and speed of the large VAX machi nes , HSC con trollers, and disk fa rms in the computer room . In addition , users wi l l have fu l l access to the same data files as do users on those mai nframes. AU these systems wi l l be ru nning the same opera ting syste m, be centrally managed , be highly avai l abl e, and offer the same software environment to all users. No other prod uct comes close to offe r i ng such total system integration from the data center to the desk top .
References
1 . N. Kronenberg, H . Levy, W. Strecker, and
R .
Merewood , "The VAXcluster Concept:
An
Overview of a Distributed Syste m,"
Digital
Techn ical journal
(September 1 987 , this issue) : 7 - 2 1 .
2 . W. Snaman and D . Thiel , "The VAXjVMS
Distributed Lock Manager,"
Digital Techni cal journal
(September 1 9 8 7 , this issue) :
2 9 -4 4 .
3 . D . Duffy, "The System Communication
Architecture , "
Digital Technical journal
(September 1 98 7 , this issue ) : 2 2 - 2 8 .
68
Di?,ital Technical journal
No.
5 Sejilelllber
I
')87
Edward E. Balkovich
Prashant Bhabhalia
William R. Dunnington
Thomas
F.
Weyant
VAXcluster Availability Modeling
VAXcluster systems use redundant hardware-processors, interconnects, and storage el emen ts-and software to achieve high system availability.
No special hardware or software is required. A simple, first-order availability model is used to illustrate how this redundancy improves availability. Four VAXcluster configurations are analyzed to show that redundancy decreases system unavailability by two orders of magnitude.
Decomposition techniques were used to develop these first-order availabil ity models, which were then analyzed using "textbook" reliability analysis techniques. More complex configurations and models of broader classes of faults will require the support of more sophisticated modeling tools.
An increasing number of speciali zed computer systems are being dedicated to tasks that are critical to the success of an organization . For example, i n the financi a l services i ndustry or i n manufacturing, i t must be possible t o access a computing system to deliver a service or to man ufacture a product. Any loss of access to the com put i ng system adversely impacts business . The ability to access a computing system when it is needed (commonly referred to as avai labil i ty) is becomi ng an i m portant metric used to select such computer systems. Obviously, h igh avail abil i ty also i mproves the qua l i ty of service pro vided by general-purpose computing systems, such as those providing t i mesharing services.
VAXcluster systems provide high availabil i ty . 1 They can be configured so that there i s no single point of fai lure . Each c luster is a multiple computer system, built from standard hardware and software e lements. VAXcluster systems can be expanded in increments to provide the com puting power, data resources, and storage capa bilities typically associated with mainframe sys tems.
Although these systems are not fau l t tolerant, they can detect , isolate, and recover from faults in their processor, i n terconnect, and storage sub systems. (Fault tolerance generally implies that a recovery from a fault is completely invisible to an application. ) Wh i le VAXcluster systems can detect , isolate, and recover from faults, the recovery from some types of fau lts impacts the applications and their design . For example, a
VAXc luster system will retry an 1/0 operation i f a fau l t is detected in either the interconnect or storage subsystems.
The integrity of the ljO operation is ensured by the operating system. If a processor fai ls, how ever, the computations hosted by it are lost.
A user must start a new session on another (avail able) processor. The user must depend on an application, not the operating system, to recover the state of the computation to the point at which the fau lt occurred . For example, a journal file can be used to recover an editi ng session or data base transaction . In this case, the i n tegrity of the computation is assured by the application, not by the operating system .
This paper documents a study using simple first-order models to show how the inherent redundancy of VAXcluster systems is used to achieve high availab i l i ty . Although more sophisti cated models are possible, the models used in this study were sufficient to i l lustrate the main points. I t is assumed that the reader is fami liar with the basic technical concepts of VAXcluster systems presented in our companion papers.2.3 I t is not assumed that the reader is familiar with the standard methods of analyzing availab i l i ty used to i l lustrate the points of this study.
VAXcluster Structure
Figure I i l lustrates a simple VAXcluster system with terminals connected to the system via a IAT server. Either processor is accessible through that server, and dual-ported disks are accessible through either H ierarchical Storage Controller
(HSC) . The HSC devices and the processors are
Digital Technical journal
No. 5
September
I 987
69
VAXcluster A vailability Modeling
VAX
HSC
VAX
HSC
Figure 1 Simple VAXcluster Configuration connected by a Star Coupler, a passive device offering two independent datapaths between each node of the system . Multiple disks are used to shadow a volume of information. This simple system illustrates all the basic forms of redun dancy in VAXcluster systems.
Processor Failures
If a processor or its Computer Interconnect (CI) adapter fails, all computations in progress on that processor will be lost. The processor and the adapter can detect some types of faults and inform the VAXcluster system of them immedi ately. Other types of faults are detected by the other VAXcluster processors by way of time-outs.
When other processors detect a fault in a pro cessor or its adapter, they reconfigure themselves to remove the failed processor from the cluster.
The reconfiguration times depend on the number of locks in the system and on the number of 1/0 devices in the configuration. The average recon figuration time after a processor failure is a small number of seconds.
4
After the reconfiguration is complete, the user can begin a new session on the remaining processor. Appropriately constructed appl ications, such as those employing journaling, can then be recovered to the point of the failure.
Interconnect Failures
The Star Coupler, a passive device, has a negligi ble failure rate compared with the other ele· ments. The individual CI paths attached to a single adapter have active elements, however, and the failure rates for those paths must be considered .
If a single path fails, the Cl adapter will retry the transmission on the redundant path. The retry is invisible to both the processor and the HSC device using the adapter.
If both paths fail , neither the processor nor the
HSC device attached to the adapter can commu nicate with other elements of the VAXcluster con figuration . The effect is similar to a processor or
HSC failure. However, other processors and HSC devices can continue to communicate with each other.
Hierarchical Storage Controller Failures
HSC failures are managed by the VAX processors.
The HSC device can detect some faults and inform the cluster about them immediately.
Other types of faults are detected by the VAX pro cessors and the disks by time-outs. When a fault is detected in an HSC device, the VAX processors will retry any 1/0 operations in progress by using the redundant HSC device.
An
HSC failure is invisible to the process issuing the QIO opera tion. The times required to reconfigure the sys tem after an HSC failure depend on the number of outstanding ljO operations, the number of
1/0 devices , and the use of volume shadowing.
The average time is typically a small number of seconds.
Volume shadow sets, hosted by an HSC de vice, must be reconstructed if that device fails.
Although the shadow set is available during reconstruction, this process involves additional
1/0 that competes with user requests to read or write to the volume shadow set .
Disk Failures
HSC devices detect disk fai lures. Volume shadow ing allows an HSC device to retry a failed 1/0 operation using another member of the volume shadow set. The failure of a disk in a shadow set is invisible to the process issuing the QIO operation . When a fault is detected, the volume
70
Digital TechnicaiJournaJ
No. 5
September
1987
shadow set wi l l be reconfigured to remove the fai led volume . Once again , the average time requ ired to reconfigure the shadow set after a disk failure is a small n umber of seconds.
VAXcluster Configurations
Considered
Modeling Procedure
This paper focuses on the avai labil ity modeling of four simple VAXcluster configurations. The goals of the study were tO
•
•
Demonstrate the sensitivity of different reli abi lity and availabi lity parameters
Demonstrate how different types of redun dancy improve VAXcluster availability
These goals were achieved by first modeling the avai lability of a basel ine configuration con sisting of a VAX processor, an HSC storage con trol ler, and a disk drive. Each element in the con figuration represented a si ngle point of failure.
Next, redundancy in the form of a second VAX processor was added tO the baseline configura tion to create a second configuration . Another
HSC stOrage controller was then added tO create a third configuration . Finally, a disk drive and vol ume shadowing were added to create a fourth and fully redundant configuration . These four simple configurations were used to study the principal forms of redundancy in a VAXcluster system.
Referring to Figure 1 , the configurations con sidered here consisted of VAX processors, a Star
Coupler, HSC storage control lers, and disk drives; they did not i nclude the Ethernet , the LAT server, or the user terminals.
Baseline Configuration - Model 1
The baseline configuration, Figure 2 , consisted of a VAX processor, an HSC storage control ler, and a disk drive . The processor and the storage con troller were connected by way of a Star Coupler whose failure rate is negligible compared to that of the other elements. Figure 2 also shows the configuration diagram translated into a rel iability block diagram in which the series positioning of each element represents a single point of failure for the configuration.
Redundant Processor
Configuration - Mode/ 2
The second configuration considered in the study, Figure 3 , added redundancy in the form of a second VAX processor. The failure of either processor or its Cl adapter requires a failover process to the redundant processor with its asso ciated VAXcluster reconfiguration acttvtttes.
These activities usually complete in a matter of seconds.
In the reliability block diagram for the hard ware model, the redundant VAX processors are shown in parallel because both must fai l for the configuration to fai l . However, the HSC device
VAX
HSC
HSC
CON FIGU RATION
VAX
R E LIABI LITY BLOCK DIAGRAM
FOR HARDWARE MODEL
HSC
CONFIG URATION
VAXcluster
Systems
VAX HSC
RELIABILITY BLOCK DIAGRAM
Figure 2 Baseline Configuration (Model
1)
Digital Technical journal
No. 5 September 1987
RELIABI LITY BLOCK D I AGRAM
FOR RECONFIGU RATION MODEL
Figure
3 Configuration with Redundant
Processor (Model 2)
7 1
VAXcluster Availability Modeling and the disk drive are stil l shown as single points of failure.
If either processor fails, the VAXcluster system wi ll undergo a reconfiguration . Depending on the user application, the system may be unavail able during the fai lover process. 5 This condition is represented in the reliability block diagram by the two VAX processors in series.
Simi larly, the reconfiguration operation is re peated when a repaired VAX processor is re established in the VAXcluster system . Again, depending on the user appl ication, the system may be unavailable until the reconfiguration completes. Since either VAX processor coul d fai l , the reliability block diagram i s again valid for this cond ition.
Redundant Storage Controller
Configuration - Model
3
In the third configuration, Figure 4 , additional redundancy in the form of a second HSC stOrage controller was added to the Model 2 configura tion, which already had a redundant VAX proces sor. Now the failure of either a VAX processor or an HSC storage controller requires a failover pro cess to either the redundant processor or the con troller with the associated VAXcluster reconfigu ration activities.
When a repaired HSC storage controller is re-established in a VAXcluster system , there is no reconfiguration operation . Instead, the HSC device is placed in "warm stand-by" redundancy.
That is, the device is not actively re-established in the VAXcluster system unless the other HSC device fails. This situation contrasts with that of the active redundancy of the VAX processor, which is immediately reconfigured back into operation as soon as i t is repai red .
Fully Redundant Configuration -
Model 4
A fourth configuration, Figure
5 , added further redundancy in the form of a second disk drive and volume shadowing to the Model
3 configuration , which alre�dy had a redundant VAX processor and HSC storage controller.
In volume shadowing, write commands are applied tO all available volumes in the shadow set . Read commands are accompl ished using any avai lable volume. A fault in a disk causes it to be removed from the shadow set. A repaired vol u me is merged back into a shadow set by first copying the data from an available volume as a background activity. Only upon becoming identical to existing members of the set will the repaired vol ume again become a n avai lable member of the shadow set.
A detailed description and analysis of the
Model 4 configuration is given later.
Modeling Approach
Several formal definitions are needed to quantify
VAXcluster avai lability.
Availability is the proportion of time that ser vice is available from a VAXcluster system to per form a user appl ication.
I t is i mportant to remember that this definition of availability is a general one. As the nature of the application, the size of the VAXcluster config u ration, and the amount of redundancy change, availabi lity can be defined in more complex
CONFIGURATION
RELIABILITY BLOCK DIAGRAM
FOR HARDWARE MODEL
RELIABILITY BLOCK DIAGRAM
FOR RECONFIGURATION MODEL
Figure 4 Configuration with Redundant
Processor and Storage Controller
(Mode/ 3)
72 Digital Technical journal
No. 5
September
1987
ways. For the configurations used i n this study, at least one of each type of element must be running for the VA.Xcluster system to be operational.
Unavailability is the proportion of time that service is interrupted and that a VAXcluster sys tem cannot perform a user application .
I n this study, the related metric of downtime i n minutes per year w i l l be used rather than the sys tem unavailabi lity.
Reconfiguration time is the time taken to ini tially detect a failed element and remove it from the VAXcluster system. For a failed VAX proces sor, this time also i ncludes the time taken later to re-establish the repaired element's membership in the cluster.
CONFIGURATION
RELIABILITY BLOCK DIAGRAM
FOR HARDWARE MODEL
RELIABI LITY BLOCK DIAGRAM
FOR RECONFIGU RATION MODEL
Figure 5 Configuration of Fully Redundant
System (Model 4)
Note that the HSC device employs "warm stand-by" redundancy and therefore does not have any significant reconfiguration time associ ated with re-establishing membership in the cluster.
VAXcluster reconfiguration activities usually complete i n a matter of seconds; however, in extremely rare cases, much longer times are possible .
Overview
The most common approach to model ing com plex systems consists of structurally d ividing a system into smaller subsystems, such as proces sors, controllers, and d isks.6 The availability of each subsystem is then analyzed separately, and the i ndividual subsystem solutions are combined to obtai n the system solution . One important assumption must be made to achieve a solution: · the behavior of each subsystem must be indepen dent from that of any other subsystem .
Furthermore, a decomposition technique can be applied to certain behaviors that cause system outages due to failures in redundant subsystems.
In these cases, the recovery to an operational sys tem happens q u ickly . Similar behavior is also present when the failed subsystem is repai red and is ready to rejoi n the system to make it a fully configured system. This type of decomposition is cal led behavioral decomposition.
With this approach to structural and behavioral decomposi tion , hardware fai lures and VAXcl uster reconfigurations are modeled separately. Such a decomposition allows the model to analyze both
VAXcluster reconfigurations and complete sys tem failures due to hardware fai lures. It also allows the model to analyze the sensitivity of sys tem availability to each factor.
In this study, availabil ity model i ng captured the following factors:
•
•
•
•
Hard fai lures requiring a repair cal l
VAXcluster reconfigurations during which the
VA.Xcluster system was assumed to be unavail able in this analysis
Response time for maintenance personnel
Time-to-repair
The following factors were nor considered
(except for the impact of reconfigurations due to hardware failures) :
•
I ntermittent failures
•
Transient failures
VAXcluster
Systems
Digital Technicaljournal
No. 5
Septem ber
1987
7 3
VAXcluster A vailability Modeling
•
•
Quorum disks
Operational errors
•
Software errors
•
The fol lowing modeling parameters were used :
The mean time-between-failures (MTBF) and mean t ime-to-repair (MTTR) of each of the fol lowing elements:
- VAX processor
- HSC storage control ler
- Disk drive
•
VAXcluster reconfiguration times caused by
- VAX processor failure
- Re-establishment of the repaired VAX processor into the VAXcluster configuration
- HSC storage controller failure
- Disk drive failure
•
Response time for maintenance
The remainder of this section describes i n detail the modeling of the fourth configuration
(Model 4 ) .
Analysis of Hardware Failure
Consider the structural decomposition of the
VAXcluster configuration. Three subsystems were connected in series, each consisting of two elements in parallel . At least one element in each subsystem had to be operational for the VAXclus ter system to be operational . The hardware reli ability block diagram is shown in Figure 5 .
Repairable systems are those for which an auto matic or manual repair can be made if an element fails. Assume that each element is subject to fail ure and has its own repair facility 7 I f the time-to failure of element with failure rate i is exponentially distributed
A1 , and the time-to-repair of ele ment rate i is exponentially distributed with repair f.Lt, the instantaneous availability can be obtained by the following equation:
As t approaches i nfinity, A1(t) approaches the steady-state availability and
A1 equals f.L;/(A; +f.L; )
.
The steady-state availability of a single element is given by the following equation :
A = f.L/(A + f.L) in which A is the failure rate of the element and f.L is the repair rate of the element. The time-to-fail ure and the time-to-repair are assumed to be exponentially distributed .
The steady-state availability of two elements in parallel iss
In Model 4 , the elements in each subsystem are two VAX processors, or two HSC storage con trollers, or two disk drives. Using the equation above , the availability of the processor subsys tem ,
Ap , can be expressed as
Sim ilarly, the availability of the HSC storage controller subsystem ,
Ah , and the avai lability of the disk drive subsystem ,
A" can be expressed as and
The aggregate availability of the VAXcluster system is
For exponentially distributed times, the fail ure rate,
A, is I jMTBF and the repair rate, f.L, is l jMTTR .
Analysis of Reconfiguration Times
Next, consider the behavioral decomposition caused by the reconfiguration that occurs when one element in a subsystem fails and an automatic failover tO a second (redundant) element takes place . During this process, a reconfiguration occurs when a failed element leaves the VAXclus ter system. For processors only, another reconfig uration occurs when a repaired processor later rejoins the VAXcluster system . Depending on the user application , the VAXcluster system may be unavailable to perform user applications during these reconfigurations .
74 Digital Technicaljournal
No. 5
Septem be•·
1987
For example, consider the following time l i ne : t,
,
,-t-----t.--'t
tz b t.
Figure 6
TIME
Time t1 to t2 is the VAXcluster reconfiguration time for a failed VAX processor to be detected and removed from the VAXcluster membership.
Time t2 to t3 is the repair time for the fai led hard ware element . Time t3 to t4 is the time for the repaired VAX processor to be re-established in the VAXcluster membership.
Figure
5 incl udes the reliability block dia gram representing the VAXcluster reconfigura tion behavior of the Model 4 configuration. Each subsystem is shown as two elements in series. If any single element is not operational, the sub system can be unavailable due to a VAXcluster reconfiguration.
For two elements in series, the availability is8
A = A1
X
Az
In model
4 , the elements in each subsystem are two VAX processors, or two HSC storage con trollers, or two disk drives.
Applying the equation above for elements i n tem ,
Ap, is
Note that for the VAX processor, the rate llp is the reciprocal of the sum of the times t
1 to t2 and t3 to t
1 .
Similarly, the availability of the HSC storage controller subsystem,
Ab , the disk drive subsystem ,
A,, is
A { llb
=
(J..b + llb)
}2 and
Ar = {(J..r�llr)f
The aggregate avai lability of the VAXcluster system is
As = Ap
X
Ah
X
Ar
Assuming an operation running 24 hours a day,
365 days per year, the downtime equals
(l -As)
X
5 2 5 , 600 minutes per year. This fig ure is the downtime caused only by reconfigura tions. The total downtime is the sum of the down time caused by hardware fai lures and the down ti me caused by VAXcluster reconfigurations.
Extensions to the Models
The simple mode.ls considered in this study can be extended in several dimensions.
The complexity of the configurations can be i ncreased either by adding more VAXcluster e le ments or by extending the bounds of the models to include the Ethernet and its attachments. A complex configuration could include multiple clusters and multiple Ethernet segments. More complex definitions of availabil ity are needed as the configurations increase in complexity. These definitions range from the single-user view to a measure of system productivity.
Only permanent
( hard) hardware failu res are considered in this study. I ntermittent and tran sient hardware and software fai lu res, as we ll as operational errors, can be added as extensions to future models. The downtime allocation reponed in the l i terature typically attributes about one third of the total to each of the hardware, soft ware, and operator-induced failures.9 This result includes the effectiveness of system recovery that can be hardware based, software based, or both.
Certain insidious failures can result in ineffec tive recovery, even in the presence of hardware or software redundancies. The term "fault cover age" represents the joint probability of fault detection and successful failover to a redun dant element. A fault-coverage factor of one is assumed in this smdy.
This study also assumes that the subsystems of
VAX processors , HSC storage controllers, and disk drives are independent. Relaxing this assumption adds to the complexity of the modeling approach. Similarly, a simplistic mai ntenance strategy is assumed in which each cluster ele ment has its own repair facility.
The extensions described above add more real ism to the model ing approach at the expense of added complexity in both model formulation and solution technique . Moreover, the textbook for mulae used i n this study are li miting and often inappropriate.
Markov modeling is a particu larly useful ana lytic technique for formu lating and solving these complex models_7 Simulation is an alternative but computationally less efficient technique.
VAXcluster
Systems
Digital Technicaljournal
No .
5
September
1 98 7
75
VAXcluster A vailability Modeling
Another valuable industry-wide tool is the Sym bolic Hierarchical Automatic Reliability and
Performance Evaluator (SHARPE) software . 1 0
SHARPE's hierarchical feature allows complex subsystem models tO be combined into a system model for efficient solution . SHARPE also employs state-of-the-art matrix-solving routines to solve large and often ill-conditioned problems arising from the Markov model formulation of these complex configurations.
Results and Conclusions
This section discusses the results of this study in detail.
The Impact of Initial Redundancy
I n Model
1 ,
In Model no redundancy exists in the syste m .
2 , the redundancy of the additional
VAX processor reduces the total downtime to
16 percent of the downtime in Model
1 .
I n Model 3 , the redundancy of an additional
VAX processor and an HSC storage controller reduces the total downtime to almost 7 percent of the downtime in Model
1 .
In Model 4 , the total redundancy of an addi tional VAX processor, an HSC storage controller, and a disk drive reduces the total downtime to slightly under
1 percent of the downtime in
Model l .
These results show that redundancy does work to increase the availability of the system . Figure
7 shows the effect on total downtime as different forms of redundancy are introduced. A fully redundant configuration reduces system down time by two orders of magnitude.
VAXcluster Reconfiguration Downtime
Figure 8 is an expanded view of the decrease in total downtime for the three models that i nclude
�
�
�
Cl
-' g redundancy. It also shows the contribution of
VAXcl uster reconfigurations to total downtime.
Here the typical duration of reconfiguration is used. Since Model 1 has no redundancy, the VAX cluster reconfiguration downtime is zero.
Impact of Increased Frequency of Reconfigurations
Since the previous results considered the fre quency of reconfigurations equal to that of hard ware failures, it was necessary to study the impact of an increased frequency of reconfigura tions on downtime .
Figure 9 shows the l inear relationship between reconfiguration downtime and an increase in the frequency of reconfigurations. It also shows the trend in the reconfiguration downtime as the duration of reconfiguration is first varied to three and then to six times the typical value.
As shown, the key to reduced downtime is keeping the dura tion and the frequency of reconfigurations as low as practical . High-reliability hardware is a major factor in keeping the frequency of reconfigura tions low.
Contribution of Individual
VAXcluster Elements
This study also examined how much downtime an individual VAXcluster element contributes toward the total downtime.
Figure
1 0 shows the contribution of each ele ment (CPU, HSC, and disk) toward the total downtime for Model
4 .
At a given MTBF, the VAX processor contributed 8 2 percent of the total
I
2
I
3
VAXcluster MODELS
4
Figure 7 Impact of Initial Redundancy
2 3
VAXcluster MODELS
4
KEY: c::J
RECONFIGURATION
HARDWARE
Figure 8 Total System Downtime by Model
76 Digital Tecbnical]ournal
No. 5
September
1987
MODEL
4
::2 i= z
3:
0
..
-
....
-
- -
- -
_ .,..-
.,.,... ....- - - .. - - - ·
_ _ _ . _ _ _ ...
2
4 6
FREQUENCY OF RECONFIGURATIONS
KEY:
---- 1
X TYPICAL
- - 3 X TYPICAL
--
6
X TYPICAL
Figure 9 Reconfiguration Downtime by
Frequency of Reconfigurations
8 arc generalized tO the (N of M) redundancy case.
The assumption is that N processors are required for capacity and
M processors represent
M -
N redundancy. The steady-state availability is defined as the probabi lity of at least (N of M) processors working. The cluster is assumed to be unavailable when less than N processors are working. Note that, depending on the configura tion and application, clusters with less than N working could be considered as partially avail able. The case of the partially available cluster is not considered here.
The (N of
M) availability , as defined above, is
Availability(NofM) =
,U-N
•
�
1�11
Ml i!(M - i) !
( ) (1 )
I
MODEL 4 downtime . When the MTBF of that particular VAX processor was i mproved, i ts contribution dropped to
5 7 percent .
Typical VAXcluster configurations would gen erally include more than the two disks used i n this study. Having more d isks wou ld change the contribution of the disk subsystem to the system unavailabil ity. (Analyzing the impact of addi tional disks is outside the scope of this paper.)
The reliability i mprovement in the MTBF of the
VAX processor decreased both the hardware and the reconfiguration downtime. Figure I I shows a decrease of approximately
5 8 percent in total downtime.
Hardware Downtime versus
Response Time
This study included a response time for mainte nance for each cal l as part of the recovery time. If an on-site maintenance person were ava ilable, the response time would be eliminated , thus speeding the recovery of a failed element. When this strategy is considered , the hardware down time drops by almost 60 percent. Figure I 2 shows this reduction as applied to Model 4 .
The N of M Redundancy Case
The results given so far have been for ( 1 of I ) and
(I of 2) configurations of VAX processors, stor age controllers, and disks. In this section, the hardware downtime resul ts for VAX processors
::2 i=
FIRST MTBF
KEY:
VAX c:::::::J
H SC c::::::J
DISK
Figure
1 0
Contributions of Individual
VAXcluster Elements to Downtime
MODEL 4
FIRST MTBF IM PROVED
MTBF
Figure
1 1
Total System Downtime by
VAX Processor MTBF
VAXcluster
Systems
Digital Tecbnical]ournal
No. 5 September
1987
77
VAXcluster A vailability Modeling
::! i= z
3:
An application of the (N of M) availability expression for VAX processors is shown in Fig ure 1 3 . The number of VAX processors required to run applications to capacity was set to 1 , 2 , 3 , and 4 . The values for M were set to N +
0 ,
N + I , and N + 2 . High availability is typically measured in values much greater than
0 . 9 9 .
Therefore, to distinguish the variation in availability, the origin in Figure 1 3 is not zero but much greater than
0 . 9 .
With no redundancy (M = N + O) , availabil ity decreases with an increase in the number of processors. That decrease occurs because more
CPUs must be available to deliver the applica tion, bringing about a greater Likelihood of fail-
ON-SITE MAINTENANCE
SERVICE CALL
RESPONSE TIME
Figure
12 Hardware Downtime versus
Response Time
1 .00
KEY: rfl
..-r.--
..-....-
I"""
�
....r-
..
I ·
I"""
:
,
�;
,
1 ' ..
1 2 3
4
PROCESSORS R EQU IRED FOR APPLICATION (N)
N + O c:::J
N +
1 c=J
N + 2
MODEL
4
Figure N of M) VAX Processor
Redundancy Case ure and outage . This result is shown in the graph by the downward trend of the " N + O" bars.
Adding a single redundant CPU
(M
=
N + 1 ) greatly i mproves system availability. Adding a second redundant CPU (
M
= N
+
2) has little additional effect on availability. The additional improvement is not visible on the graph , even with the expanded vertical scale. It can therefore be assumed that " N + I " redundancy is sufficient for most applications.
Summary
VAXcluster systems achieve high availability by eliminating single pointS of failure with redun dant hardware. Redundancy is introduced at the level of standard processors, interconnects, stor age elements, and software. No special-purpose hardware or software is required. The same hard ware and software could be used to construct a less available u niprocessor system without vol u me shadowing.
The simple analytic models of VAXcluster availability developed in this study show that redundancy yields dramatic improvements in sys tem availability for the system configuration shown in Figure 1 . The average downtime of the system is reduced by nearly two orders of magni tude from that of a similar u niprocessor system without volu me shadowing.
Because they can be expanded incrementally,
VAXcluster systems requiring a minimum num ber of N processors to achieve a performance goal can achieve significant improve ments in avail ability with the addition of a single redundant processor. There is no requirement to fully repli cate all the original N processors.
The system configurations analyzed in this study are simple ones designed to illustrate the most important concepts of VAXcluster systems.
The downtime of a more complex VAXcluster configuration, with many additional processors,
HSC devices, and disk drives, changes system downtime in complex ways. In genera l , addi tional redundant hardware causes multiple hard ware failures to become less of a factor. When faults do occur, however, time is required to reconfigure the system . Some applications may view these small reconfiguration times as a source of system downtime . I n such cases, addi tional hardware increases both the frequency of reconfigurations and their contribution to sys tem downtime . Continuing effortS to improve hardware reliability are particu larly i mportant to
78
Digital Technicaljounull
No.
5
September
1987
reduce the downtime due to multiple hardware failures and the frequency of reconfigurations that might be counted as downtime by an appli cation.
The analysis used in this study uses structural and behavioral decompositions of systems. Struc tural decomposition is the most common approach to modeling complex systems. How ever, this approach assumes that each subsystem behaves independently. For the systems and phe nomena considered in this study, recovery to an operational state happens quickly following a system reconfiguration caused by a fault in a redundant subsystem . Similar behavior is also present when a fai led VAX processor subsystem is repaired and is ready to rejoin the system .
These modeling approaches were applied to the VAXcluster system, which was considered to be repairable. Structural decomposition was used to model the hardware failures of each VAX pro cessor, HSC device, and disk drive in the system .
Behavioral decomposition was used separately to model the reconfiguration times.
Notes and References
1 . This paper is limited to Cl-based VAXcluster systems. Local Area VAXcluster systems, implemented with Ethernet, are not consid ered in this analysis. The reader should be aware that there are significant configura tion differences between Cl-based VAXclus ter systems and Local Area VAXcluster sys tems that lead to important differences in system availability.
2. N . Kronenberg, H . Levy, W. Strecker, and R .
Merewood , "The VAXcluster Concept:
An
Overview of a Distributed System," Digital
Technical journal (September 1 987, this issue) : 7-2 1 .
3 . VAXcluster Systems Handbook (Bedford:
Digital Equipment Corporation, Order No .
EB-28858-46, 1 986) .
4 . E . Los, S. Snaman , S. Szeto, and D . Thiel,
Corrections to "Cluster State Transitions,"
VAXcluster Systems Quorum , vol . 2, issue 3 (Digital Equipment Corporation,
February 1 987) : addendum .
5 . During reconfiguration , significant proces sor resources are used to reconstruct the
Jock manager database. Some real-time app lications may view the reconfiguration time as a system outage .
6 . S. Bavuso et al . , Dependability A nalysis of
Typical Fault- Tolerant Architectures Using
HARP, CS- 1 986- 1 8 .
7 . K . Trivedi, Probability and Statistics with
Reliability, Queuing and Computer Sci ence Applications (Englewood Cliffs: Pren tice Hal l , 1 982) .
8 . P. O 'Connor, Practical Reliability Engi neering (Chichester: John Wiley & Sons,
Ltd . , 1 985) .
9 . D . Siewiorek and R . Swarz , The Theory and
Practice of Reliable System Design
(Bed ford : Digital Press, 1 982) .
1 0 . R . Sahner and K . Trivedi, SHARPE: Symbolic
Hierarchical A utomatic Reliability and
Performance Evaluator, ( Durham : Duke
University Department of Computer Sci ence , September 1 986) .
VAXcluster
Systems
Digital Technical journal
No. 5
September
1987
79
System Level Performance of
VAX
8974 and 8978 Systems
Daeil Park
Rekba D. Von Ehren
Tzyh-jong Wang
Nii N. Quaynor
This paper describes the results of performance tests on the VAX
8974 and
8978 systems in two different situations: a scientific environnrent, and a transaction processing environnrent. Benchmarks were run in both envi ronnrents to collect application throughput, IjO activity, and other per formance data. The results of a VAX
8700 were used as a baseline compari son. Based upon measured data, two models, one for each environment, were constructed to predict system performance under different configu rations. These models were run with various parameters to construct per formance curves. Subsequent test results showed that both models pre dicted performance accurately. The
8974 performed
3.2 to
4 times faster, and the
8978, 6 to
8 times faster, relative to the
8700.
The VAX 8974 and VAX 8978 systems are power ful new systems based on Digital's VAXcluster technology. These systems consist of either four or eight VAX 8700 processors respectively, pack aged with an I/0 subsystem of storage controllers and disk arrays. This paper presents the perfor mance of the VAX 8974 and VAX 8978 systems in both a scientific environment and a transaction processing environment. For comparison, the corresponding VAX 8700 data is presented as the base-level performance.
The scientific environment was measured using multistream batch jobs. The transaction processing environment was measured using a multiuser interactive workload that simulated an order entry and inventory control system . The measured performance for both environments is presented in terms of user-visible performance , system behavior, and resource utilization of the appl ications.
Based on the measured data, performance mod els of VAX 8974/8978 systems u nder each of the two environments to predict the performance for different configurations. The construction of the model and some resul ts are discussed following each measured performance section .
VA.Xcluster Performance Overview
A VAXcluster system is a highly integrated organi zation of VAXjVMS systems can be viewed as a single-domain information management system .
It is a state-of-the-art distributed system provid ing fu ll data-sharing functions. All the accesses to files and records are coordinated by locking schemes implemented by the distributed lock manager l The distributed lock manager is a VMS feature that has been extended w provide syn chronized readjwrite resource sharing among the nodes in a VAXcluster system . Being a multicom puter system of a single management domain , a cluster offers increased availability and per formance .
The performance of a VAXcluster system can be observed at many levels, such as the Computer
I nterconnect (CI) and the System Communica tion Architecture .
2
The context used in this paper, however, is the system-level, or user-per ceived , performance. The questions that immedi ately arise about VAXcl uster performance are how it grows as add itional processors are added , whether the performance grows in a linear scale, and if not, what performance range is expected compared to the single-system performance.
There are two primary factors that affect the performance of a VAXcluster system : a communi cation overhead, and a locking overhead. The first factor is related w the management of the
VAXcluster system. It is the cost ro maintain the mu ltiple processors in an integrated system and includes such overhead as the compute time £O maintain the connections between the nodes. A communication overhead always exists in a VAX-
80 Digital Technical journal
No. 5
September
1987
cluster system , regardless of the applications and the size of the cluster, a lthough that overhead is general ly sma l l .
The second factor comes from sharing a resource clusterwide . Every access made to a shared resource by the processes must be regu lated by a certain synchronization scheme . In a
VAXcluster environment, this synchronization is implemented by using locks. A lock operation may involve sending and receiving messages between processors. A previous study shows that a lock request in a VAXcluster system may take seven times as long as that in a single VAXfVMS environment .-> Therefore, the performance of a
VAXcluster system w i l l depend upon the degree of data-sharing of a particular app lication .
This study has been conducted to understand what implications these factors , especial ly the locking overhead, have on the system-level per formance of a VAXcl uster system . The two appli cations used in this study show the extremes i n terms of degrees of data-sharing. The scientific workload had no files being shared by the pro cesses, whereas with the transaction processing workload, all the files and records are shared clusterwide by a ll the processes . The goal of this study was to find the relative performance range of a VAXcluster system across the entire applica tion space by tracing the performance of the two extreme applications d iscussed above .
Scientific Environment
Workload Description
The scientific workload , called SCIENCE, is a suite of multistream (homogeneous) batch jobs.
These jobs are we ll-known programs frequently used i n science and research environments. Four benchmarks commonly used in physics are ISA
JET and GEISHA, two Monte Carlo simulations used in high-energy physics applications, and
TAIR and 1WING , two tests used in aerodynamics applications. Three other programs used in chemistry are GAUSSIAN 8 2 , a quantum chem istry package ; MOPAC, a general-purpose semi empirical molecu Jar orbital package; and RS/ 1 , an interactive data ana lysis software package fre quently used i n chemistry labs.
Performance Metric for
SCIENCE Workload
The most important performance metric is throughput. Throughput is defi ned as the number of jobs that the system can process in a given time. This metric was derived in the fol lowing manner, using the elapsed times extracted from the batch log fi les. For a closed system with one job,
Throughput
=
1
.
Average elapsed ume
The following steps were used tO apply this equation tO the multinode, multistream system:
Average elapsed
_ time per job
Sum of elapsed ti mes for all jobs
Total number of jobs in which Total number of jobs
=
Number of nodes
X
Number of streams, and
Th h
_
The SCIENCE workload is a su ite of repre sentative programs, each yielding a throughput for each system. To compare the performance of systems u nder this workload, the multiple relative performances based on the individual throughput comparison have to be aggregated .
The geometric mean is chosen tO aggregate the relative performances, with equal weight on each program 4 ·5
Test Methodology
The basic methodology of this study was tO increase the load on the system gradually until the processors were ful ly uti lized , thus yield ing a peak throughput for a particular configuration .
Since all the benchmarks were run as batch jobs, this saturation was achieved using multistream batch jobs. Up to five batch streams on each pro cessor were run for each benchmark tested.
Potential 1/0 and memory bottlenecks were m inim ized by allowing large si zes of user work ing sets and by al locating one d isk per job stream for data and scratch fi les.
Hardware and Software Configuration
The hardware environment consisted of the fol lowing elements:
•
A VAX 8700 system with one CPU, two HSC70 stOrage control lers, and two SA4 82 storage arrays
•
A VAX 8974 system with four VAX 8700 CPUs, two HSC70 stOrage controllers, and six SA4 82 storage arrays
VAXcluster
Systems
Digital Technical journal
No. 5
Septem ber
1987
8 1
System Level Performance of VAX 8974 and 8978 Systems
•
A VAX 8978 system with eight VAX 8700
CPUs, fou r HSC70 storage controllers, and twelve SA4 82 storage arrays
The software environment consisted of the
VAXjVMS version 4 . 4 operating system and
FORTRAN version 4 . 3 .
Characterization ofthe
SCIENCE Workload
The seven benchmarks of the SCIENCE workload were grouped into two categories based on their
I/0 behavior. One group included the bench marks with virtually no I/0 activity; the other with those that generated some I/0 activity.
MOPAC and TWING both generate few IjOs, thereby falling i nto the first category. The re maining five benchmarks, ISA)ET, GEISHA, TAI R,
RS/ 1 , and GAUSSIAN 82 exhibit some I/0 activ ity. Among a l l , GAUSSIAN 82 is the most 1/0 intensive . MOPAC and GAUSSIAN 82 were chosen as being representative of each category. Before starting the experiments, we ran the representa tive benchmarks on a VAX 8700 system to study the characteristics of the system resource usage.
The following graphs give a profil e of the two cat egories in terms of these stud ies .
Figure 1 shows the profi les of MOPAC and
GAUSSIAN 82 in terms of processor utilization plotted against elapsed time . Note that a single stream of MOPAC saturated the VAX 8700 proces sor during the entire run of a l most 40 minutes, doing virtual ly no I/0. On the other hand , GAUS
SIAN 82 consumed the most CPU power in the first five minutes and then remained at a lower rate (67 percent) of CPU utilization for the rest of the run time . For the first five minutes , GAUSc..
0
0 f-
0 cr:
UJ c.. a
UJ
N
:J
1 00
80
60
40
20
0
0 5 1 0
1 5
20 25 30
ELAPSED TIME (MINUTES)
35 40
KEY:
"' MOPAC
D
GAUSSIAN82
Figure 1 Transient CPU Utilization
SIAN 82 generated little ljO activity. Then , how ever, it generated a heavy I/0 load - up to
2 1 IjOs per second - to the user disk during the rest of the run. The ljO transfer size of
GAUSSIAN 8 2 is the largest of a l l the tests, around
2 5 kilobytes
( KB) per request. The I/0 data rate of a single GAUSSl.A.!'\1 82 test , col lected using the
Software Performance Monitor (SPM) program with 60-second intervals, shows as much as
530KB per second during this IjO i ntensive period .
Results and Observations
MOPAC Results. Figure 2 p lots the throughput of the MOPAC benchmark against the total num ber of streams i n the cl uster. The throughput increases linearly up to one job stream per pro cessor. Beyond this point the curves remain flat .
This flattening occurs because the benchmark is very CPU intensive, and one stream saturates a single processor with an average utilization of
9 9 . 6 percent. Therefore, adding more streams does not increase throughpu t.
The throughputs at which the curves flatten out are 1 .6 , 6 . 4 , and 1 2 . 8 jobs per hour re spectively for the VAX 8 7 0 0 , VAX 8974, and
VAX 8978 systems. In terms of relative perfor mance , the throughput of the VAX 8974 and
VAX 8978 systems were 4 . 0 times and 8 . 0 times respectively greater than the throughput of a sin gle VAX 8700 CPU, a l l showing linear growth with the number of streams.
GAUSSIAN 82 Results
Figure
3 shows the throughput for the GAUS
SIAN 8 2 benchmark plotted against the total number of concurrent streams on all the systems.
Ill
0
2 fc.. ir 1 6
::J
I
1 4
(jj 1 2
1 0
8
6
::J
0
4
2
5 1 0
1 5
TOTAL STREAMS
20
25
KEY:
•
...
0
VAX 8978
VAX 8974
VAX 8700
Figure
2
MOPA C Throughput
82 Digital Technical ]om-nal
No. 5
September
1987
::::> a..
I
::::>
0 a:
I
I-
�
::::>
0
I
(j)
10
0
2-
30
25
20
1 5
1 0
5
The curves show how throughput grows as the number of processors increases i n the cluster.
The VAX 8974 system achieved a maximum throughput of 1 2 . 1 jobs per hour with 1 6 con current streams. This throughput is 3 . 8 times that of the VAX 8700 CPU, which achieved
3 . 2 jobs per hour. The peak throughput of the
VAX 8978 system was 2 1 . 9 jobs per hour, or
7 . 0 times that of the VAX 8700 CPU. The relative figure for the VAX 8978 system is somewhat low because there was an imbalance in the use of the
1/0 subsystem.
Table 1 shows the 1/0 activities for each
HSC70 device during the five-stream run of
GAUSSIAN 82 on the VAX 8978 system . All the numbers are averaged for the entire run time .
One can dearly see in this table that some HSC70 devices were loaded more than others . Most disks were connected tO the two HSC70 controllers, labeled HSCO 1 1 and HSCO 1 4 , i nd icating that the other two were hot-standbys for the case of failovers. This loading variation happened because user disks were randomly assigned tO the job streams. The data rate of over 2 megabytes
(MB) per second on HSCO l l was only the aver aged number; the peak rate was close tO 4MB per second , thus limiting the IjO rate . The total data rate on the CI bus of the VAX 8978 system was over 4MB per second, 2 . 3MB of which was through one HSC70 device. This limited the per formance of five processors in the cluster.
TOTAL STREAMS
KEY:
•
•
D
VAX 8978
VAX 8974
VAX 8700
Figure
3 GAUSSIAN 82 Throughput
Note that within i nd ividual system configura tions, throughput increases as the number of streams increases. With the VAX 8974 system , for example, one stream per processor produced a throughput of 2 . 2 3 jobs per hour, increasing up to 3 .06 jobs per hou r - a 3 7 percent increase - with five streams.
Performance Summary
Table 2 shows the relative performance of each benchmark in terms of maximum throughput achieved with respect to a single VAX 8700 CPU.
The performance of the VAX 8974 and VAX 8978 systems ranged from 3 . 76 to 4 . 00 times, and
6 . 9 5 to 8 . 00 times that of the 8700, with geo metric means of 3 . 88 and 7 . 4 0 respectively.
Simulation ofthe GAUSSIAN 82
Workload on the 89 74!89 78
Based on the measured data, a model called
SIMsci was developed tO describe the perfor mance of the 8974/8978 under GAUSSIAN 8 2 , the mu ltistream , scientific computation work load.
As described earlier, GAUSSIAN 8 2 , a com putational package for quantum chemistry, is a collection of routines for different calculation needs. The key computational behavior patterns
Table
1
HSC70
H SC01 1
HSC01 2
HSC01 3
HSC01 4
1/0 Activities per HSC Device l/0 Rate
No. of (Requests/
Spindles Second)
Data Rate
(KB per
Second)
24
2
2
1 2
96.4
1 2.7
1 2.5
65.4
2 1 26.2
238.8
247.7
1 464.2
Table
2
Program
GEISHA
ISAJ ET
TAI R
TWI NG
MOPAC
RS/1
GAUSS IAN 82
Geometric Mean
SCIENCE Performance
Relative to the VAX 8700
VAX 8974
3.76
3.88
3.86
4 .00
4 .00
3 .82
3.84
3.88
VAX 8978
7.02
7.40
7.29
7.97
8.00
7.22
6.95
7.40
VAXcluster
Systems
Digital Technicaljournal
No. 5
September
1987
83
System Level Performance of
VAX
8974 and 8978 Systems of this workload modeled by SIMsci are
•
An executing stream places significantly dif ferent loads on t he CPU and the disk at differ ent times of execution (see Figure
1 for the transient CPU utilization pattern) .
•
An executing stream has a lot of
1/0 and CPU overlap (i . e . , computation continues while
1/0 is in progress) .
As shown in Figure 4 , SIMsci consists of batch jobs (as concurrent streams) , processors, and 1/0 devices. An executing batch job accesses both
CPUs and 1/0 devices. The execution of a job is modeled as several interconnected stages. Each stage represents an executing interval during which the job has similar utilizations of the CPUs and the
I/0 devices. These stages are introduced to capture the transient behavior of GAUSSIAN 82 shown in Figure
1 .
Note that the number and types of stages depend on the input data to GAUS
SIAN 8 2 , which triggers d ifferent routines to exe cute accordingly.
The CPUs and
I/0 devices are the principal resources consumed by a typical batch job. SIM sci models a CPU as a single-server queue ( i . e . , it can serve one batch job each time) . When more than one batch job competes for the same CPU, the jobs are served in a round-robin, time-sl iced fashion. The CPU serves a job exclusively either for a fixed duration (e.g. , 200 milliseconds) or until the job gives up the CPU (e .g., issues an 1/0 request) , which then switches to another waiting job. The I/0 device is simply modeled as a time delay since the GAUSSIAN 82 experiments are designed to avoid 1/0 resource contention . The presence of simultaneous CPU computations and
1/0 operations (over 30 percent of the time, as observed from direct measurement) , was mod eled. For a certain percentage of times, a job con tinues its computations within the CPU while its
IDLE BUSY
Figure 4 Model Structures ofS/Msci
I/0 request is being processed. For the rest of the times, a job is on hold while its 1/0 request is in progress.
S!Msci uses the following model parameters to describe the interactions of job, CPU, and 1/0 devices:
•
TotalStage, the total number of distinguishable stages of a batch job
•
Nio (l) , the total number of I/0 requests at stage I
•
TcpuUser(l ) , the total CPU time used by
GAUSSIAN 82 at stage I
•
TcpuSys(I) , the total CPU time used by the
VMS software at stage I
•
Tcpuldle (l) , the total CPU idle time due to page and swap waits at stage I
•
•
TioWait(I) , the total time that the job waits for its 1/0 to complete at stage I
RTio(I) , the average response time of disk 1/0 at stage I
The values of these parameters were derived from the measurement data . Several assumptions were made about the relationships between these parameter values and the VAXcluster con figurations and job loads per node . First, it was assumed that each job's Nio, TcpuUser, and
TioWait should have the same values for both the
VAX 8974 and VAX 8978 configurations and for different job loads (i.e . , number of streams per node) . These assumptions were made because each GAUSSIAN 82 workload woul d always exe cute the same codes with the same data in any of the environments.
Second , it was assumed that TcpuSys increases as the number of nodes and the number of streams i ncrease, thus adding communication load within the cluster and scheduling load within each node . The third assumption was that Tcpuldle increases as the number of nodes increases, since more page or swap requests would be placed on the pagejswap d isk, which is shared by all nodes in the cluster. It was also assumed, however, that Tcpuldle decreases as the number of streams per node increases. The more streams per node, the higher the probabil ity that at least one job without page faults exists and can utilize the CPU while other jobs are doing paging or swapping. These assumptions were consistent with the measurement results .
84
Digital Technical journal
No.
5
September
1987
SIMsci was validated against the measured data of three key metrics, job elapsed time , CPU utiliza tion, and disk 1/0 rate, with less than 5 percent difference.
The performance data collected were through put per hour and CPU utilization . Figure 5 shows that the measured and modeled results overlap for both the VAX 8700 and VAX 8974 systems, thus indicating the accuracy of the mode l . The
8978 curves , however , differ from each other.
The previous section discussed the fact that the measured throughput of GAUSSIAN 82 was some what low due to the i mbalanced r;o subsystem .
Therefore, the model results here give us a best case throughput when there is no I/0 boule neck . Although SIMsci produces reasonably accu rate results with little effort, i t does have i ts limitations. One major one is that SIMsci cannot predict the saturation of the IjO subsystem .
SIMwic assumes that IjOs are always free of bottlenecks; thus it cannot predict the perfor mance of the VAX 8974/8978 systems under heavy workloads (e.g.,
1 0 or more streams per node) .
Transaction Processing Environment
Workload Description
The warehouse and inventory control (WIC) workload is a transaction processing program based on the on-line support required to manage the movement of i tems into and out of a ware house . Although WIC is a warehouse app.lica-
�
:::J
0
I
;n
[!J
0
2
1-
:::J
Q_
I
C)
:::J
0 a:
I
1-
30
25
20
1 5
1 0
5
1 0 20
30
40
NUMBER OF STREAMS
KEY:
.A. v
•
•
0
0
8978 MEASUREMENT
8974 MEASUR EMENT
8700 MEASUREMENT
8978 SIMU LATION
8974 SIMULATION
8700 SIMU LATION
50
Figure 5 GAUSSIAN 82 Throughput
Model versus Measured
60 tion , it is a representative transaction processing application.
A WIC workload is divided into five functional parts, each associated with one task type. The five task types and the percent of total tasks rep resented by each type are given as follows:
•
•
•
•
•
Receiving - Performs the functions needed to log the receipt of parts from the loading dock i nto the warehouse ( 1 7 percent)
Inventory - Queries and u pdates the files conta ining inventory i nformation ( 1 0 per cent)
Warehouse - Performs the functions needed to pick parts based on se leered orders ( 1 0 per cent)
Order entry - Places orders to be fi lled by the warehouse ( 46 percent)
Purchase order - Composes purchase orders
(with outside vendors) for parts to be stocked in the warehouse ( 1 7 percent)
Each task is performed a specified proportion of the execution time. The task selection percent ages reflect the assu mption that the average flow of i tems into the warehouse equals the flow out of the warehouse during peak-hour operations.
Each task consists of a nu mber of transactions.
A transaction is defined as one or more user input steps followed by computation , database I/0, and output to the terminal user. Each task has an aver age of 7 . 8 transactions in the WIC app.lication .
Since a transaction implies the initiation of work by the system , throughput is measured in terms of transactions per second .
Al l menus and forms are implemented by re quests to the VAX Transaction Data Management
System. Inqu i ry and update operations take p lace on seven different application fi les i n the VAX
Record Management Services ( RMS) software.
Performance Metrics for WIC Workload
•
System throughput is defined as the total num ber of transaction processed systemwide i n constant time (one second) , o r transactions per second (TPS) . This number i ncludes all types of transactions . Figure 6 i ll ustrates the user and system actions needed for one trans action .
•
User prod uctivity is the average number of transactions each user completes i n a unit of time, expressed in transactions per user per hour.
VAXcluster
Systems
Digital Tech11ical]ournal
No. 5
September
1987
85
System Level Performance of VAX 8974 and 8978 Systems
USER
THINKING
TIME
USER
TYPING
TIME
USER
PR ESSES
CARRIAGE
RETURN
SYSTEM
BEGINS
RESPONSE
SYSTEM
COMPLETES
RESPONSE
- sERVICE TIME-
TRANSACTION
Figure 6 Transaction
•
Mean service ti me is defined as the average time required to complete a transaction . This time does not i nclude the input typing time or think time, but does include the time taken for screen output. A specific receiving transac tion, called REC3, was chosen for the evalua tion of this metric . REC3 i nvolves updating three records and writing one record several times, which represents a moderately complex unit of work.
Test Methodology
The transaction processing environment was cre ated by using remote terminal emulators (RTEs) , which emulated all activities of terminal users.
The RTEs also kept track of each transaction and the time of i ts occurrence and maintained the transaction mix throughout the experiment. Sev eral systems of the VAX 8600 cl ass were used as
RTEs to load the systems under test, called SUTs.
To establish a base level of performance, the initial set of experi ments was carried out with one VAX 8700 CPU as the SUT. The VAX 8974 and VAX 8978 systems were then tested by vary ing the number of users, and hence the number of transactions.
The RTEs logged users i nto the SUTs in the cluster at four-second i ntervals (users were evenly distributed between the SUTs i n the clus ter for all the configurations tested) . After log ging in, each user started his application , also at four-second i ntervals. After the SUTs reached a steady state, data was collected for 2 0 m i nutes on both the SUTs and the RTEs.
Hardware and Software Configuration
The hardware environment for each VAXcluster configuration included the same 1/0 subsystem .
The hardware components of the configurations consisted of the fol lowing elements:
•
A VAX 8974 system with four VAX 8700 CPUs, each with 3 2 MB of memory, two HSC70 con trol lers, one SA4 8 2 storage array for the sys tem ; and the pagingjswapping software, and three SA4 82 arrays for the database.
•
A VAX 8978 system with eight VAX 8700
CPUs; the other hardware was the same as the
VAX 8974 system 's above.
The software environment consisted of the
VAXjVMS version 4 . 5 operating system , VAX- 1 1
ACMS version 2 . 0, VAX- 1 1 TDMS version 1 .4 ,
VAX- 1 1 CDD version 3 . 1 , VAX- 1 1 COBOL ver sion 3 . 1 , and SPM version 3 0 .
In addition t o the general tuning of the SYSGEN parameters, several appl ication-specific parame ters were adjusted for the best performance .
These include the number of appl ication server processes, and the size of the RMS global buffer used to buffer some portion of each RMS fi le. In a distributed system l i ke a cluster, increasing the buffer size can resul t in additional 1/0 requests caused by more frequent buffer invalidations.
The database consisted of 1 4 RMS indexed sequential files spread over 1 2 disk spindles to balance the 1/0 rates.
Performance Results and Observatio ns
System Throughput
Figure 7 displays the system throughput (the nu mber of exchanges processed) at different user loads on the different configurations. These curves give a global i ndication of the overal l rela tive performance of the VAX 8974 and VAX 8978 systems.
UJ a._
(/) z
0 f=
<(
(/) z
<( cr
1-
0 z
0 u
UJ
(/)
60
50
40
30
20
1 0
200 400 600 800 1 000 1 200 1 400
N U M B E R
OF
USERS
K E Y :
0 VAX
.6. VAX
1:> VAX
8978
8974
8700
Figure 7 WIC Throughput
86 Digital Technical journal
No. 5
September
I 987
35
30 w
�
25
20
�
1 5 en
0
1 0
5
0
The VAX 8700 CPU peaked at 1 0 . 5 trans actions per second (TPS) while servicing
280 users. The VAX 8974 configuration achieved its maximum throughput rate of around 3 4 . 5 TPS while servicing 960 users. Thus the maximum throughput of the VAX 8974 system is about
3 . 3 times that of a single VAX 8700 CPU . The performance gain is not linear in this case because the degree of data-sharing is quite h igh in the WIC application, causing the locking over head typical in a cluster environment.
The limiting resource for the VAX 8974 system and the VAX 8700 CPU was processor power. The
8700 and each processor in the 8974 were fully utilized at around 960 users for the 8974 and
280 users for the 8700. The corresponding I/0 rates for the peak user levels were 2 20 and 60 respectively for the 8974 and the 8700.
The VAX 8978 system achieved a maxi mum throughput of 4 7 . 5 TPS while servic ing
I
, 200 users, which is only 4 . 5 times the
VAX 8700 throughput. Even taking into account the cluster overhead, this resul t is a very low rela tive performance gain. Clearly, this result indi cates that with the current implementation of the application the VAX 8978 performance was lim ited by some resource.
After more investigation , we found that the disks were this limiting resource . We observed a peak of 3 2 0 disk I/Os per second at 1 , 200 users on the VAX 8978 system . Let us assume that the
480 960
NUMBER OF USeRS
1 200
KEY:
-
DISK 1
DISK 2
c:::::::J
0
IS K 3 c::::::J
0
IS K 4
Figure 8 Disk f/0 Rates for WIC
(Top Four Disks) ljOs were uniformly distributed between the
1 2 spindles (which they were not) . In this case, dividing the peak of 3 2 0 ljOs between the spin dles yields 27 ljOs per spindle. However, the actual maximum observed on any one spindle was actually around 35 IjOs per second. Figure 8 plotS the four highest 1/0 rates.
I nvestigating further, we found that these disks also had large queue lengths associated with them (up tO 4 requests at 1 , 200 users) . Clearly, the IjO rates above coupled with the large queue lengths established that disk IjOs were the limit ing resource for the VAX 8978 configuration . In the section Simulation of the WIC Workload , where the modeling of VAXcluster systems is dis cussed , more data on the VAX 8978 performance will be presented without this l imiting factor.
Figure 9 gives a view of system performance in terms of throughput and processor utilization.
Note that the more processors there are in the sys tem, the more processor power it takes to do the same amount of work. For example, to obtain a throughput level of 30 TPS, the VAX 8974 system required 300 percent of the processor power and the VAX 8978 system required around 3 4 0 per cent. This extra power is needed by the cluster overhead, which involves locking activities and message transfers between the processors.
User Productivity
Figure
I
0 provides another view of throughput in terms of user productivity, defined as the throughput per user (the throughput in Figure 7 divided by the number of users) .
60 w en a: w o._
50
40
0 t=
20
(.)
..: en z
..: a: f-
30
1 0
0
0
/
200 400
CPU UTILIZATION
600
KEY:
0
£
.t;.
VAX 8978
VAX 8974
VAX 8700
Figure
9 Throughput versus CPU Utilization
VAXcluster
Systems
Digital Tecbnica/Journai
No. 5
September
1 987
87
System Level Performance of VAX 8974 and 89 78 Systems
200 400 600 800 1 000 1 200 1 400
NUMBER OF USERS
KEY:
0 VAX 8978
J1.
VAX 8974
!!.
VAX 8700
Figure 1 0 User Productivity
2.50
2.00
(j)
0 z
0 u w
(j)
1 .50
1 .00
0 50
0
0
�
200 400 600 800
1 000
NUMBER OF USERS
1 200 1 400
KEY:
D
J1.
VAX 8978
VAX 8974
!!. VAX 8700
Figure 1 1 WIC Service Time
This figure shows that the maximum through put per user for this workload is around 1 50 TPS for any configuration . This graph also indicates the number of users that can be supported by each system while mainraini ng a certain level of user productivity. For example, at 1 4 0 TPS , the
8700, 8 9 7 4 , and 8978 support 2 5 0 , 8 5 0 , a nd
1 , 2 00 users respectively. More users can be sup ported at lower user productivity levels.
Figure 1 0 also indicates the level of users at which one m ight consider switching to a larger system to maintain a certai n l evel of user pro ductivity . For example , to maintain a user pro ductivity level of approximately 1 5 0 TPS, one must switch to a VAX 8974 system at around
2 4 0 users, and to a VAX 8978 system at around
7 2 0 users.
Mean Service Time
The VAX 8700 and VAX 8974 service ti mes remained under one second for a l l user l evels tested. The VAX 8978 service-time curve a lso fol lowed this trend u p to the 960-user level . How ever, after that level, the service time degraded quickly due to the large number of IjOs and queue lengths at the disks as the 1 200-user level was approached . These patterns are shown i n
Figure 1 1 .
ENQ Rate
So far, only user visible performance and some system behavior has been discussed . Now some of the cl uster aspects of the systems are examined, mainly the locking activities.
As mentioned at the beginn i ng of this paper, the WIC workload assumes ful l data-sharing (i . e . , a l l the database fi les are shared by all users) . This sharing i nvolves locking and un locking fil es and records every time they are accessed . The locking and unlocking operations are performed by sys tem services called ENQ and DEQ . An ENQ request is serviced by the d istri buted lock man ager, which exa m ines outstand ing locks to the resource and a llows access if there is no confl ict .
The SPM software records the the number of
ENQs on a particular processor. The total ENQ rates at different user levels for differem configu rations were extracted from SPM data and graphed in Figure 1 2 . This cu rve cl osely resem bles the throughput curve, implying a strong cor relation between locking activities and through put. Around 26 ENQ operations were required on the average to perform each exchange .
Total Remote ENQ Rate
A remote ENQ occurs when the resource of inter est is mastered by a process that runs on an other processor in the c luster. Remote locks are more costly than local locks because additional interprocessor commun ication over the CI bus is required between the requesti ng and mastering nodes.
Figure 1 3 plots the remote ENQ rates aga inst the tOtal ENQ rates for different configurations.
88
Digital Technical journal
No. 5
September
1 <)87
200
KEY
:
0
•
D.
VAX 8978
VAX 8974
VAX 8700
400 600 800 1 000 1 200 1 400
NUMBER OF USERS
Figure
12
Total
ENQ
Rate
TOTAL ENO/SECOND
KEY:
0
•
D.
VAX 8978
VAX 8974
VAX 8700
Figure 13 Remote versus Total
ENQ
Rates
The increasing slopes of the different curves ind icate that the remote EN Q rate also i ncreases with the nu mber of processors in the system as well as with the total nu mber of users. Generally, i n an N-processor homogeneous distributed sys tem i n which all resources are equally accessed by all processors and all accesses require locking operations, the remote locki ng operations wi ll equal (N - 1 ) /N times the total locking activity.
This result occurs because each processor has an equal opportunity to master a particu lar resou rce. This relationship held in the case of t he remote versus the total new ENQ rates observed in the VAX 8974 and VAX 8978 systems, in which the ratios were 75 percent and 8 7 . 5 percent respectively. Figure I 3 shows, however, that on the average only 60 percent and 80 percent of the ENQs were remote for the 8974 and the 8978 respectively. These resu lts occu rred because t he ploned ENQ rate includes the convened ENQ rate as well as the new ENQ rate; most convened
ENQs were found to be loca l . lnterprocessor Com munication
The communications between the processors are achieved by the Systems Communication Archi teCture by way of transmitting and receiving sequenced messages. Figure 1 4 shows the num ber of sequenced messages transferred between the processors every second . Most of these mes sages are generated by t he distributed lock man ager for clusterwide locking purposes.
0 z
0 frl 4 .00
� 3.50
(f)
0 z
3.00
;?,
2.50
6
2.00
I
1-
I
1 .50
(f)
�
1 .00
0.50 l{l
::2
0 �����----�--�----�--�--�
0
200 400 600 800 1 000 1 200 1 400
NUMBER OF USERS
KEY:
D
•
D.
VAX 8978
VAX 8974
VAX 8700
Figure 14 Message Rate between Processors
Cl Traffic
The traffic on the CI consists of three packet types: datagrams, sequenced messages, and block transfer messages. In this appl ication , datagrams were used only for error loggi ng and t herefore did not exist . Sequenced messages are used for communications between the processors and the
HSC70 controllers . Most of these short packets are either packets between the d istri buted lock managers to perform clusterwide lock ing (dis cussed earl ier) or packets between a processor and an HSC70 controller to request and response to I/0 operations. Each 1/0 request to the disks
·Digital Technical journal
No. 5
September I
')87
89
VAXcluster
Systems
System Level Performance of VAX
89 74 and
89 78
Systems or tapes controlled by an HSC70 device requires a pair of messages to be exchanged between the processor and the controller. Block transfer mes sages are data packets for 1/0 operations. The transfer rates of each message type are recorded by the SPM software . Figure 1 5 pl ots the CI traf fic against the n umber of users. The CI traffic, expressed i n KB per second , is calculated from the data collected by the SPM software.
This figure shows that, in general , the CI bus is rather underutilized , peak ing around I , 2 65KB per second at 1 , 200 users for the VAX 8978 sys tem . This utilization is less than I 5 percent of the raw bandwidth of a single CI wire , or 7 . 5 percent of the bandwidth on each CI path. It should be noted, however, that this data includes neither the extra bytes of the lower level protocol over head nor the additional traffic incurred by retransmissions. Thus the actual CI util ization will be a little higher than these figures.
WIC Database Partitioning
Extended Study
The results presented i n the previous section indicate that the application as currently imple mented presented a problem with the disk IjO.
More IjOs were being generated to several files, resulting i n too many d is k IjOs to several spin dles. To reduce the nu mber of ljOs, we parti tioned both the application and the database, anticipating that the number of ljOs to each spindle wou ld be reduced. This section su mma rizes the results from this study.
The main difference between this study and the previous one is the number of disk spindles used. This study used 24 spindles (6 SA4 82 s) , whereas the previous study used only 1 2
(3 SA4 82 s) . The throughputs achieved with this new configuration are plotted in Figure 1 6 .
It is clear that with this configuration the
VAX 8978 system performed much better with
2 4 spindles than with 1 2 . The system achieved a peak throughput of 66 transactions per second with I ,60 0 users, which was 6 . 3 times the throughput of the VAX 8700 CPU . This resu lt ill ustrates the importance of having a system ba l anced in regards to its processing power and 1/0 capacity.
Simulation ofthe WIC Workload
Based on the measurement data , a model called
SIMwic was developed to describe the perfor mance of VAX 8974/8978 systems under WIC, the multiuser, on-line transaction processing workload. WIC characterizes the on- l ine transac tion processing of items ( i .e . , parts) that flow into and out of a warehouse and supports multi ple concurrent access to the WIC database. The model structure of S!Mwic is shown in Figure I 7 .
The fol lowing components of WIC were mod eled in SIMwic:
•
Users (who generate transactions)
•
Lock messages
•
CPUs
•
Shared IjO passages (CI bus, HSC70 con trol ler, channel )
•
Disks
0
5
�
1 .60
1 .40
1 .20
0.60
0
0
(.) w
(/)
60
70
�
1 00 a: w ll.
60
50
(/) z
40
�
0.60
> al
="
30
0.40
0.20
0
(.)
<{
(/) z
20
1 0
�--�---L--�----L----L--�--�
0
200 400 600 BOO 1 000 1 200 1 400 1-
0
NUMBER OF USERS
KEY:
D VAX 6976
4
VAX 6974
1'>
VAX 6700
500
1 000
NUMBER OF USERS
1 500
KEY:
4
0
24 DISKS
1 2 DISKS
Figure
15
CI Traffic
Figw-e 1 6 WIC Throughputfor
1 2 versus 24 Disks
2000
90
Digital Technica/journa/
No.
5 September 1987
-
USER LOGIN
TASK DELAY
DATA ENTRY DELAY
NEXT TASK
INITIAL TRANSACTION
TRANSACTION DELAY NEXT TRANSACTION
LOCK MESSAGES
Figure 1 7 Model Structure of S!Mwic
A user generates one task at a time tO access the
WIC database, eac h task consisting of severa l transactions. Each transaction uses the CPU for a certain amount of time and sends several I/0 requests through the shared r;o passage to access the WIC database disks. Each IjO request wil l first send lock messages to ensure that the data is accessible and then i ni tiate the IjO opera tions .
The fol lowing parameters are used by SIMwic to describe the interactions of the users, lock messages , CPUs , shared I/0 passages, and d isks:
•
Jntertask Delay, the delay after the completion of a task prior tO the initiation of another task by the same user
• lntertransaction Delay , the delay after the completion of a transaction but prior to the initiation of the next transaction by the same task
•
Task Mix, the percentages of each task type of the WIC workload
•
•
Total Transaction , the total number of transac tions for each task type
Total Disk IjO, the total number of d isk 1/0 for each transaction
•
•
ProbDisk , the probabil i ty of selecting disk for r;o
CPU Delay, the CPU ti me tO process a transac tion on each visit
•
•
Lock Delay, the CPU time to process lock mes sages due tO an I/0 request
CI Delay, HSC delay , Disk Delay, and Channel
Delay , delays due to data transfer and disk seeks
The values of these parameters were obtained from several sources, including workload specifi cations, direct measurements, other performance studies, and hardware specifications.
SlMwic was val idated on measurements of CPU uti li zation , throughput, and disk I/0 rates. The differences between simu lated and direct-mea-
Digital Technicaljournal
No 5
September
1 987
9 1
VAXcluster
Systems
System Level Performance of VAX 89 74 and 89 78 Systems sured results were within five percent, as shown in Figure 1 8 . The performance data collected were task life-cycle, throughput rate, CPU uti l i zation , and disk 1/0 rate.
A5 discussed earlier, the performance of the
VAX 8978 system under the WIC workload can be sign ificantly improved by spreadi ng the data base over 24 d isks i nstead of 1 2 . S IMwic modeled such a database expansion and confirmed the per formance improvements on the throughput, as plotted in Figure 1 9 .
Summary
The performances of VAX 8978 and VAX 8974 systems were studied i n two environments: a scientific, compute-intensive batch environment a z a
() w
(/) a:
(l.
(/) z
0 f=
<(
(/) z
<( a:
1-
60
50
40
30
20
1 0
0
0
500 1 000
NUMBER OF USERS
1 500 2000
KEY:
.11.
MEASUREM ENT
0
SIMULATION
Figure
I
8 WIC Throughput - Model versus Measured
�
80
8
70 w
(/)
60
50 cg_
40
§
30
�
(/)
20
::i:
1 0
�
0 '--
0
-'---
500 1 000
NUMBER OF USERS
1 500
---'
2000
KEY:
.II.
24 DISKS
0 1 2 DISKS
Figure
I
9 Model Results: Throughput with
I
2 versus 24 Disks using the SC IENCE workload, and an on- li ne transaction processi ng environment using the
WIC workload. These two environments were chosen to capture the range of the relative per formances VAXcluster systems can achieve compared with the performance of a single sys tem . Using both measurement and modeling approaches, i t was shown that the 8974 has from 3 . 3 to 4 . 0 t imes the performance of a single
VAX 8700 CPU, depending on the degree of fi le sharing, when there is no substantial bottleneck in the 1/0 subsystems. A 8978 was shown to have between 6 . 0 and 8 . 0 times the performance of the VAX 8700 CPU, again depending upon the application's characteristics, especially the amount of remote locking activi ty.
Acknowledgment
The authors wish to thank Joe Marconis for his support with the WIC workload, and Bill
Youngs for providing a suit of scientific pro grams. Also thanks to Jory Tsai for the d iscussion on the VAX 8 6 5 0 cluster model , to Hossein
Hosseini for the WIC experiments, and to Ray
Kopacko, who developed the SCIENCE workload and performed the experiments using i t .
References
1 . W. Snaman and D . Thiel, "The VAXjVMS
Distributed Lock Manager,"
Digital Techni cal journal
(September 1 987, this issue) :
2 9-44 .
2 . D . Duffy, "The System Communication
Architecture, "
Digital Technical journal
(September 1 987, this issue) : 2 2- 2 8 .
3 . N . Kronenberg, H . Levy, and W. Strecker,
"VAXclusters: A Closely-Coupled D is tributed System," A CM Transactions on
Computer Systems, vol . 4, no. 2 (May
1 986) : 1 30- 1 46 .
4 . P . Fleming and ). Wal lace, " How Not To Lie
With Statistics: The Correct Way To Summa rize Benchmark Results," CA CM, vol . 29, no. 3 (March 1 986) : 2 1 8- 2 2 1 .
5 . F. Colon Osorio, N . Quaynor, D. Park, X .
Cao, "Axiomatic Approach t o Summarizing
Benchmark Resul ts," Annual Review/
Reports, System Performance Group
( 1 986) .
92
Digital Technicaljournal
No. 5
Septem ber
1 987
Xi-Ren Cao
Fernando
Nii N. Quaynor
C.
Colon Osorio
CI
Bus Arbitration Performance in a VAXcluster System
CI bus performance is difficult to evaluate with a conventional queuing network approach. Therefore, a new mode� a generalized semi-Markov process, is used to model the process on the CI bus under its arbitration algorithm. This new model is implemented in a PASCAL program that is run for different configurations of VAXcluster systems. The simulation results demonstrate the properties of the arbitration algorithm. The results also suggest that a centralized control scheme could improve the CJ utilization, and that some load-balance schemes can reduce the average response time. The method may be useful for designing other products.
This paper relates the study of performance of the CI bus in a VAXc luster environment. The cluster nodes (computers and storage control lers) are connected through a Star Coupler by a dual-path CI bus. An arbitration algorithm deter mines which node wil l be allowed to send pack ets over that CI bus. The performance of the CI bus may d irectly affect the cluster's performance, and studying the performance of the CI bus algo rithm shoul d yield some useful insights to en hance the designs of future computer-intercon nect products.
Our approach is first to buil d a model that cap tures the main feature of the algorithm , 1 and then to consider other aspects as parameters of the model . The most important parameters are the length of the packets and the length of the quiet slot.
Because arbitration is complicated, a conven tional queuing network model would be inade quate for model i ng the CI process. For example, the CI bus cou ld not be modeled as a server since packet transmission cannot start immediately after a request arrives, even if the CI bus were idle. Thus we propose another model based on the general ized semi-Markov process (GSMP) .
Moreover, this model may be useful for studying other processes in VAXcluster systems.
CI Bus Arbitration Algorithm
Here, we briefly review aspects of the C I arbi tration related to the performance study. Refer ence 1 contains details of the CI bus arbitration .
A Simple Description of a CI Bus
Let us assume a VAXcluster system in which there are N nodes attached to a CI bus. Each node can send both information and acknowledge packets through the bus to any other node. Upon receiv ing an information packet, a node first checks the cyclica l redundancy check (CRC) information in that packet. If the CRC succeeds, the receiving node will immediately send back to the transmit ting node an acknowledge packet with either an acknowledgment (ACK) if the node accepts and stores the packet correctly, or a non-acknowledg ment (NAK) if not. If the CRC fai ls, the node will send no response .
A time period, called the quiet slot, is reserved to guarantee the transmission of the acknowledge packet. The quiet slot (QS) is defined as the period of time needed to accommodate the time delay through a node's front-end logic, plus the round-trip cable and coupler delays for the longest path in a CI c luster installation . Only the node that generates the acknowledge packet for the i nformation packet just received can grasp the CI bus during the quiet slot fol lowing the transmission of any information packet. Thus, as an approxi mation, the transmission time of the information packet may be extended to incl ude the transmission time of the acknowledge packet.
After sending an information packet, the trans m i tting node wai ts for the length of an acknowl edge time-out period. If that node receives an
ACK during that period , the transm it is com pleted . Upon receiving a NAK or no response within the time-out period , however, the trans-
Digital Teclmical]ournal
No. 5
Septem ber
1987
93
Cl Bus Arbitration Performance in a VAXcluster System mitting node must retransmit the packet. The acknowledge time-out period is greater than the sum of one quiet slot , plus the CI bus turnaround time, plus the time to verify and accept the acknowledge packet at the transmitting node .
In addition, in any such "shared" multinode bus structure, the arbitration for use of the bus so as to avoid col lisions is a critical element of the design . The CI bus architecture implements the distributed arbitration scheme d iscussed below.
CJ Bus A rbitration
Two identical Cl paths are used in a VAXcluster system, and all nodes are connected to both of them. Each node can randomly pick one path before transmi tting an information packet. Once having chosen a path, the node will use it until an acknowledge packet from the destination node has been received. However, each node cannot transmit and receive simultaneously using two different paths. Figure 1 illustrates the structure of a VAXcl uster system in which VAX CPUs and
HSC devices are connected to one CI pat h .
Arbitration must b e performed by all nodes prior to the transmission of any information packet. The acknowledge packet, following re ceipt of an information packet, does not require arbitration. This method is called a slotted-carrier sense multiple access (CSMA) protocol, a lso referred to as dual-count round robin. The fol lowing parameters are used in current VAXclus ter systems :
•
The clock u n i t (TCLK) is s e t a t 1 1 4 . 2 8 nano seconds ( ns) .
•
The va lue of the quiet slot can range from 7 to
64 TCLKs, or
800 to
7 , 3 1 4 ns, depending on the cable length of the cluster. The QS for the for the discussion of this paper simulation is
1 , 1 4
3 ns.
HSC i
+ 1
DISKS
. . .
HSC n
DISKS
Figure 1 A Typical VAX cluster System
•
The maxi mum number of nodes in the cluster,
N, is
1 6 for the current algorithmic implemen tation .
•
The IO numbers of the nodes are
1 , . . . , N
1
, one for each node .
1 = 0 ,
The arbi tration a lgorithm operates as follows:
1 . Upon starting a transmit operation , node I chooses randomly one of the C I paths and sets the value of i ts arbitration counter, C, to
N + l + l .
2 . I n each TCLK period , the node determ ines whether or not the C I bus is busy. If i t is busy, the arbitration counter will remain unchanged .
3 . Once the node senses that the CI bus is not busy, it will start counting quiet slots. That is, the arbitration counter is set to C - 1 , and the node then wai ts for one QS period .
If C >
0 at the end of one QS period , the node wil l inquire if the CI bus is busy. If i t isn ' t busy, C i s set t o C 1 , and the node waits during one additional QS period . lf the
CI bus is busy, the arbitration counter is set to another va lue that depends on the node
I D .
•
I f the C l bus is occupied by a node whose
I D is greater than I, or if this is the node's first attempt to grasp the CI bus, then C is set to
N + I +
1 ( i . e . , the initial value of C for this node) .
•
I f the CI bus is occupied by a node whose
1 0 is less than I and this is not the first attempt of node
C is set tO
I to grasp the CI bus, then
I +
1 . After the arbitra tion coun ter is reset, control returns tO step 2 above.
If C
=
0 at the end of the QS period , the node inquires again if the CI bus is busy. If so, the arbitration counter is set to another value that depends on the node 10, as explained just above . I f the CI bus is not busy, the node inquires if a packet is being received from t he other path.
•
I f the node is receiving from the other path, C is reset to N, and control goes to step 2 above .
•
If the node is not receiving, it starts the transmission i mmediately.
94
Digital Tecbtzical]om-nal
No. 5
September
1 <)87
Figure 2 shows a possible case of
CI arbitra tion. This figure depicts a short histOry of the arbitration times for three nodes, labeled
2 ,
6, and 8 . During the period [O , t i ] , the
C I bus is transmitting a packet from some other node while both node
2 and node 6 have requested tO transmit. The arbitration counters of these two nodes are set respectively to 1 9 ( 1 6 + 2 + 1 ) and
2 3 ( 1 6 + 6 + 1 ) . At time t
1 , the
CI bus becomes idle, and nodes 2 and 6 both start counting quiet slots. At time t2 ( t2 - t1 = 1 9QS) , the arbitration counter of node 2 becomes zero; hence node 2 wins the bus. At this instant, the arbitration coun ter of node
6 is 4 . After detecting that the bus has been captured by a node whose I D is less than its own , node 6 sets its arbitration counter number to 7 (6+ 1 ) . (Assume that this is not the first attempt of node 6 . )
The transmission of the packet from node
2 ends at
14 •
Node 6 starts counting again at
14 with an arbitration counter of 7 and wins the bus at h (t6 - t4 =
7
QS) .
Figure
2 also shows that requests arrive at the ports of nodes
2 and 8 at t3 and t5 respectively. At t6, the arbitration counter of node 2 becomes 1 9 ( 1 6 + 2 + 1 ) since the bus was won by a node whose
10 is bigger than that of node 2 . The arbitration counter of node 8 is set to
2 5 ( 1 6 + 8 + 1 ) since this is node 8 's first attempt to occupy the
C1 bus.
For simplicity, we wi ll study the properties of only one
CI path in this report. The principle for studying two
C I paths shou ld be the same .
Some Prelim inary A nalysis
Although a complete analysis of the
CI bus pro cess is difficult, some preliminary analyses may help us to understand the properties of this process and perhaps validate the simulation results.
When two packets attempt to pass through the same path of the
CI bus simu ltaneously, both packets will be destroyed. Therefore , packets can be passed successfu lly only if, before sending a packet, each node determines whether the
CI bus is busy. Even with this check, two nodes can sti l l send their packets simultaneously i f each node detects at the same instant that the
CI bus is idle.
The situation is even worse because of the propa gation time of a packet from the transmitting node to the detecting node.
The introduction of the QS concept into the arbitration algorithm almost eliminates the possi bility of packet collisions when the
CI bus is satu rated . In this case nearly every transmit request will find the
CI bus busy and must wait until the end of the transmission of the current packet. At the end of a transmission period from a node, denoted as
I0, a l l other nodes having an outstand ing transmit request wil l start counting quiet slots simu l taneously. The arbitration counters of nodes whose transmit requests are made during the transmission period have the form N + I + 1 .
The arbitration counters of those nodes whose transmit requests were made in previous trans m ission periods have the form
N + I + 1 if
I
>
I0, or the form
I if
I
<
I0 .
Thus at a given time, each node has a unique arbitration count. The node whose arbitration counter reaches zero first will grasp the cr bus.
After each transmission period, there is a short i ntenral ( 1 6 quiet slots) in which no transmis sions occur on the
CI bus. However, every requesting node is sti ll counting the quiet slots during this period . For example, suppose that in one QS, the smal lest arbitration counter is
25 a:
I-
20 a:
IiD
<(
0 z
Q
1 5 t;:
1 0
5
VAXcluster
Systems
Figure 2 A rbitration among Three Nodes
Digital Tecbnicaljournal
No. 5
September
1987
9 5
Cl Bus A rbitration Performance in a VAXcluster System
N + I +
1 , and that in the same QS, node I ini tiates a transmit operation. In this case the arbi tration counters of node arbitration counter is
I and the node whose
N + I +
1 in that QS are always the same . Therefore, these two nodes could start to transmit at the same time, and a col lision cou ld occur, even though its probabil i ty is very sma l l .
The CI bus could b e considered a s a server.
From the arbitration scheme discussed above, however, customers do not start services i mmedi ately after arriving at t h e server, even i f i t is id le.
One may argue that the arbitration time can be modeled by a separate server. I n this case, how ever, the customer i n that server does not have a fixed service time (the arbitration counter needs to be reset frequently) . Therefore, the Cl bus cannot be modeled as a standard queuing system .
Fortunately, many stochastic processes exist that can be used to model real-world processes. One stochastic process, called the general ized semi
Markov process, has a characteristic very similar to the process on the CI bus under the above arbi tration rules.
In the next section, we give an description of this process.
Generalized Semi-Markov Processes
The general ized semi-Markov process, or GSMP, is one of the most promising stochastic processes in operations research for modeling complex phenomena. GSMP was introduced by Matthes,2 and i nvestigated further by other researchers, among them Schassberger,3 and Whitt.4
A GSMP can be described as follows. Let S and
R be subsets of positive i ntegers. We regard the elements s of subset S as possible states of the
GSMP. Some events may occur at each state. R denotes the indices of all possible events that may occur during the evolution of a GSMP. All events that can occur i n state s are denoted as set E ( s ) , which i s a subset of R .
The system will stay i n a state s until a n event i
E
E(s ) triggers a transition of the system to another state s'. Let p (s
'
, s , i) be the probability that the new state is s', given that event i triggers a transition from state s .
An event can trigger a transition only at the end of i ts lifetime . Associ ated with each event i is a clock whose reading is denoted as
C; .
The clock runs at a speed r ( s ,i ) , which depends o n both the event i and the state s . If at time 0 the clock is set to C; , then at time t the reading of the clock will be c'; = C; - r (s , i) X t . The l i fetime of an event ends when the associated clock reading reaches zero .
We assume r ( s , i) > O for some i E E(s) . When r(s ,i) = O for i E E(s ) , event i is regarded as i nactive in state s .
The events associated with state the set s' are i n
E ( s ') . The c lock readings after the transition are determi ned as follows. New clock readings are i ndependently generated for each j
E
N(s', s ,i ) - E (s') - (E(s) - i) . The new clock reading for event j E
N(s',s , i) has a cumu lative probabil ity distribution , or c.p.d . , of
F(x;s', j,s , i ) . For events i n both E(s) and E (s') . except for event i , the old clock readings are kept after the transition, i .e . , for j E
O (s',s , i) = E(s') n (E (s ) - i ) , ci
=
C/ (s , c )
For events in E(s) but not i n E(s') , the c locks arc set equal to zero ( i . e . , i f j E
(E(s ) - i) - E (s') , then c1 = co after the transition .)
For the purpose of modeling the CI process, the above scheme of determ ining the clock read ings has to be modified slightly. We associate each event i with a set of events
H ( i) .
Only for events i n j E
O (s',s ,i) = E (s') n !E (s ) - H(i)l are old clock readi ngs kept (i.e.,
For events in c1 = c/ (s , c ) ) .
N(s', s , i ) = E(s') - IE(s ) - H ( i ) l . new clock read ings have t o be assigned according to the c.p.d. F(x;s', j , s , i ) . We call the process with this clock-reading assignment scheme a modified GSMP. A block diagram is shown i n
Figure 3 .
The next transition occurs according to the same rules. These transitions describe the evolu tion of the system.
The Stochastic Process on the CI Bus
To describe the process on the Cl bus, we use a continuous time doma in as opposed to a dis crete domai n ( i . e . , we consider the clock unit
1 1 4 . 2 8 nanoseconds to be i nfinitesimally sma l l compared with other event t i mes, such a s trans mission times.) Furthermore, to make the prob lem tractable , we make the fol lowing stochastic assumptions:
•
•
•
The transmission ti mes requ i red by every node are independent of each other.
The times between two successive transmis sion requests are independent.
The destinations of the transmitted packets arc independent of the transm itting node and the transmission time.
96 Digital Technical journal
No. 5 September I ')87
Under the above assumptions, the CI system can be characterized by the following items:
•
The number of nodes, N
•
The cumulative distribution functions of the transmission time of each node, denoted as
- 1
•
The cumulative distribution functions of the time between two successive transmis sion re-quests of each node , denoted as
G1 (x) , i = O ,
1 , . . . ,N
-
1
•
The probabi lity that a packet from node go to node j, denoted as
Pt,J i will
The state x of the CI bus consists of the following elements:
•
An index j, indicating the node that is trans mitting a packet (We use j = N to indicate that the Cl bus is idle . )
•
The number of transmission requests made by nodes i = O, 1 ,
. . .
,N
- 1 , denoted as n0, n 1 , . . . ,nN- 1
STATE AT
1(1)
WITH EVENTS e , , e2. AND e3
!
DETERMINE LIFE TIMES e3 FROM DISTRIBUTIONS, CHOOSE RATES r,,
1,,
12, l3 FOR e, TO r2. f3 ACCORDING TO SOME RULES
!
EVENT e, DIES AT t,
= I( I)
+
I , / r ,
ETC.
SUPPOSE 12 = MINIMUM {!, .12.13}
!
NEXT TRANSITION HAPPENS AT
1(1
+ 1 )
=
1(1)
+
12
!
DETERMINE THE STATE AND EVENTS
(SAY e2. e3. AND e,) AFTER TRANSITION
!
ASSIGN NEW LIFE TIME AND RATES FOR e2, e3, AND e,
Figure 3 Block Diagram of Modified GSMP
•
The residual transmiSSIOn times of nodes i = O ,
1 ,
. .
.
,N- 1 , denoted as bo, b, , . . . ,bN- 1
(Except for node j , these values are the same as the transmission times.)
•
The residual times between two transmission requests of nodes i = 0,
I
,
. . . ,N1
, denoted as t0, t, , . . . ,tN- 1
•
The arbitration counters for the first request of nodes i = 0, 1 , . . . ,N - 1
, denoted as a0 ,a , , . . . ,aN- I (Note that a1 = 0 . )
The process o n one C I path can b e described as a modified GSMP. Let
L be the length of a QS period . The arbitration counters can be translated into continuous nu mbers d1 = a1 X L .
These con tinuous numbers can be viewed as clock read ings. When j = N ( i . e . , no packet is being trans mi tted on the CI bus), these clocks run at a rate r = 1 until one of the readings reaches zero .
When j 'f'= N, then these clocks run at a rate r = O ; this means that when a server is transmitting packets, all arbitration counts do not change . The clock readings may jump to some other values at some transition times.
Now we can describe the process on one CI path. Let s = {j , n0 , n1 , . . . , nN- d -
Using the termi nology of GSMP, we call s the state of the process.
Associated with each state s , there are at most 3N events in
E (s)
( i . e . , the end of a transmission from each node, the grasp of the CI bus by each node , and a new request arrival at each node) .
The clock readings corresponding to these events are b1 , t1 , and d1, i = O , l , . . . ,N - 1 . The clock rates are always one for all t1 , one for b1 , zero for b1 if i 'f'= j , and one for all d1 if j 'f'=
0 and zero for all d1 if j = N. For convenience, we also use b1 , t1 , and d1 to denote the corresponding events. Thus
E(s) = { b1 , i : n1 > 0; t1 , aU i; d1 , i : n1 > 0}.
The only remaining work for specifying the
GSMP on the Cl path is to determine the clock rates r (s , c ) , transition rules p (s , s ' , i ) , and clock reading distributions
F(x ,s',j ,s , i ) .
These can be done by examining carefully the arbitration scheme . The details can be found in reference 7 .
We have modeLled the C I process as a modified
GSMP. This concept helps us to simplify the underlying mechanism of the process. This mechanism is no more complicated than state transition and clock readings. A simulation algorithm based on this model is given i n the next section .
VAXcluster
Systems
97
Digital Tecbnicaljournal
No. 5
September
1987
CI Bus Arbitration Performance in a VAXcluster System
Simulation Algorithm
Although the GSMP concept looks sophisticated, its simulation is not difficult. In fact, the simu la tion of a GSMP consists mainly of two steps:
1 . Use the c lock readings and c lock rates to determine the next transition time and the event that triggers this transition .
2 . Determine the new state and the new clock readings after each transition.
Thus the GSMP model simplifies the concept of the mechanism of CI arbitration to these two steps.
The specific rules and distributions for deter mining the process on one CI path were de scribed in detail in the previous sections. The simu lation algorithm is given as follows:
1 . Initialize the system .
•
Choose a n initial state s = {j ; n0 , n 1 , . . . , nN- d. n1 is the number of transmission requests of node i. j is the node transmit ting, and j = N means that the bus is idle.
•
Assign initial clock readings for events.
For all nodes, the next transmission request happens at a time with distribu tion
G1
(x) .
The transmission time of a request on each node has a distribution
F1
(x) .
Set the arbitration counts accord ing to the arbitration rule.
•
Set the value of the simulation clock, v , to 0.
2 . Determine the clock rates for events accord ing to the state s .
The rates for the next trans mission request are always 1 . The rates for the transmission completion are 1 for nodej, and 0 for all other nodes. The rates for arbi tration counters are 1 for all nodes if j = N
(CI bus idle) , 0 ifj =F N (CI bus busy) .
3 . Using the clock rates, find the event whose clock reading reaches zero the earliest. This event triggers the transition. Set the simula tion clock to the time when this reading reaches zero.
4 . Using the transition probabilities, determine the next state of the process.
5 . Assign new clock readings and rates for the new state. (This can be done as described in steps 1 and 2 above.)
6 . If the terminating condition is not met, go to step 3 . If the condition is met , stop the simu lation .
There are some points that should be noted about this algorithm.
First, the model for two CI paths can be easily obtained by combining two models for one CI path and making the following modification. At the end of the arbitration of each node, the model checks to determine if the node is receiving from the other path. If not, the node starts transmis sion; otherwise, the model sets the arbitration count C of that node to N and starts the counting again.
The second point is, the ACK or NAK transmis sion times are i ncluded i n the i nformation packet transmission times (i .e . , the distribution
F1 (
x
) describes the total transmission times of both an information packet and its ACK or NAK) .
As mentioned earlier, we wrote a PASCAL pro gram to implement this algorithm . The next two sections discuss the problems of choosing parameters for this model and the performance results obtained .
Choosing Parameters
As mentioned earlier, the maximum number of nodes in a CI-based VAXcluster system is 1 6 ; therefore, N is set to 1 6 in the simulation. QS is set to I , I 4 3 ns.
The remaining problem is choosing the mean transmission times and the mean i nterrequest times, all of which depend on the node types and specific applications. In this simu lation , these values are taken from the results of two previous experiments performed at DigitaJ .5·6 The first of those observes the CI packet traffic in a system running ASYNCQIO; the second measures the
IjO performance of a system running lOX .
(ASYNCQIO and lOX are both workload pro grams used for simulations.) The fol lowing is the mean interrequest and the mean transmission times of these two experiments; we use them as parameters in our simulation .
For ASYNCQIO, we have:
•
The mean interrequest time of a VAX 8600
CPU with a CI780 bus is rv. 1
= 8 , 3 0 0 micro seconds (J.1S) .
•
The mean transmission time is
Sv,I
= 6 . 4 J.lS.
•
The mean interrequest time of the HSC device is rH. 1
= I ,4 0 0
J.lS.
98
Digital TecbnicalJournal
No. 5
September
1987
•
The mean transmission time of a packet from an HSC device is sH. I
= 60 . 5 J.LS.
For lOX, we have :
•
The mean interrequest time of a VAX 8600
C PU with a CI780 bus is rv.2 = 2 2 ,900 microseconds (J.Ls) .
•
The mean interrequest time of the HSC device is rH.z
= 3, 800 J.LS.
Since we assume that lOX reads the same num ber of blocks per request as ASYNCQIO, the mean transmission times
Sv,2 and sH,2 are the same as sv. 1 and sH. I ·
These values are obtained by assuming that the
VAX CPU runs only one stream of ASYNCQIO or lOX on one disk. If the CPU runs m streams simultaneously, it is reasonable to take rv,;fm and rH,;jm , for i = l , 2 , as the mean interrequest times.
Fina l ly, both ASYNCQIO and lOX are 1/0 intensive workloads. Therefore, the simu lations described in the next section, using the data derived from these two workloads, represent the performance of
1/0 intensive programs. The cal culations here just yield reasonable values for parameters.
Simulation Results
The mean values obtained in the previous section were used in the simulations. In each simulation run , half the nodes were VAX systems, the other half were HSC devices. Also, half the VAX systems ran ASYNCQIO, the other half ran lOX . To study the CI performance, we ran four sets of simula tions.
The first set had 1 6 nodes, or eight VAX sys tems and eight HSC devices. The average trans mission time for the VAX systems was 6 . 4 J.LS, and for the HSC devices 6 0 . 5 J.LS. The interrequest times were chosen tO model the systems in which each VAX system runs from one to three streams of the
1/0 intensive workloads. Specifically, the mean interrequest times for a system running two streams are half those for a system running only one stream , and so forth .
The Cl utilization rates of this first set of simu lations are shown in Figure 4 , the other results in
Table 1 . The CI bus transmits packets during busy time, arbitration occurs during arbitration time , and the bus is idle during idle time . Idle time does not include any arbitration time . The busy, idle, and arbitration rates are the ratios of
80
70 w
�
60
50
�
40
(,) ffi
30
20
1 0
0 r--
,..-n
I
,..--
..--
-
2
NUMBER OF STREAMS
.....-
-
3
KEY: c:::J
BUSY RATE
CJ
IDLE RATE
ARBITRATION RATE
-
Figure 4 CI Performance for First Simulation
Table 1 First Set of Results
Simulation 1 . 1 1 .2
1 .3
No. of nodes: n
1 6 1 6 1 6
No. of streams
Busy time - seconds
Idle time
Arbitration time
Busy rate - %
Idle rate
Arbitration rate
Arbitration/busy ratio
1 3 s, to s8 -
5g to s,s
!!Seconds 6.40
60.50
6.40
60.50
6.40
60.50 r, to r4
- JLSeconds rs to rs r9 to r, 2 r, 3 to r,6
Total time - seconds
8,300.00 4 , 1 50.00
22,900.00 1 1 ,450.00
1 ,400.00 700.00
2,800.00
7,600.00
470.00
3,800.00 1 ,900.00 1 ,270.00
43.79 21 .90 1 4.69
1 0.56
27.27
5.96
24
62
1 4
0.58
2
1 0.56
5.95
5.38
48
27
25
0.52
1 0.57
0.42
3.70
72
3
25
0.35
Response time -
!!seconds
RE,
RE2
RE3
RE4
RE5
REs
RE7
RE6
RE9
RE1 0
RE1 1
RE, 2
RE,3
RE14
RE,5
RE, 6
1 20
1 27
1 35
1 39
1 43
1 47
1 50
46
48
48
50
51
55
57
61
1 1 5
2 1 2
238
269
257
263
270
278
95
97
1 01
1 81
80
86
86
92
95
213
235
246
261
256
256
265
269
1 ,0 1 5
1 , 1 79
1 , 1 7 1
1 ,281
539
545
555
552
99
Digital Tecbnica/Jom·nal
No. 5 September
1987
VAXcluster
Systems
CI Bus A rbitration Performance in a VAXcluster System
80
70
60
UJ
;::
50 a:i
40
() ffi
30 a..
20
1 0
0
16
1 2 8
NUMBER OF NODES
4
KEY:
c:::J
BUSY RATE
c:::J
I DLE RATE
ARBITRATION RATE
Figure
5
Cl Performance for Second
Simulation
Table 2
Second Set of Results
Simulation 2.1 2.2
No. of nodes: n
1 6 1 2
6.40
60.50
6.40
60.50 s1 to sn12 - 11seconds
Sn/2+1 to
Sn r1 to rn/4
- 11seconds rn/4+1 to rn/2 rn/2+1 to r3nf4 r3nf4+ 1 to rn
Total time - seconds
Busy time - seconds
Idle time
Arbitration time
Busy rate - %
Idle rate
Arbitration rate
Arbitration/busy ratio
Response time -
11seconds
RE9
REI O
RE1 1
RE1 2
RE13
RE14
RE1s
RE16
RE,
RE2
RE3
RE4
REs
RE6
RE7
REa
239
1 ,002
1 ,086
1 ,037
1 ,091
471
475
482
481
2 1 5
237
235
245
232
235
243
94
99
1 03
1 03
1 07
1 1 0
21 8
249
283
248
255
258
-
-
-
-
2.3
8
6.40
60.50
59
60
64
64
1 32
1 53
1 58
1 62
-
-
-
-
-
-
-
-
2.4
4
6.40
60.50
2,800.00 2,800.00
7,600.00 7,600.00
2,800.00 2,800.00
7,600.00 7,600.00
470.00 470.00
1 ,270.00 1 ,270.00
470.00 470.00
1 ,270.00 1 ,270.00
1 4.69
1 0.57
0.56
3.56
72
4
24
0.33
1 9.56
1 0.55
4.59
4.41
54
23
23
0.43
29.45
1 0.55
1 4.28
4.61
36
49
1 6
0.44
58.95
1 0.56
44. 1 5
4.25
1 8
75
7
0.39
41
41
1 0 1
1 08
-
-
-
-
-
-
-
-
-
-
-
the busy, idle, and arbitration times tO the total time respectively.
From these results, we can see that the arbitra tion time takes about 23 to 24 percent of the total time if the CI bus is busy for more than 50 per cent of the total time. The ratio of arbitration time to busy time decreases as the busy rate increases. We can also see that the response time is somewhat sensitive to the i nterrequest time .
HSC controllers have a longer response time than
VAX CPUs since the interarrival times of the con trollers are shorter. The results also reveal that while the arbitration is almost fair for all nodes, some very small degree of unfairness still exists.
For example, nodes 1 3 to 1 6 have the same mean interrequest and transmission times; however, the response times increase slightly as the
1 0 number of the node increases. These properties will be explained later. Of course, such a small degree of unfairness will not affect the perfor mance of the CI cluster.
The second set of simulations compared the performances of clusters with 4 , 8, 1 2 , and
1 6 nodes. The node I D numbers are 0 to 3 for the
4-node experiment, 0 to 7 for the 7-node experi ment, and so forth . Each VAX CPU runs three streams of lOX or ASYNCQIO.
The results are shown in Figure 5 and Table 2 .
These results confirm the properties observed in the first set of simulations. As far as the CI traf fic is concerned , reducing the number of nodes is equ ivalent tO decreasing the traffic intensity on the bus.
The third set of simulations examined the effect on performance of the lengths of packets transmitted on the CI bus. The average transmis sion times of a packet are assumed to be either
60 . 5 , 60 . 5/2 . 6 0 . 5/ 3 , or 6 0 . 5/4 11-s, depending on the number of streams. The results are shown in Figure 6 and Table 3 .
As we expected , the ratio of arbitration time to busy time increases as the length of a packet decreases. If the average packet length is one fourth of a block, the system will spend more time arbitrating than transmitting.
The fourth set of simulations kept the interre quest times of eight nodes constant at 1 , 000
J.LS, but varied the times of the other eight nodes from
300 to 1 , 000 11-s. The parameters are l isted in
Table 4 , and the results reported in Figure 7 .
Figure 7 shows that i f the mean interrequest times of nodes 1 to 4 and 9 to 1 2 are between
700 and 1 ,000 11-s, the average response times of
1 00
Digital Technicaljournal
No. 5
September
1987
all nodes wi ll be very similar. If the interrequest times of these nodes decreases further, their response times increase rapidly. In this case a load balance scheme would be needed to achieve better performance .
CI Arbitration Properties
We can make the following observations from the simulation resul ts:
•
The response time increases rapidly i f the Cl bus is nearly saturated. This behavior is similar to that of a single-server queue.
•
The arbitration algorithm is almost fai r for a l l nodes. There i s only a very sma l l degree of unfai rness. The response times of nodes with lower ID numbers are a l ittle bit smal ler than those of nodes with higher IDs.
To explain this unfairness , let us consider two nodes, node
I and node 1 0. Two cases in which node node
1 gets h igher priority than
1 0 are given as follows:
I . Assume that the CI bus is idle, and that node
I
0 requi res a transmission at t1 while node 1 requires a transmission at t1 + 9QS .
In this case, node 1 wil l win the bus despi te the fact that node request before node 1 .
I 0 submitted i ts
•
2 . Assume that the C I bus is busy, and that during this busy period both nodes 1 and
1 0 require transmissions. As soon as the CI bus becomes idle, both nodes wiJl start counting quiet slots. In this case, node 1 will always win the bus whether or not it was the first to make the request.
Under the current arbitration a lgorithm, the response ti mes are sensitive to the inter request times, especially when the CI bus is highly utilized . For example, in Simu la tion 1 . 3, the response times for two nodes with mean i nterrequest times of 470 and
7,600 J.LS are approxi mately 1 ,050 and 2 4 0 J.LS respectively.
This resu lt will occur because, under satura tion, the arbitration is approximately a round robin algorithm . If there are three requests i n node
I and six requests in node
2 , t h e CI bus must serve the three requests in node 1 and the first three requests in node
2 before it can serve the last three requests in node
2 .
This algorithm gives higher priority to requests i n node
1 than t o those in node
2 .
80
70
60
LLJ
�
50 dJ
40
(.) ffi a..
30
20
1 0
0
60.5 30.25 20. 1 7 1 5 . 1 2
MEAN TRANSMISSION TIMES, - !'SECONDS
KEY:
CJ
BUSY RATE
c:::J
IDLE RATE
ARBITRATION RATE
Figure 6
CI
Performance for Third
Simulation
Table 3 Third Set of Results
Simulation
No. of nodes: n s, t o s8 -
�!Seconds
� to s,6 r, t o r4
- 14seconds r5 to r6 r9 to r,2 r13 to r,s
Total time - seconds
Busy time - seconds
Idle time
Arbitration time
Busy rate - %
Idle rate
Arbitration rate
Arbitration/busy ratio
3.1
1 6
3.2
1 6
3.3
1 6
3.4
16
6.40
60.50
6.40
30.25
6.40
20. 1 7
6.40
1 5 . 1 2
2,800.00 2,800.00
2,800.00 2,800.00
7,600.00 7,600.00 7,600.00 7,600.00
470.00 470.00 470.00 470.00
1 ,270.00 1 ,270.00 1 ,270.00 1 ,270.00
1 4.69 1 4.69 1 4.69 1 4.69
1 0.57
0.56
3.56
72
4
24
0.33
5.38
4 . 1 6
5.15
37
28
35
0.95
3.65
5.59
5.45
25
38
37
1 .48
2.78
6.34
5.57
1 9
43
38
2.00
Response time -
�!Seconds
RE,
RE2
RE3
RE4
RE5
RE6
RE7
RE6
RE9
RE10
RE1 1
RE, 2
R£,3
RE14
RE15
RE,s
215
23 7
235
245
232
235
243
239
1 ,002
1 ,086
1 ,037
1 ,091
471
475
482
481
56
60
63
65
65
69
69
72
1 1 1
1 2 1
1 31
1 4 1
1 37
1 38
143
1 46
93
95
98
1 01
54
55
76
81
87
93
41
44
46
48
49
52
44
46
48
49
36
39
41
42
63
6
7
72
76
77
79
82
84
VAXcluster
Systems
Digital Technical ]om-nal
No.
5
September
1 98 7
1 0 1
CI Bus Arbitration Performan ce in a VAXcluster System
1 .00
0
�
0.80
0.60
I
0.40
::2 i=
0.20 w
(f)
6
Bi w cr:
0
L---�-----L----J_----�----L---�
0 200 400 600
800 1 000 1 200
INTERREQUEST TIME - �tSECONDS
KEY: o
•
NODE
5
NODE 1
Figure 7 CI Performance for Fourth
Simulation
Table 4
Fourth Set of Results
Simulation 4 . 1 4.2 4.3 4.4 4.5
No. of nodes: n
1 6 1 6 1 6 1 6
1 6 s, to sa -
�tseconds
5g to s16
6.40
60.50
6.40
60.50
6.40
60.50
6.40
60.50
6.40
60.50 r, to r4 -
!'seconds r5 to ra r9 to r,2 r13 to r16
1 ,000.00
1 ,000.00
1 ,000.00
1 ,000.00
900.00 800.00
1 ,000.00 1 ,000.00
900.00 800.00
1 ,000.00 1 ,000.00
700.00 600.00
1 ,000.00 1 ,000.00
700.00 600.00
1 ,000.00 1 ,000.00
Simulation 4.6 4.7 4.8 4.9 4 . 1 0
No. of nodes: n
1 6 1 6 1 6 1 6 1 6 s,
�tseconds s9 to sa to s,6
6.40
60.50
6.40
60.50
6.40
60.50
6.40
60.50
6.40
60.50 r, to
'• -
!'seconds r5 to ra r9 to r12 r,3 to r16
500.00
1 ,000.00
500.00
1 ,000.00
450.00 400.00
1 ,000.00 1 ,000.00
450.00 400.00
1 ,000.00 1 ,000.00
350.00
300.00
1 ,000.00
1 ,000.00
350.00
300.00
1 ,000.00
1 ,000.00
•
Figure
7 shows the effect of the relative inter request time on the response times. The response times of nodes
5 to
9 i ncrease rapid ly when their interrequest time is between
50 and 60 percent of the time for nodes 1 to 4 .
•
The results of the first and second sets of simu lations show that the higher the CI busy rate, the smal ler the total arbitration time . For example, in Simulation
1 . 1 , the total arbitra-
• tion time for 200 ,000 requests is 5 .86 sec onds, while that figure in Simu lation 1 . 2 is
5. 1
I seconds.
If the CI bus rate is low, the average transmis sion request from node I will have to wait an arbitration time of (N + I + 1 ) QS . If the Cl busy rate is high , however, each request can always find some node whose ID is lower and which can occupy the CI bus earlier. In this case the average request spends only (I + 1 ) QS on arbitration .
In the third simulation, the arbitration time rate increases from
0. 24 for an average packet length of one block to
0 . 38 for a length of one fourth of a block. The absolute val ue of arbi tration time also increases. This result occurs because the arbitration time i s the same for packets with different lengths.
Conclusion
This paper describes the performance of the algorithm for CI bus arbitration as measured by a generalized semi-Markov process mode l . The simu lation results show the following:
•
The arbitration algorithm is almost fair to a l l nodes.
•
•
•
The ratio of arbitration to busy times depends on the average length of packets transmitted; the smaller the length, the bigger this ratio.
The ratio of arbitration to busy times also depends on the traffic i ntensity; the larger the intensiry, the smaller the ratio.
The response times of packets at a node are sensitive to its I/0 rate compared to other nodes; the higher the rate, the longer its response time .
•
Because of the arbitration time, the CI bus is not fu lly utili zed . I n experiment
1 . 3 , the effective bandwidth for one path of the CI bus is about 75 percent. This effective bandwidth a lso depends on the average length of packets.
The resul ts indicate where problems can be anticipated, especially when the CI bus is highly uti l i zed, and suggest some ways to improve Cl performance.
Acknowledgments
The authors are indebted to Jory Tsai for d is cussing the possibility of using PAWS to imple· mem the GSMP model , and to Hossei n Hosseini for assistance in preparing the report.
1 02
Digital Technical Jou1-nal
No. 5
September
198 7
References
I . V. Boaen et a l . , "Computer I nterconnect
Specification, " Digital Equipment Corpora tion Standard 1 6 1 -0 , 1 986.
2.
K. Matthes, "Zur Theorie der Bedienungs prozesse,"
Transactions of the Third
Prague Conference on Information The ory ( 1 96 2 ) .
3 .
R. Schassberger, "Insensitivity of Steady state Distributions of Generalized Semi
Markov Processes, Part 1 , "
A n nuals of Prob ability 5.
( 1 977) : 8 1 -99.
4. W. Whitt, "Continuity of Generalized Semi
Markov Processes,"
Mathematics of Opera tions Research , vol . 5 , no . 4 ( 1 980) : 4 9 4 -
5 0 1
5. B. Murray, "CI Traffic Observations: A Com parison of the CI780, CIBC I , and CIBCA,"
Digital Equ ipment Corporation Internal
Technical Memorandum (OctOber 1 986) .
6. X. Cao and H . Hossei n i , " I/O Properties of a
VAXcluster: Part I , " Digital Equipment Cor poration Internal Technical Memorandum
(OctOber 1 986) .
7 . X. Cao, N . Quaynor, and F. Colon Osorio
"CI Bus Arbitration Performance in a VAX
: cluster," D igital Equipment Corporation
Internal Technical Memorandum (March
1 987) .
VAXcluster
Systems
Digital Tecbnical]ounlal
No . .5
September
1 987
1 03
. . ..
ISBN 1-5 55 58-004-l
Printed in USA EY-8258E,,D.I'"Cojlytigh<
©
Scp<c"!ber 1.1)8.7 Digital Equipment Corporation
डाउनलोड
विज्ञापन