Digital AlphaServer 2100, 2000 Server Technical journal
Below you will find brief information for AlphaServer 2100, AlphaServer 2000. AlphaServer 2100 and AlphaServer 2000 are server systems designed to take advantage of the high-performance Alpha microprocessor architecture developed by Digital The AlphaServer 2100 and AlphaServer 2000 systems implement an I/O subsystem that is based on the Peripheral Component Interconnect (PCI) and Extended Industry Standard Architecture (EISA) buses. The systems provide high performance through their multiprocessing architecture and share common CPU, memory, and I/o architectures. The systems provide flexibility through the use of PC-style I/O subsystem and are designed to support multiple generations of Alpha processor technology
Advertisement
Advertisement
•
•
•
AlphaServer Multiprocessing Systems
DEC OSF/1 Symmetric Multiprocessing
Scientific Computing Optimizations for Alpha
Digital Technical Journal
Digital Equipment Corporation
Volume 6 Number 3
Summer 1994
Cover Design
The cover design captures two major concepts in this issue-symmetry and pamllelism. At the hardware level, the AlphaServer multiprocess ing systems provide symmetrical access to hard ware system resources. As processors are added to the multiprocessing system, the DEC OSF/1 operating system provides the parallelism that allows applications to take advantage of the added processor power. The KAP preprocessor also provides parallelism, specifically for pro grams running on symmetric multiprocessing systems. In each case, symmetry and parallel ism are among the keys to achieving designs that offer the highest levels of performance.
The cover was designed by joe Pozerycki, Jr., of Digital's Design Group.
Editorial
Jane C. Blake, Managing Editor
Helen L. Patterson, Editor
Kathleen M. Stetson, Editor
Circulation
Catherine M. Phillips, Administrator
Dorothea B. Cassady, Secretary
Production
Terri Autieri, Production Editor
AnneS. Katzeff, Typographer
Peter R. Woodbury, Illustrator
Advisory Board
Samuel H. Fuller, Chairman
Richard W. Beane
Donald Z. Harbert
Richard J. Hollingsworth
Alan G. Nemeth
Jean A. Proulx
Jeffrey H. Rudy
Stan Smits
Robert M. Supnik
Gayn B. Winters
The Digital Technical journal is a refereed journal published quarterly by Digital
Equipment Corporation, 30 Porter Road LJ02/Dl0, Littleton, Massachusetts 01460.
Subscriptions to the journal are $40.00 (non- U.S. $60) for four issues and $75.00
(non-U.S. $115) for eight issues and must be prepaid in U.S. funds. University and college professors and Ph.D. students in the electrical engineering and computer science fields receive complimentary subscriptions upon request. Orders, inquiries, and address changes should be sent to the Digital Technical journal at the published by address. Inquiries can also be sent electronically to [email protected]. Single copies and back issues are available for $16.00 each by calling DECdirect at 1 -800-DIGITAL
(1-800-344-4825). Recent back issues of the journal are also available on the Internet at http://www.digital.com/info/DTJ/home.html. Complete Digital internet listings can be obtained by sending an electronic mail message to [email protected].
Digital employees may order subscriptions through Readers Choice by entering VTX
PROFILE at the system prompt.
Comments on the content of any paper are welcomed and may be sent to the managing editor at the published-by or network address.
Copyright© 1994 Digital Equipment Corporation. Copying without fee is permitted provided that such copies are made for use in educational institutions by faculty members and are not distributed for commercial advantage. Abstracting with credit of Digital Equipment Corporation's authorship is permitted. All rights reserved.
The information in the journal is subject to change without notice and should not be construed as a commitment by Digital Equipment Corporation or by the companies herein represented. Digital Equipment Corporation assumes no responsibility for any errors that may appear in the journal.
ISSN 0898-90IX
Documentation Number E Y-S799E-TJ
The following are trademarks of Digital Equipment Corporation: Alpha, AlphaServer,
DEC, DEC Fortran, DEC OSF/1, DECpc, DECthreads, Digital, the DIGITAL logo, Micro VAX,
Open VMS, Storage Works, and ULTRIX.
CR AY-1 is a registered trademark ofCray Research, Inc.
Intel is a trademark oflntel Corporation.
KAP is a trademark of Kuck & Associates, Inc.
Microsoft and MS-DOS are registered trademarks and Windows NT is a trademark of
Microsoft Corporation.
MIPS is a trademark of MIPS Computer Systems, Inc.
Multimax is a trademark of Encore Computer Corporation.
Open Software Foundation is a trademark and OSF/ I is a registered trademark of Open
Software Foundation, Inc.
PAL is a registered trademark of Advanced Micro Devices, Inc.
SPECfp, SPECint, and SPECmark are registered trademarks of the Standard Performance
Evaluation Council.
SPICE is a trademark of the University of California at Berkeley.
TPC-A and TPC-C are trademarks of the Transaction Processing Performance Council.
UNIX is a registered trademark licensed exclusively by X/Open Company, Ltd.
Book production was clone by Quantic Communications, Inc.
Contents
6 Foreword
Steve Holmes
AlphaServer Multiprocessing Systems
8 Design of the AlphaServer Multiprocessor
Server Systems
Fidelma M. Hayes
20 The AlphaServer 2100 1/0 Subsystem
Andrew P. Russo
DEC OSF/1 Symmetric Multiprocessing
29
DECOSF/1 Version3.0Symmetric
Multiprocessing Implementation
Jeffrey M. Denham, Paula Long, and
James A. Woodward
Scientific Computing Optimizations for Alpha
44
DXML: A High-performance Scientific
Subroutine Library
Chandrika Kamath, Roy Ho, and Dwight P Manley
57
The KAP Parallelizer for DEC Fortran and
DEC C Programs
Robert H . Kuhn, Bruce Leasure, and Sanjiv M . Shah
2
I
Editor's Introduction
jane C. Blake
Managing Editor
Designs that capitalize on Digital's 64-bit AJpha
RJSC processors or that enhance the performance of scientific applications are the subjects of papers in this issue. Featured topics include the wel l received AlphaServer multiprocessing systems, the DEC OSF/1 symmetric multiprocessing operat ing system, a high-p erformance math library, and a preprocessor program developed by Kuck &
Associates, Inc.
To develop a price/performance leader for the server market, designers of the AJphaServer 2100 and 2000 mu ltiprocessing systems had to make decisions that were at once creative, pragmatic, and timely. Fidelma Hayes, an engineering manager for the Server Group, presents an overview of these high-performance servers that incorporate Alpha
RJSC technology and re-style 110 subsystems, and support three operating systems-Microsoft's
Windows NT, DEC OSF/1, and Open VMS. Because of the engineering team's persistent focus on perfor mance, cost, and time-to-market, all these goals for the AlphaServer systems were surpassed .
Introducing two PC buses i n the Alp haServer multiprocessing system was an importan t factor i n market success and a n interesting engineering chal lenge. Andy Russo discusses the benefits of a dual level 1!0 strucrure that contains both the wiclely used EISA bus and the newer high-performance
PCI bus that connects to a 128-bit multiprocessing system bus. He describes several innovative tech niques that promote efficiency in the hierarchi cal bus structure, the advantages offered by the selection of bus bridges (one custom ASIC ancl o ne standard chip set), ancl the 1/0 interrupt scheme that combines familiar technology with custom support logic.
The next paper presents the significant software work done to ensure high performance and reliabil ity as crus are added to the 2100 and 2000 multipro cessing systems. Jeff Denham , Paula Long, and Jim
Woodward first review the fou ndations of DEC
0Sf11 ve rsion 3.0, Digital's imple mentation of UNIX for the AlphaServer multiprocessing systems. They then examine issues that arise when moving an operating system from a un iprocessor to a shared memory SMP platform, in particular, the design team's efforts in lock-based synchronization and algorithm modifications aimed at paral lel ism within the operating system kernel.
The total impact of 64-bit RJSC systems and oper ating system support for shared memory SMP plat forms is demonstrated by meeting the demands of scient ific and technical app l ications. A tool for accelerating appl ication performance on all Alpha systems is the DXM.L Extended Math Library.
Chanclrika Kamath, Roy Ho, and Dwight Manley briefly discuss the role of mathematical l ibraries and then present an overview of DXMI. compo nents, which include both public domain BLAS and
LAPACK libraries and D igital proprietary software .
Using example routines, they explain optimization techniques that effectively exploit the memory hierarchy and provide substantial performance improvements.
Another tool for optimizing scientific application performance is KAP, a preprocessor to paral lelize
DEC Fortran and DEC C programs. As authors Bob
Kuhn, Bruce Leasu re, and Sanjiv Shah from Kuck &
Associates describe it, the KAP product is a super optimizer, performing optimizations at the sou rce code level that go beyond those performed by the compilers. Their paper reviews adaptations to KAP for SMP systems and the key design aspects, such as data dependence analysis and the selection of loops to parallelize from among many in a program.
The editors thank Andrei Shishov, M id- range
AlphaServers Program Manager, for his help in developing this issue of the jo urnal.
Biographies
I jeffrey M. Denham A principal software engineer in the UNIX Software
Group, jeffrey Denham is a contributor to the DEC OSF/1 version 3.0 symmetric multiprocessing effort. Prior to this, he helped add POSIX.Ib features to the DEC
OSF/1 operating system and worked on the VAXELN real-time kernel. Jeff came to
Digital in 1986 from Raytheon Corporation. He holds a l:l.A (1979) from Hiram
College, an MA. (1980) from Tufts University, both in Engl ish, and an M.S. (1985) in Technical Communication from Rensselaer Polytechnic Institute.
Fidelma M. Hayes
As an engineering manager in the Server Group, Fidelma
Hayes led the development of the AlphaServer
2100
and AlphaServer
2000
sys tems. Prior to this work, she led the design of the DECsystem
5100.
She has con tributed as a member of the development team for several projects, including the DECsystem
5800 CPU,
the PRJSM system design. and the Mi croVAX
3100.
Fidelma jo ined Digital in
1984
after receiving a bachelor's degree in electrical engineering from University College Cork, Ireland . She is currently working toward a master's degree in computer science at Boston University.
Roy Ho
As a principal software engineer in Digital's High Performance
Computing Group, Roy Ho developed the signal-processing routines used in
DXML. Prior to this work, he was a member of the High Performance Computing
Technology Group. There he designed the clock distribution system for the VAX fault-tolerant system and the delay estimation software package for the VAX
9000 system boards. Roy has B.S.
(1985)
and ;vrs
(1987)
degrees in electrical engi neering from Rensselaer Polytechnic Institute. He joined Digital in
1987
Chandrika Kamath Chandrika Kamath is a member of the Applied Computa tional Mathematics Group. She has designe<.l and implemente<.l the sparse linear solver packages that are included in DXML. She has also optimized customer benchmarks for Alpha systems. Chandrika holds a Bachelor of Technology in electrical engineering (1981) from the Ind ian Institute of Technology, an M.S. in computer science (1984) and a Ph . D. in computer science (1986), both from the University of Illinois at Urbana-Champaign. She has publ ished several papers on numerical algorithms for parallel computers.
3
Biographies
Robert H. Kuhn Robert Kuhn joi ned Kuck & Asso ciates as the Director of
P roducts in 1992. H is functions are to form ulate technical business strateg y and to ma nage product deliveries. From 1987 to 1992, he worked at Al liant Compu ter
Systems, w here he managed compiler develo pment and a ppl ica tion software fo r parallel processing. Bob rece ived his Ph.D. in com puter science from the University of Ill inois at Champa ign-Ur bana in 1980 He is the a uthor of several tech nical publ ica tions and has participated in organizing various techni cal conferences.
Bruce Leasure Bruce Leasure. one of three founders of Kuck & Associates in
1979, serves as V ice President of Technolog y and is the chief scientist for the company As a charter member and executive director of the Parallel Computing
Forum (PCF), a standards-setting consortium, he was a leader i n effor ts to stan dardize basic forms of pa ral lelism. The PCF subsequently became the AJ'<SI X3H5 comm it tee f()r Para l lel Program Constructs for H igh-leve l Languages, which he chaired. Bruce received B.S. and M.S. deg rees in computer science from the
U niversity of Ill inois at Champaign-U rbana.
Paula Long Since joining Digital i n 1986. Paul a Lo ng has contributed to vari ous operati ng system p rojects. Present ly a princiral software eng i neer with the
UNIX Softw are G roup. she leads the development of symmetric mult iprocessing
(SMP) capabil ities for the DEC OSF/1 operating sys tem. In previo us positions, she
Jed the DEC OSF/ I real-time and DECwindows on VAXELN projects. Paula received a BSC.S. from Westfield State Col lege i n 1983.
Dwight P. Mantey Dwight Manley is a consul ting software engineer i n the
Appl ied Computational M athematics Group. He jo ined the DXJ.'<IL Group in 1989 a nd continues to s upport and enha nce the DXM L ancl KAPF products. Since jo ining
Digi tal in 1979. be has worked on system measur ement and model ing projects and was re spo nsible for all perform ance model ing of the VAX 9000 CPU design.
He is l isted as a coinventor on 11 patents and as a coauthor of a paper on matrix com putation theo ry. Dwig ht has a BS in mathematics from the University of
M assachusetts and an M.S. i n operations research fro m Nort heastern University.
Andrew P. Russo Andy R usso is a principal hardware engineer in the Alpha
Server G roup. Since joining D igital in 1983, Andy has been a project leader for several i nternal org anizations, including the M id-range 1/0 Options G roup, the
Faul t To lerant Grour. and the Alp ha Server G roup . W h ile at D igital, Andy has contributed to the architecture and design of high-performance ASICs and mo dules to provide a va riety of end-prod uct requ irements. A ndy holds several patents and has authored two papers. He received a B.S. in computer engineer ing from Bosto n University.
4
Sanjiv M. Shah Sanjiv Shah received a B.S. in computer science and mathe matics (1986) and an M.S. in computer science and engineering (1988) from the
University of Michigan. In 1988, he joined Kuck & Associates' KAP development group as a research program mer. He has since been involved in researching and developing the KAP Fortran and C products and managing the KAP development grou p. Currently, Sanjiv leads the research and development for parallel KAP performance.
James A. Woodward Principal software engineer James Woodward is a mem ber of the UNIX Software Group. He is responsible for DEC OSF/1 symmetric mul tiprocessing (SMP) processor scheduling and base kernel support. In previ ous work, Jim led the ULTRIX SMP project and the VAX 8200, VAX 8800, and VAX
6000 ULTRIX operating system ports. He also wrote microcode for the VAX 8200 systems as a member of the Semiconductor Engineering Group. Jim joined
Digital in 1981 after receiving a B.S.E E from the University of Michigan.
I
6
I
Foreword
Steve Holmes
Engineering Group
Manager, Sen'er
Platform Del'elopment, and Directm; Office
Server Product Line
The engineering developments described in this issue represent the second of many planned gener ations of products that will be designed to fulfill
Digital's Alpha vision. That vision is (a) to make
Alpha systems open, and (b) to deliver a rich set of
Alpha system products that lead the market both in performance and price/performance. It is heart ening to see the vision being realized. It is yet more heartening to see it unfolding simultaneously with appreciable improvements in Digital's business practices. These combined events have already resulted in substantial market acceptance of
Digital's AlphaServer products.
The particular set of papers in this issue is for tuitous in that it demonstrates the large number of individuals :l!ld range of engineering skills required to bring about an industry phenomenon such as Alpha. Included are papers focused on the
AJphaServer multiprocessing systems, on the sym metric multiprocessing implementation of the DEC
OSF/ I operating s�'stem. on the optimization of mathematical subroutine libraries for the Alpha architecture. ami on the KAl' preprocessor. If one can imagine these technical efforts multiplied manyfolcl. the scope of the Alpha undertaking will emerge.
The first generation of products based on the
Alpha architecture was introduced in 1992. The
AlphaServer 2100 system and DEC OSF/1 SMP operat ing system. introduced in mid-1994, together repre sent the beginning of the second-generation Alpha server products. The overarching development goal was to give our present and future customers a compelling reason to buy. The resultant direction was to provide very low cost multiprocessing sys tem capability with inclustry-stanclard open l/0 buses. in this case PC! and EISA. To capitalize on these attributes and to ensure that a complete solu tion was delivered, the engineering teams main tained a customer-focused perspective. It is this perspective that has enabled the AlphaServer 2100 to achieve rapid market acceptance.
Truly, though. the most significant achievement for the present round of Alpha server products is this: a whole new standard of price/performance for the industry has been reached. Computing that in the past could have been performed only with very expensive high-end machines or extensive dis tributed networks is now performed by affordable
A! phaSe rver systems.
This price/performance breakthrough augments
Digital's strong capabilities.
•
•
A tru ly open environment that supports UNrx and
Windows NT operating systems on Alpha systems
The ongoing strength of the world's best fu l l featured com mercial operating system, the
Open VMS system
•
A world-cl;�ss ;�ncl wo rldwi de service ;�nd del iv ery org;�niz<�tion
•
An extensive <�nd growing network of channels
•
Over;�ll, D igit<�l's renewed <�nd me;�ningful com mitment to be responsive to the demands and needs of the markets
This is a very exciting ami productive time in
Digital's h istory
If this were the end of the story, there would be much of which to be proud . In fact, there is more to come across the range of AlphaGeneration prod ucts, incl uding workstations, PCs, clustering, oper ating systems, and networking. In the server area speci fica lly, the recently announced AlphaServer
2000 increases the price/performance lead of the
2100 system. Processor and cache upgrades have increased the absolute performance of the family.
Just around the corner are similar advances for other members of Digital's server p roducts. A l ittle further away are s igni ficant enhancements in our clustering capabi l ities and in our server m anage ment tools.
Al l these developments are of d i rect and measur able benefit to our customers. All are guided by what the markets are telling us they want. The trend and pace of these enhancements will al low
Digital to continue to deliver on the promise of the
Alpha vision.
Performance measurements, for example,
SPECmark data and transaction-per-second tests, and competit ive comparisons support the state ments above. However. the case is made most con vincingly by the early acceptance and rapid ramp u p of AlphaServer 2100 system purchases by our customers. In the h ighly competitive server arena, success is being demonstrated dai ly.
I wou ld l ike to take this opportunity to offer a very enthusiastic thank-you to all whose work is represented in the accompanying technical papers, most especially to the AlphaServer 2100 develop ment team w hose work I have had the privi lege to observe since the team's formati on. The hard work and dedication of everyone is recognized, appreci ated, and needed for the fut ure.
This foreword will conclude in favor of the sub stantive papers that detai l the technical contribu tions made by the au thors and their col leagues. It is my expectation that readers of this issue of the
Digital Technical.fournal will gain usefu l technical insights. It is my hope that they will also see, as I do, that the future of Digital compu ting is bright.
7
Fidelma M. Hayes
I
Design of the AlphaServer
Multiprocessor Server Systems
Digitar� AlpbaSen>er multiprocessor systems are bigbjJeJfonnance sen>ers that
C01Jibine multiprocessing tecbnolog)' with PC.stj>le 110 subsystems. The �ystem a rcbitecture allou•s j(mr processing nodes, four memo1y nodes (up to a maximum of 2 GB), and tu•o I/O nodes. All nodes communicate tbroug!J a system bus. Tbe
S)'Stem bus tms designed to support multiple generations of Alpha processor tech nology Tbe arcbitect!lre can be implemented in different zmys, depending on tbe size of the .)ystem packaging
The AlphaServer 2100 (large pedestal) and the
AJphaServer 2000 (small pedestal) servers from
Digital combine multiprocessing Alpha technology with an I/O subsystem traditionally associated with personal computers (PCs). The 110 subsystem in the
AlphaServer systems is based on the Peripheral
Component Interconnect (PC!) ami the Extended
Industry Standard Architecture (EISA) buses. All
AJphaServer products. including the AJphaServer
2100 cabinet version. share common technology and support at least three generations of the Alpha processor. In addition. the servers support three operating systems: Microsoft's Windows NT version
3.5. and Digital's DEC OSF/ 1 version 3.0 (and higher) and OpenV.VIS version 6.1 (and higher).
The AI phaSe rver svstems are designed to be general-purpose servers for PC local area network
(LAN) and database applications. All models of the system use a common multiprocessing bus inter connect that supports different numbers of nodes. depending on the system configuration. The systems share a common Cl'll. memory, and 1/0 architecture.
The number of CI'L!s. the amount of memory, the number of 1/0 slots. and the amount of internal stor age vary depending on the mechanical packaging.
The flexibility of the architecture allows the quick development of new anu enhanced systems.
This paper discusses the transformation of a set of requirements into high-performance, cost effective product implementations. The following section describes the evolution of the AlphaServer design from an advanced development project into a design project The paper then describes the CPU module, the multiprocessor system bus. and the memory module. Subsequent sections discuss module and silicon technology and the high availability features incorporated into the design .
The paper ends with a performance summary and conclusions about the project.
Concept Development
The engineering investigations of a client-server system originated from a business need that Digital perceived when it introduced the first systems to incorporate the Alpha technology in late 1992
Among Digital's first products in the server market were the OEC: 4000 high-performance departmental system. the DEC 3000 deskside workstation/server, and the ElSA-based Alpha PC. The lack of an explic itlv identified. general-purpose system for the mid range system market generated many requests from
Digital's MicroVAX rr system customers. Requests from these customers propelled the AlphaServer product development effort.
From the beginning of the project, two major constraints were evident: The schedule required a product by mid-1994. and the budget was limited.
Accordingly. the product team was required to leverage other developments or to find newer. less costly ways of achieving the product goals. Work on the AlphaServer systems started as a joint effort between an advanced development team and a business planning team. The business team devel oped market profil.es and a I ist of features without which the system would not be competitive. The business team followed a market-driven pricing model. The profit expected from the system dic tated the product cost for the system. This cost is referred to as "transfer cost." The business team's cost requirement was critical: if it could not be met. ll)/. 6 , \' u . j Slllll!ller 19')'1 Digital Te<"IJnical journal
Design of the AlphaSeruer Multiprocessor Seruer Systems the project wou ld be canceled . Furthermore, the entry- level system was required to l . Support at least two CPUs, with p erformance for a single CPU to yield 1 20 SPECmarks and 100+ transact ions per second benchmark.
(TPS) on the TPC-A
2. Support at least 1 gigabyte (c;B) of memory.
3. Support m ul t iple I/O buses with at least six option slots supported o n the base system.
4. Prov ide high-availabi l ity features such as red un dant power suppl ies, red u ndant array of inex pensive d isks
(RAJD),
"warm swap .. of drives, and clusteri ng.
'>. Provide a number of critical system connec tivity options, includ ing Ethernet, fiber d istrib uted data i nterface (FODI), and syn chro nous controllers.
6. Support the Windows NT, the DEC: OSF/1, and the
OpenVi'viS operating systems. c;iven these criteria, the engi neering team decided to base the development of the new server on concepts taken from two Digital products ancl combine them with the enclosures, power sup plies. and options commonly associated with PCs.
The DEC 4000 server is a m u ltiprocessor system wit11 a Futurebus+ l/0 subsystem; it provided the basis for the m u ltiprocessor bus design.1 The
DECpc 150 PC is a u n iprocessor system with an EISA
1/0 subsystem; i t provided a model for designing an
I/O subsystem capable of running the Windows NT operating system. The engineering team chose re style peripherals because of their low cost.
A strategic decision was made to incorporate the emerging PCJ bus into the product in addition to the EISA bus. Major PC vendors lud expressed high interest i n its development. and they believed the l'CI bus wou ld gain acceptance by the PC commu nity. The PCl bus provides a high-performance, low cost 110 chan nel that allows connections to many options such as small computer systems i nterface
(SCSI) adapters and other common PC peripherals.
After the i nitial design had been completed, chang ing market and competitive environments imposed additional requ irements on the design team.
I. The initial transfer cost goal was reduced by approximately 13 percent.
2. Support for a max imum of four processor mod ules was necessary.
To meet these new requ irements, the design team had to modify the system design dur i ng the product development phase.
System Overview
The base architecture deve loped for Digital's
AlphaServer m u ltiprocessor systems allows four processing nodes, four memory nodes (up to a max imum of 2GB), and two 1/0 n odes. All nodes com municate through a system bus. The system bus was designed to su pport multiple generations of
Alpha processor techno logy. The archi tecture can be i mplemented in d i fferent ways. depending on the size of the system packaging. It is flexible enough to meet a variety of market needs. Two implementations of the architecture are the
A lphaServer 2100 and the AlphaServer 2000 prod u cts. Figure 1 is a block d iagram of the AlphaServer
2100 implementation of the architecture .
I n t h e AlphaServer 2 100 large pedestal server. the system bus supports eight nodes. It is imple mented on a backplane that has seven slots. The seven slots can be configured to support up to four processors. Due to the number of slots avail able, the server supports only 1 GB of memory when four processors are installed. It supports the fuJI 2 GB of memory with three processors or less. The eighth node. which is the system bus to-PC:J bridge, is resident on the backplane. Th is provides a 32-bit J>C:I bus that operates at 33 mega hertz (MJ-Jz). It is referred to as the primary J>CI bus on the system.
A second 110 br idge can be i nstaJled in one of the system bus slots. This opti on, which will be available in 199'j, wil l provide a 64-bit PCI bus for the system. A 64-bit PC! is an extension of a 32-bit
PC! bus with a wider data bus. It operates at 33 ,\1Hz and is completely interoperable w ith the 32-bit PC! specification.
2
Options designed for the 32-bit
PC! bus will also work i n a 64-bit PC! s.lot.
EISA slots are supported through a bridge on the primary PCI bus on the system. Only one EISA bus can be supported in the system since many of the addresses used by EISA options are fixed.·1 Support of a single EISA bus is not perceived as an issue given the migration fro m the EISA bus to t he much higher performing PC! b us
.
The maximu m supported bandwidth on an EISA bus is 33 megabytes per second (iVIB/s) versus the maxim u m bandwidth on a 32-bit PCI bus of 1 32 MB/s. The EISA bus is used in the system for support of older adapters that have nor migrated to PC!.
Digital Technical journal Vol.
6
No.3
.\tl/11111er
I'J'J1
AlphaServer Multiprocessing Systems
SERIAL
CONTROL
BUS
80
<�======ll�===ll
PC I
BUS - 32 BITS ll
I
ETHERNET
I
8
EISA B U S r---------------1
I 8259A-2 I
: DDDDDD
I
KEYBOA RD--
MOUSE--
8242
KEYBOARD
AND MOUSE
CONTROLLER
PARALLEL
0
FLOPPY
SERIAL PORT
SERIAL PORT
I)
0
0
0
Figure I Block Diagram of the AlphaServer 2100
System Architecture
The AlphaServer 2000 small pedestal system sup ports five nodes on the system bus. The backplane provides four system bus slots, allowing a maxi mum configuration of two processor modules and two memory modules. The system bus-to-PCI bridge resides on the backplane and is the fifth node. A system bus slot can also he used to support the optional second 1/0 bridge.
The AlphaServer 2100 cabinet system is a rack mountable version of the larg e pedestal
AJphaServer 2100 system. The rackmountable unit provides a highly available configuration of the
10 11>1. 6 No
.
.i Siill/1111'1' 1')')4
Digital Teclmit:al journal
Design of the AlphaSeruer !Vlultzprocessor Seruer Systems pedestal system. It i ncorporates two separate back planes. One backplane supports eight system bus nodes that are i mplemented as seven system bus slots. The eighth node (the system bus-to-PCI bridge) resides on the backplane. The second back plane provides the 1/0 slots. The numbe r and configuration of 110 slots are identical to the
AlphaServer 2 100 pedestal system . The rackmount unit provides m inimal storage capacity. Addi tional storage is supported i n the cabinet version through
Storage\Xforks shelves. These storage shelves can be powered i ndependently of the base system unit. providing a h ighly available configuration.
Table 1 gives the specifications fo r the
AlphaServer 2 100 and the AlphaServer 2000 pedestal systems. I nformation on the cabinet version is not i ncluded because i ts characteristics are similar to the AlphaServer 2 100 large pedestal version. Al l multiprocessing members of the
AlphaServer fam i l y use the same processor a nd memory modu les and d i ffer only i n system packag i ng and backplane implementati o ns. This i l lustrates the flexibility of the archi tecture developed for the system a nd decreases the development time fo r n ew models.
CPU Module
The CPU modu le contains a n Alpha processor, a secondary cache, and bus i n terface appl ication specific i n tegrated circuits (AS1Cs). As previously mentioned, the system arch itecture al lows m u l t iple processor generations. M u l tiple variati o ns of the processor module are available for the system, but d i fferent variations cannot be used i n the same system. Software has timing loops that depend on the speed of the processor and cannot guara ntee
Table 1 AlphaServer System Specifications
Specifications Large Pedestal
AlphaServer
2100
System
Height, inches
Width, i nches
Depth, inches
Maximum DC power output,
watts per supply
27.6
1 6.9
31 .9
600
Number of system slots
Number of processors supported
M i nimum memory
Maximum memory
Embedded 1/0 controllers supported
Optional I/O control lers supported
32-bit PCI slots
64-bit PCI slots (on separate 1/0 controller modu le)*
EISA slots
Serial ports
Parallel port
Ethernet ports (AUI and 1 0Base-T)
7
4
32 M B
2 GB
3
2
1
1
8
2
Small Pedestal
AI
phaSe rver
2000
System
23.8
1 6.9
25.6
400
SCS I II controller
Removable media bays
I nternal warm-swap drive slots
·
Future option
1
3
1 6
4
2
32 M B
640 MB
1
1
3
2
7
2
Not integ ral
to system
1
2
8
Comments
Two possible per system in either redundant or current shared mode
Up to 18 total network ports supported on system via PCI and
EISA options
Digital Technical journal V!>i. li No
.
.
> Sllll/11/l'r 1'}')1 1 1
AlphaServer Mult iprocessing Systems synchronization between processors of d i fferent speeds. The CPU modu les provide a range of pert<>r mance and cost options for the system owner.
The cost-focused processor mod u le uses the
Alpha 21064 processor operating at 190 MHz. This chip was designed with Digital's fourth-generation complemen tary metal-oxide semiconductor (G<IOS) technology. It has sep arate o n-chip caches for i nstruction and data. The instruction cache hotels 8 kilobytes (KB) of memory. and the data cache hollb
8 KB. The 1-M13 second- level data cache is imple mented in 15-n a nosecond (ns) static random-access memory (SRA,VI) dev ices. It is a write-back, d irect mapped cache. The access rime ro the second- level cache is a m u l tiple of the CPU clock cycle. The use of 15 -ns SRA.VIs resulted in a read-and-write cycle time of 26. :'> ns to the secon d-le vel cache. Th is is a five-rimes multiple of the CPU cycle rime. The addi tional 1 1 .3 ns is needecl for rou nd- trip etch delay and address buffer delay. The use of 1 2-ns SRA:VIs was considered, bur the read-ancl-wrire cycle time wo uld have ro decrease to 21 ns to i mprove perfor nunce. The reduction of 3 ns was not sufficient to meet the riming requirements of the module: there fore, the less costly 1 "; -ns SRA.VI s were useu .
Higher performance processor modu les are also available for the system. These modules are based on the Alpha 2 1064A processor. which was designed using fifth-generation GviOS tt·chnology.
The Alpha 2I064A processor mod ule operates at
275 Iv!J-Iz. The processor has separate on-chip instruction and data caches. The 10-KB instruction cache is direct mapped, and the 1 6 -KB data cache is a 2-way, set -associative cache. The backup cache holds 4 MB of memory The combinatio n of higher processor speed, larger internal on-chip caches, ancl a large second- leveJ cache reduces the number of accesses to main memory and processes clara at a h igher rate. As a resu lt, the performance of the system is increased by approx imately 20 percen t .
Multiprocessor System Bus
The tech nology developed for the system bus in the
DEC 4000 cleparrmenral server provided t he basis for t he mu ltiprocessor bus designed for the
AlphaServer system . ' The system bus in the DEC
4000 product has the fol lowing features:
1. The 128-bit m u l tiplexed address and data bus operates at a 24-ns cycle time. The bus runs synchronously.
2. The bus supports two CPU nodes. four memory nodes, and a single 110 node.
3. The bus supports add ressing for block transfer only. A block is 32 bytes of data.
4 J/0 is treated as either primary or secondary.
Primary 1/0 refers to devices that cou ld respond without stal ling the system bus. This designation is res tricted mainly to con trol and status regis ters (CSRs) that exist on system bus nodes, e.g .. the I/O bridge.
:;. Al l 1/0 on remote buses is referred to as secondary
1/0 and is accessed via a mail box protocol .
M a i lboxes were invented ro hide slow accesses to CSRs on remote 110 buses.
A CSR read could potential ly take 1 to 10 micro seconds. which is very slow relative to the proces sor cycle time. The bus is " n onpendeu:· which means it wou ld sta l l during a slow access. When a bus stal ls, a l l accesses to CPUs and memories have to wait until the CSR access is complete. This cou ld cause data to back up and potentia l ly overflow. To avoid this stare, either the system bus or the soft ware device driver has to be pended.
A m a i l box is a software mechanism that accom pl ishes "device driver pend ing ... The processor bui lds a structure in main memory cal led the mail box data structure. It describes the operation to be pnformed , e.g .. CSR read of a byte. The processor then writes a pointer ro this structure into a mail box pointer register. The 1 /0 node on the system bus reads the mailbox uata structure, performs the operation specified . and returns status and any data to the structure in memory. The processor then retrieves the data from this structure and the trans action is complete. I n this way, the mailbox proto co.! a l lows software pend ing of CSR reads; it also al lows the software to pass byte information that is not avai lable from the Alpha 2 1064A processor. < '
Changes to the .�ystem Bus
Although t he DEC 4000 system bus proviued many features desi rable in a m u l tiprocessor interconnect, it did not meet the system requ irements defined during the concept phase of the AlphaServer proj t'Ct. Two major hurdles existed. One was the Jack of support for fou r CPlJs and m u l tiple J/0 nodes.
A second, more important issue was the incom patibi l ity of the mailbox 1/0 structure with the
Windows
NT
operating system.
The initial port of the Windows N T operating sys tem to t he DECpc 1')0 PC assumed direct- mapped
1/0. With direct mapping the 1!0 is physical ly mapped into the processor's memory map. and al l
1 2 l'o/. (, No . . )
S/111/JJ/<'1'
I'J'J4
Digital TeciJnical journal
Design of tbe AlpbaSerl'er Multiprocesso r Sen•er Systems
reads/writes to l/0 space are hand led as u ncached memory accesses. Clearly, this was incompatible with the nonpended bus, wh ich assumes the use of mailboxes. Consequently. the designers studied the advantages and d isadvantages of using mailboxes to determine if they shou ld be supported in the
W i ndows NT operating system. They found that the software overhead of manipu lating the mailbox structure made CSR accesses approx imately three t imes s lower than direct accesses by the hardware.
Thus the CPU performing the 110 access waits longer to complete. For this reason. the designers chose not to use mailboxes.
The designers also had to ensure that the system bus wou ld be avai lable for use by other processors while the 1/0 transaction was completing. To satisfy this requ ire ment, they added a retry mechanism to the system bus. The retry support was very simple and was layered on top of existing bus signals.
A retry condition exists when the CPU initiates a cycle to the l/0 that cannot be completed i n one system bus transaction by the I/O bridge. The
CPU
involved in the transaction is notified of the retry condition. The
CPU
then "backs otr the m u l tipro cessor bus and generates that transaction some period of time later. Other processor modules can access memory during the slow 110 transaction.
The retry procedure continues u ntil the 1/0 bridge has the requested data. At that stage, the data is returned to the requesting
CPU.
Byte Addressing Byte granu larity had been han d led in the mai lbox data structure. After the d irect map ped 1/0 scheme was adopted, the designers had to overcome the Jack of byte addressab i l ity in the Alpha architecture. Therefore, the designers participated in a col laborative effort across Digital to define a mechanism for adding byte add ress ability in the Alpha architectu re. The new scheme required the use of the four lower available Alpha
Ad: [08:05] address bits to encode byte masks and lower order address bits for the PC! and EISA buses.
For more details. see the paper on the AlphaServer
2 100 110 subsystem in this issue 6
The designers required a redefi nition of the add ress map. All 1/0 devices are now memory mapped . The Alpha 21064 A processor has a 34-bit address field that yields an address space of 16 GR.
This 16-CB address region may be subdivided into
4-GB quadrants. Each quadrant can be individual ly marked as cacheable or noncacheable memory. The
DEC 4000 system arch itecture split the 16-GB region in half: 8 (;B was al located as cacheable memory space and the remaining 8 GR as noncacheable space. Memory-mapped 1/0 devices are mapped into noncacheable space. The decision to support multiple 1/0 buses in the new systems together with the decision to memory map all I/O (i.e., no mailbox accesses) yielded a noncacheable memory require ment in excess of the 8 CB allocated in the DEC 4000 system. Therefore the designers of the AlphaServer systems changed the address map and allocated a single quadrant (4 GB) of memory as cacheable space and the remaining 12 GR as noncacheable.
These 1 2 GB are used to memory map the 1/0
Arbitration
The bus used in the
DEC
4000 system supports two
CPU
nodes and a single 1/0 node. To achieve the AlphaServer prod uct goals of m u l tiple
110 bridges and mu ltiple
CPU
nodes, the designers changed the address map to accommodate CSR space for these extra nodes and designed a new arbi ter for the system . The arbiter i ncl udes enhanced functionality to increase the perfor mance of future generations of processors. Some key features of the arbiter are l isted below.
1 . The arbiter is implemented as a separate chip on all processor modules. The logic was partitioned i nto a separate chip to accom modate a flexible architecture and to al low additional arbitrating nodes in the future. As many as fou r arbiters can exist in the system. Only one arbiter is enabled in the system. It is on the processor installed in slot
2 of the system bus.
2. 110 node arbitration is interleaved with CPU node arbitration. The a rbitration is rou nd robin and leads to an ordering scheme of CPU 0, 1/0,
CPU
1 ,
1/0,
CPU
2 , 1 / 0
,
CPU 3. 1/0. This scheme attempts to minimize 1/0 latency by ensuring many arbi tration slots for 1/0 devices. Processors sti ll have more than adequate access to the system bus due to the nature of 1/0 traffic (generally bursts of data in short periods of time). On an idle bus, the arbiter reverts to a first-come. first served scheme.
3. The arbiter implements an exclusive access cycle.
This allows an arbitrating node to retain the use of the system bus for consecutive cycles. This cycle is used by the 1/0 bridge in response to a
PCI
lock cycle. A I'Ci lock cycle may be generated by a device that requires an atomic operation, which may take multiple transactions to complete. For exa mple, the AlphaServer 2100 and AlphaServer
Digital Tecbnica/ ]ow-nat
Vol. o No . .> Summer 1994
AlphaServer Multiprocessing Systems
2000 systems use a PCI-to-EISA bridge chip set
(Intel 82 i 30 chip set).- This chip set requests a lock cycle on PC! when an an ato mic read-mod
EISA device requires
ify
-write operation.
The use of atomic read-modify-write operations is com moo in older 1!0 adapter designs. The 1/0 bridge on the system bus requests an exclusive access cycle from the arbiter. \Vhen it is granted. all buffers in the path to memory are flushed and the device has exclusive use of the l'CJ and the system bus until its transaction is completed. The use of this mode is not recommended for new adapter designs due to the unfair nature of its tenure on the system bus. It was implemented in the AlphaServer product design to support older EISA devices.
Memory Module
Main memory is accessed over the system bus either by processors (after missing in their on-board caches) or by 1/0 nodes performing direct memory access
(DMA) transactions. They are called com manders.
The memory controller incorporates a number of performance-enhancing features that reduce latency in accessing the dynamic RAoVI (DRA.vl) array. One concept used is called a stream buffer. Stream buffers reduce the read latency to main memorr
Reads to main memory normally require 9 to
10 cycles on the system bus, depending on the speed of
DRAMs in the array. The use of stream bufters reduces this time to 7 cycles. The stream buffers provide a facility to load data fetched from the DRANI array prior to the receipt of a read request for that data.
A stream is detected by monitoring the read addresses from each com mander on the system bus. The logic simply keeps a record of the memory addresses of the previous eight read transactions from each com mander and compares each subse quent read address to see if the new address is con tiguous to any of the recorded addresses. If a new address is determ ined to be contiguous to any of the previ ous eight addresses. a new stream is declared. As a result, one of the stream buffers is allocated to a new stream.
A stream buffer is implemented as a four-deep, first-in, f irst-out (FIFO) buffer. Each entry in the
FIFO buffer is 32 bytes. which is equivalent to the system bus l ine size. Each memory module con tains four stream buffers that can be al located to dif ferent com manders. A least recently used (LRU) algorithm is used to al locate stream bu tie rs. When a new stream is detected, or an existing stream is empty. the stream buffer fills f rom the DRAM array by using successive acklresses from the head of the stream. After a buffer has been allocated and some amount of data has been placed in the FIFO buffer,
" hit" logic compares incom ing read addresses from the system bus to the stream address. If a compari son of these two addresses is successful, read data is del ivered from the memory module without incurring the latency of accessing the DRAM array.
An invalidation scheme is used to ensure that the stream bu ffers stay coherent. Write cycle addresses are checked to see if they coincide with a stream buffer address. If the write address i s equal to any address currently in the stream buffer. that entire stream buffer is declared invalid. Once it is invalidated, it can be reallocated to the next detected stream.
Wri tes to main memory complete on the system bus in six cycl .
es, which is achieved using write buffers in the mem ory controller. The write transac tions are essentiall y "dump and run." The total write buffering available in each memory module is 64 bytes. which is large enough to ensure that the sys tem bus never has to stall during a write transaction.
The implem entation of the memory module dif fers from the AlphaServer 2 100 to the AlphaSe rve r
2000 system. Both memory modules conta in the same memory controller ASIC:s, but the implemen tation of the !WA.V! array is different. Due to space constraints on the AlphaServer 2100, the DRAM array was i mplemented as a flat, two-sided surface mount module . On the AlphaSe rver 2000, single in-line memory modules (SIMMs) were used for the
D RAM array. Memory module capacities vary from
32 .V!B to 512 ,VI B. The Al phaServer 2100 system pro vides four system bus slots that can be populated with memory modules. The m:�xi mum supported configuration is 2 GB with four memory moclu les.
This l i m its the maximum system configuration to three processors since one of the processor slots must be used as a memory slot. The AlphaServer
2000 system prov ides two system bus slots that can be populated with memory. The maximum memory supported in this system is 640 M B . This configuration consists of one 5 12-MI3 module and one 128-MB module. The maxi mum memory con straint is dictated by the power and cooling avail able within this system package . The A lphaServer
2000 sti!J supports two processor m odules when configured with maximum memory. F igure shows a block diagram of the AlphaServer
2
2000 memory module.
1 4 1-b/. 6 !lio .
.
)
.\11111111a
/'}')4
Digital TechHical ]ourual
Design of the AlpiJaSe,-·uer Multiprocessor Server Systems
SERIAL CONTROL BUS
TO MEMORY MODULES AND CPU MODULE
I
S E R IAL CONTROL BUS
EEPROM
I
BANK 3
EIGHT X36 S I MMs
1r
I
BANK 2
EIGHT X36 S I MMs
I
BANK 1
EIGHT X36 S I M Ms
I
BANK 0
E I G HT X36 SIMMs
DATA PATH
I
1r
1r
1r II fl'
11'11 11
llJlllll ll
ADDRESS 1 28 DATA
AND 1 2 EDC r
AND CONTROL
DRIVERS
rv----
J
I
J
I
DATA PATH
1 28 DATA
AND 1 2 EDC
1
SYSTEM BUS TO MEMORY MODULE. 1/0 INTERFACE. AND CPU MODULES
Figure 2 Block Diagram of the AlphaServer 2000 Mernory Module
Technology Choices
This section briefly discusses some of the decisions and tr:otdc-offs made concerning module and sil icon techno logy used in the systems.
Module Technology
The designers partit ioned the logic into mod ules fo r two reasons:
(I)
Removable processor and memory modules al low for instal lation of addi tional memory and processors and (2) They also al low for easy upgrade to faster processor speeds. Since modu larity adds cost to a system, the designers decided that the
I/O subsystem (EISA and PCI logic) should reside on the backplane. They deviated from this stra tegy for the AlphaServer 2100 system design because the PCI to-EISA bridge was a new, unfamiliar design. Fixing any problems with this chip set or any of the supporting logic would have required a backplane upgrade, which is a time-consuming effort. For this reason, the engineers chose to build an 1/0 modu le for the
AlphaServer 2100 system that contained the PCI-to
EISA bridge; associated control logic; control lers for mouse, keyboard , print er, and floppy drive; and the integral Ethernet and SCSI control lers. This modu le was eliminated in the AlphaServer 2000 system due to the design stabil ity of the 1/0 modu le .
The Metra! co nnector specified by the
Futu rebus+ specification was chosen for the sys tem bus implementation on the DEC 4000 prod uct.
This choice was consistent with the design of the
DEC 4000 server, which is a Futurebus+ system.
Cost studies u ndertaken d u ri ng the i nitial design of the AlphaServer 2 100 system showed that the cost per pin of the Metra! connector was high and added a significant cost to the system. The team decided
Digital Technical journal Vol. o Nu. j Summer 1994 1 5
AlphaServer Multiprocessing Systems ro invest igate the use of either the PCI or the EISA connector for the system bus, since both connec tors are used w idel y in the system . The PCI con nector is actually a variant of the MicroChannel
Architecture (MCA) connector used in m icrochan nel sys tems. SPICE simulations showed that it per formed better than the Metra! connector on the
F u t urebus+ .
H The team chose a 240-pin version of the connector for im plementation because it met the system requirements and had a low cost
Due to the choice of
.
the MCA
connectOr, the board thickness was limited to a maximum of 0.062 inches. An 8 l ayer layup was cho s en for the module technology. The processor modules had a require ment for both a 5.0-V supply and a 3.0-V supply.
The de signers chose a spl i t plane to distribute the power rather than two separate power planes fo r each volt age. Ro u ting high-speed signals across the split was minimized to redu ce any emissions that m ight arise from using a split plane. Testi ng later val idated this approach as emissions from this area were m inimal.
Silicon Technology
The system partitioning required the design of four
ASJCs. These were the CPU bus interface ASIC the memory bus interface ASIC. the sy s tem arbiter. and the system bus - to-PCI b ridge. The DEC 4000 imple mentation of the F ut u rebus + used an externally supplied gate-array process that was customized to meet the performance needs of the bus and the per formance goa l s o f the first A l pha systems. Gate array costs are de ter mined by the number of chips that are produced on the chosen gate-array process.
The volume of chips produced by the gate-array process for the DEC 4000 system was low because the process was specially adjusted fo r that system application. As a resu lt. the volu m e of chips was dire c tly proportional to the volume of the DEC 4000 systems built. Therefore . the cost per component produced by this process was relatively high.
I f t hey had used this customized gate-array pro cess, the designers of the AlphaServer product could not have met t h eir cost goals. They needed a more generic process that could produce ch ips that many system vendors could use . This would ensure t hat the line utilization was high and that t he cost per component was low. Therefore. they changed the technology to one that is s ta ndard i n the industry. Gate-array proc ess technology had evolved since the DEC 4000 design. and a standard te chnology that was capable of meeting the system timing requirements was available. Extensive SPICE simulations verified the process capability. ASICs that were implemented with this process had no d iffi culty meeting the bus timing H
Another intere s tin g feature of the analog design on the AJphaServer 2 100 system involves the sup port of 11 loads on the PCI . The PCI spe c if ication recommends 10 loads as the "cookbook" design 2
T he system requ irement on the AlphaServer 2100 was ro support three PCI slots. the integral PCI
Ethernet chip. the NCRR l O (l'Cl-to -fasr -scsr con troller), and the P CI toE l S A bridge. Each PCI connector has been modeled to be equivalent to two electrical loads. Taking account of the system bus- to-P(] bridge and the additional load con tributed b y the 1!0 module con nector yielded a PC! bus w ith 1 1 el ectrical loads. Extensive SPICE simu lations of the bus and careful routing to ensure a short bus guaranteed that the new design wou ld meet the electrical specifications of the PCI bus H
System Start-up
The design te am incorporated many availability fea tures in to the AlphaServer 2 100 and AlphaServer
2000 servers. These included support of "hot -swap" storage devices that can be removed or installed
\vhile the svstem is operating. error correction code
(ECC)-protected memory. redundant power sup pi ies. and CPl
· recove n·. P erhaps the most interest ing part of the design for availabil ity was the emphasis on ensuring that the system l1ad enough built-in recovery and redundancy to al low i t to r emain io a u sable or diagnosable state. Large sys tems sometimes have complicated paths in which to access the initial start-up code. and a system fa il ure in that path can leave the owner with no visible fa ilure ind ication. Moreover. in a multiprocessor system with more than one C:Pli instal led. it is highly desirable to initialize the resident firmware and the operating system even if a l l CPUs are not in working order. The AlphaServer 2 100 and 2000 sys tems employ two schemes to help achieve this goal .
The starr-up code for the A lphaServer 2100 ancl
AlphaServer 2000 systems is located in flash read only m emory
(ROM),
which resides on a peripheral bus behind the PCI-to-EISA br idge . In starting up a multiproces s ing operating system. only one processor is designated to access the start-up code and i nitial i ze the operating system. This is referred to as the primary processor. Accessing the start-up code requires the processor. system bus, memory, and most of the 1/0 subsystem to be functional . l <l l'r1/. (> No. ,) Sllllllner I'J'J4 Digital Tee/mica/ journal
Design of the AlphaSerl'er Multiprocessor Server Systems
The AJphaServer systems have a number of fea tures that help make the start-up process more robust. Each processor module contains a separate maintenance processor implemented as a simple microcontroller that connects to a serial bus on the system. The serial bus is a two-wire bus that has a data line and a clock line. On power-up the pro cessor mod ule performs a number of diagnostic tests and logs the resu lts in an electrical ly erasable programmable read-o nly memory (EEPROM) on the module. This EEPROM resides on the serial bus. If a CPU fa ils one of its power-up tests or if it has an error logged i n its EEPROM, then it is not allowed to be the pri mary processor. Assume that four CPUs are instal led in the system; if only CPU 0 fa i ls, then
CPU 1 is the primary processor. I f CPU 0 and CPU 1 fa il, then CPU 2 is the primary processor. If CPU 0,
CPU I , and CPU 2 fail, then CPU 3 is the primary pro cessor. If a l l four CPUs fai l , then CPU 0 is the primary processor. If any one of the CPUs fails, a message is displ ayed on the operator control pa nel to i nform the user that there is a problem. Any secondary CPU that has failed is d isabled and wil l not be seen by the firmware console or the operati ng system. The pri mary processor then uses the system bus to access the start -up code in the flash ROM.
The flash ROM may contain incorrect data . The fl ash ROMs on many systems have a program update, and errors from a power spike or su rge can be in troduced into the ROM cocle during the update procedure. User error is a nother common way to i ntroduce data error; for example, a user can acci dentally press a key while the update program is running. Flash ROMs can also fail from intri nsic manu facturi ng faul ts such as current leakage, which will eventually convert a stored " 1 " into a stored " 0," thus corru pting the program stored i n the flash ROMs. Many techniques in the industry partia lly solve the problem of corrupted flash ROM data. One well-known technique uses a checksum and reports an error to the user if the data is not cor rect. Another technique provides a second set of flash ROMs and a switch that the user manipul ates to transfer control to the new set in the event of a failure. The designers stud ied many previously used methods, but rejected them si nce they required intervention by the user.
In the AlphaServer 2 100 and the AlphaServer
2000 system design, the design team implemented a scheme that did nor requ ire user intervention i n the event of flash ROM corruption. The system has
1 1v1B of flash ROM of which the first 'il2 KR co ntain the system initial ization code. This code is loaded into main memory, and many data integrity tests are performed . These i nclude single and m u ltiple bit parity checks, various d ata correction code check i ng, and a checksu m cal cu.lation. The processor detects an error if the checksum calcu lation fails, i .e . , if the calculated value is not equal to the stored value. The processor then writes a value to a regis ter on the 1/0 mod ule, which automatical ly changes the address pointing to the flash ROM to a second bank of flash ROM . This combination of hardware and software su pport prov ides a way for the
AlphaServer 2100 system user to overcome any flash ROM corruption.
Design Considerations for the
AlphaServer
2000
System
The design of the AlphaServer 2000 small pedestal system fo l lowed the AJphaServer 2100 system.
Market pressures d ictated the need for a smal ler system with a lower entry-level cost. The in troduc tion of the smal ler server was schedu led to coin cide with the release of the Windows NT version 3.5 operating system.
An examination of the AlphaServer 2100 develop ment schedule revealed the fol lowing interesting points: (1) System power on occurred nine months after the team was formed: (2) Initial system ship ments occurred e ight months later; (3) The eight month time period was spent mainly in porting and qual ifying operating system software.
Based on these facts, the system designers bel ieved that rhe key to reducing the t ime-to-market of the AlphaServer 2000 system was to eliminate the dependency on synchronizing the design schedule with an operating system release. Consequently, the new system could not requ ire any software changes at the operating system level. Any changes would have to be transparent ro software. To ach ieve this, the designers took advantage of a new feature in the
DEC OSF/1 and the OpenVMS operating systems cal led dynamic system recognition (DSR).
A DSR machine is defi ned as a machine that requires no new software development. Operat ing systems, however, requi re l icensing; this information is dependent upon the system model num ber. There are two components to build ing a DSR machine.
1. A programmer's view of the machine mu st be a subset of an a l ready supported machine. I n the case of the A lphaServer 2000, the designers
Digital Techuicaljourual Vt1/. 6 No. J S11111111er 1994
1 7
AlphaServer Multiprocessing Systems decided to make it a subset of the AlphaServer
2100. A clear u nderstand ing of how the operat i ng systems initial ized the AlphaServer 2100 sys tem was critical to u nderstanding what changes could be made. A joint team of hardware and software engineers exa mi ned various pieces of the code to identify the areas of the system design that cou l d be changed. I nvestigations revealed that the system bus configuration code for the AlphaServer 2100 is somewhat generic.
It assumes a maximum of eight nod es, which is the AlphaServer 2100 implementation. The 1!0 node to the primary PCI bus is expected to be present. The presence of addi tional processors and memories is detected by read i ng the CSR space of each module. A modu le that is present gives a posi tive acknowledgment. The design team could therefore reduce the nu mber of sys tem bus slots from seven to fou r. This had no effect on the software since nonexistent slots wou ld merely be recognized as modules not insta l led in the system.
The physical packaging of the AlphaServer 2000 also d ictated that the number of 1!0 slots be red uced from 11 (8 EISA and 3 PCI) to 10. Given the industry trend toward PCI, the desirable mix would have been 6 EISA slots and 4 PC! slots. The
PC! bus configu ration code searched for as many as 32 PC! slots, which is the number al lowed by the PC! specification. 2 After careful consid eration, the designers determined that the addi tion of another PC! slot wou ld invo lve a change i n in terrupt tables to accommodate the addi t ional interrupts and vectors required by the addit ional slot. Therefore, the team decided to implement 3 PC! and 7 EISA slots.
2. The other component to building a DSR machine is to prov ide the system model number to the operating system so that li censing information can be determi ned. The system resident code that runs at start-up is referred to as the console.
The console and the operating systems commu nicate via a data structure known as the hard ware parameter block (H\VRPB). The HWRPB is used to communicate the model number to the operating system, which uses this nu mber to provide the correct licensing information.
The AlphaServer 2000 system was completed i n approximately nine months. Qual ification was not dependent on the operating system schedules. By building a DSR machine, the design team met the project's time-to-market requirements.
Performance Summary
Table 2 summarizes the performance of the systems descri bed in this paper. The numbers are heavily i nfluenced by the processor speed, cache, memory, and 1/0 subsystems. The systems exceeded the per for mance goals specified at the beginn ing of the project. In some cases the important benchmarks that had been relevant i n the industry changed d ur ing the course of system development. I n t h e trans action processing measurement, for example, the
TPC-A benchmark was superseded by the TPC-C benchm ark.
The AlphaServer 2100 server was the price pe rformance leader in the industry at the time of its i ntroduction in April 1994 . Successive i mprove ments in processor and 110 subsystems should help the AlphaServer 2100 and 2000 products maintain that position in the industry
Table
2
System Performa nce
AlphaServer
2100 412:15
SPECi nt92'
SPECfp92'
AIM lllt
Number of AIMs
User loads
Estimated TPSt
200.1
291 .1
227.5
1941.2
850
AlphaServer
2000 4/200
1 31.8
161.0
177.5
1516.0
660
Notes:
•
Single-processor system only t Dual-processor system only
:j:
TPS is an abbreviation for transactions per second. These numbers are estimated for a quad-processor system using
OpenVMS version 6.1 running Rdb.
Conclusions
The design team exceeded all the product require ments set at the beginning of the AlphaServer proj ect. The transfer cost of the final product was 10 percent better than the goal. The reduced cost was achieved despite t he erratic price l evels for DRAMs, which were much h igher in 1994 than pred icted in late 1992
Separate cost targets were establ ished for each portion of the system, and each design engineer was responsible for meeting a particu la r goal .
1 8 Vol. II No. 3 Summer 1994 Digilitf Technical journal
Design of the Alpha Server Multiprocessor Server Systems
Constant cost reviews ensured that variances could be quickly addressed. The requirement to run three operating systems quickly expanded the size and scope of the project. The operating system devel opers became an integral part of the design team.
Multiple reviews and open communication between the hardware development team and the software groups were essential to managing this work. The hardware team performed system-level testing on all three operating systems. This proved invaluable in tracking down bugs quickly and resolving them in either hardware or software.
The project team delivered the expected perfor mance and fu nctionality on schedule. Develop ment time was allocated for new power and packaging subsystems (using third-party design companies), new modu les, new ASICs, new system firmware, and porting of three operating systems.
To attain the sched u le, development tasks were frozen at the beginning of the project. The tasks were also categorized into three classes: mandatory, nonessential, and disposable. Consequently, engi neers were able to make trade-offs when required and maintain the integrity of the product. Another key factor to meeting the schedule was the use of knowledge and technology developed for previous products. This yielded m any benefits: less design time, fewer resources required, known simulation environment, and less time to a working prototype.
References and Note
1.
B . Maskas, S. Shirron, and N. Warchol, "Design and Performance of the DEC 4000 AXP Depart mental Server Computing Systems,"
Digital
Technical journal,
vol. 4, no. 4 (Special Issue,
1992): 82 - 99
2.
PC! Local Bus Specification, Revision 2. 0
( Hil ls boro, OR: PC I Special Interest Group, Order No.
281446 -001 , April 1993).
3. E. Solari,
!SA and EISA, Theory and Operation
(San Diego, CA: Annabooks, 1992).
4. R. Sites, ed. ,
Alpha Architecture Reference Man ual
( Burlington, MA: Digital Press, Order No.
EY-L520E-DP, 1992).
5. DECchip 21064 Microprocessor Hardware Refer ence Manual
(Maynard, MA: Digital Equipment
Corporation, Order No. EC-N0079-72, 1992).
6. A. Russo, "The AlphaServer 2100 1/0 Subsystem,"
Digital Technical journal,
vol. 6, no. 3 (Summer
1994, this issue): 20 -28.
7.
82420/82430 PC!set /SA and EISA Bridges
(Santa
Clara, CA: Intel Corporation, 1993).
8.
SPICE
is a general-purpose circuit simulator pro gram developed by Lawrence Nagel and Ellis
Cohen of the Department of Electrical Engineer ing and Computer Sciences, University of Cali fornia, Berkeley.
Acknowledgments
Many people contributed to the success of this proj ect. They spent long weekends and nights working to a schedule and a set of requirements that many thought were unachievable. The inspired dedica tion of team members made this project a reality.
Although not complete, the fol lowing l ist credits those who were the most deeply involved: Vince
Asbridge, Kate Baumgartner, Rachael Berman,
Jack Boucher, John Bridge, Bob Brower, Harold
Buckingham, Don Caley, Steve Campbell, Dave
Carlson, Mike Chautin, Marco C iaffi, Doug Field,
Rich Freiss, Nitin Godiwala, Judy Gold, Paul
Goodwin, Tom Hunt, Paul jacobi, Steve jenness, Jeff
Kerrigan, Wil l Kerchner, Jeff Metzger, John Nerl,
Mike O'Neil l , Kevin Peterson, Bryan Porter,
Ali Rafiemeyer, Lee Rid lon, Andy Russo, Stephen
Shirron, Andrei Shishov, Gene Smith, Kurt Thal ler,
Frank Touserkani, Vicky Triolo, and Ralph Ware.
Special thanks also to our manufacturing team in
Massachusetts, Canada, and Scot land.
Digital Technical journal
Vol. 6 No. 3 Summer 1994
1 9
Andrew R Russo I
The AlphaServer
2100
1/0 Subsystem
The AlphaServer 2100 110 subsystem contains a dual-level 1/0 stmcture that includes the high-powered PC! local bus and the widely used EISA bus. The PC! bus is connected to the server's multiprocessing system bus through the custom-designed bridge chip. The EISA bus supports eight general-pwpose EISA/ISA connectors, pro viding connections to plug-in, industry-standard options. Data rate isolation, dis connected transaction, and data buffer management techniques were used to ensure bus efficiency in the 110 subsystem. Innovative engineering designs accom plished the task of combining Alpha CPUs and standard-system 1/0 devices.
Digital's AlphaServer 2100 server combines Alpha multiprocessing technology with an 110 subsystem typica l ly associated with personal computers
(PCs). 1 The 1/0 subsystem on the AlphaServer 2100 system contains a two-level hierarchical bus struc ture consisting of a h igh-performance primary
J/0 bus connected to a secondary, lower per formance 1/0 bus. The pri mary 110 bus is a 32-bit peripheral component i nterconnect (PC!) local bus
(or simply, PCI bus) Z The PC! bus is connected to the AlphaServer 2 100 system's m ultiprocessing system bus tl1rough a custom appl ication specific integrated circuit (ASIC) bridge chip (referred to as the T2 bridge chip). The secondary 1/0 bus is a
32-bit Extended Industry Standard Architectme
(EISA) bus connected to the PC! bus through a bridge chip set provided by Intel Corporation.:�
Figure 1 shows the 1/0 subsystem designed for the
A1phaServer 2 100 product. The r;o subsystem demonstrated sufficient flexibil ity to become the
1/0 interface for the sma l l pedestal AlphaServer
2000 product and the rackmountable version of the
AlphaServer 2100 server.
This paper discusses the dual- level bus h ierarchy and the several 1/0 advantages it provides. The design considerati o ns of the 1!0 subsystem for the
AlphaServer 2100 server are exami ned in the sec tions that follow.
1/0 Support for EISA atul PC/ Buses
The EISA bus enables the AlphaServer 2100 system to support a wide range of existing EISA or Industry
Standard Architecture (!SA) l/0 peripherals
1
The
EISA bus can susta in data rates up to a theoretical limit of 33 megabytes per second (MB/s) at a clock rate of 8.25 megahertz (MHZ). I n the current config uration for the AlphaServer 2 100 product, the EISA bus supports eight general-purpose EISA/ISA con nectors, and the EISA bridge chip set provides connections to various low-speed, system-standard
1/0 devices such as keyboard, mouse, and time-of year (TOY) clock. For most system configurations, t he AlphaServer 21 00 system ·s EISA bus provides enough data bandwidth to meet all data throughput requ irements. In light of the new requ irements for faster data rates, however, the EISA bus will soon begin to run out of bus bandwidth.
To provide for more bandwidth, the AlphaServer
2100 system also contains a PC! bus as its primary bus. With data rates four times that of the EISA bus, the PC! bus provides a direct m igration path from the EISA bus. The 32-bit PC! bus can sustain data rates up to a theoretical I imit of 132 MB/s at a clock rate of 33 M Hz. In the AlphaServer 2100 system configuration, the PC! bus provides connections to three general-purpose 32-bit PCI connectors, an
Ethernet device, a SCSI device, the J>CI-to-EISA bridge chip, and the T2 bridge chip.
A close examination of the bus structure reveals that the AlphaServer 2100 system actual ly contains a three-level, hierarchical bus structure. In addition to the PCI and EISA buses, the AlphaServer 2 1 00 sys tem includes a 128-bit multiprocessing system bus, as shown in Figure 1. Each bus is designed to adhere to its own bus interface protocols at different data rates. The system bus is 128 bits per 24 nanosec onds (ns); the PC! bus is :)2 bits per 30 ns; and the
EISA bus is 32 bits per 120 ns. Each bus is required
20 Vol. 6
No. 3 Slimmer 1')94 Digital Tecbnical journal
The AlphaServer 2100
l/0
Subsystem
MEMORY CPUs
II
SYSTEM BUS
1 28
BITS
11
THREE
32-BIT
BRIDGE
(T2 ASIC)
PCI SLOTS
I I
PCI BUS
H
II
32 BITS
II
E I G HT
32-BIT EISA
SLOTS
B R I D G E
( I NTEL)
EISA BUS
I I
32 BITS
J l
SCSI
C H I P
II
Jl
SYSTEM
STANDARD
1/0
DEVICES
ETH ERNET
C H I P
I�
1/0
S U BSYSTEM
Figure I l/0 Subsystem for the AlphaServer 2100 System to p rovide a particular fu nction to t he syste m and is positioned in the bus hie rarchy to maximize that efficiency. For example, the system bus is positioned close to the c rus and memory to maxi mize CPU memory access time, and the lower per fo rmance 1/0 devices are placed on the EISA bus because their tim ing requirements are less critical.
To main tain maximum bus effi ciency on all three buses, it is critical that each bus be able to perform its various fu nctions autonomously of each other.
In other words, a slowe r performing bus shou ld not affect the efficiency of a high-performance bus. The section below d iscusses a few techniques that we designed into the l/0 subsystem to enable the b uses to work together efficiently.
Using the Bus Hierarchy Efficiently
This section d iscusses the data rate isolation, d is con nected transaction, data buffer management, and data bursting techniques used to ensure bus efficiency in the l/0 subsystem.
Data Rate Isolation
The three-level bus hierarchy promotes data rate isolation and concurrency for simul taneous opera tions on all three buses. The design of the bus bridges he lps to enable each bus to work i ndependently: it provides bus interfaces with extensive data buffering that fu nction at the same data rates as the interfacing bus. For example. the T2 bridge chip contains both a system bus interface and a PC! bus interface that run synchronously to their respective buses but are tot a l ly asynchronous to each other. The data buffers i nside the T2 bridge chip act as a domain con nector from one bus t ime zone to the other and help to isolate the data rates of the two buses.
Disconnected Transactiom
Whenever possible, the bridges promote the use of disconnected (or pended) protoco ls to move data across the buses. Disconnected protocols decrease the interdependencies between the d ifferent buses.
For example, when a CPU res iding on the system bus needs to move data to the PO bus, the CPU does so by sending its data onto the system bus. Here the
T2 bridge chip (see F igure 2) stores the data into its internal data bu ffers at the system bus data rate. The T2 bri dge chip provides enough buffering to store an entire CPU transaction. From the CPU 's perspective , the transaction is completed as soon as the T2 bridge chip accepts its data. At that point, the T2 bridge chip must forward the data to the PC! bus, independent of the CPU . In this way, the CPU
Digital Technical journal
Vol. 6 No. 3 Srmtmer /1)')4 2 1
AlphaServer Multiprocessing Systems
SYSTEM
BUS
1 28
SYSTEM
B U S
CORNER
LOGIC
SYSTEM BUS
COMMANDER
B U F F E R
CONTROLLER
[:J
PCI BUS
MASTER
DMA READ,
D M A W R I T E , AND
PROGRAMMED
1/0 BUFFERS
BUFFER
CONTROLLER
PCI BUS
CORNER
LOG IC
SYSTEM BUS
R ESPONDER
PCI
BUS
32
:>
Figure 2 Block Diagram of the T2 Bridge Chip is not required to waste bus bandwidth by waiting for the transfer to complete to its final destination on the PC! bus.
The T2 bridge chip implements disconnected transactions for all CPU-to-PCI transactions and most
PCI-to-memory transactions. In a similar fashion, the PCl-to-EISA bridge implements disconnected transactions between the PC! bus and the EISA bus.
Data Buffer Management
In addition to containing temporary data buffering to store data on its journey from bus to bus, each bridge chip utilizes bu ffer management to allocate and deallocate its internal data buffers from one incoming data stream to another. In this way, a single
ASIC bridge design can efficiently service multiple data streams with a relatively small amount of data buffering and without impacting bus performance.
The T2 bridge chip contains 160 bytes of tempo rary data buffering divided across the three specific bus transactions it performs. These three transac tions are
(I)
direct memory access (DMA) writes from PC! to memory (system bus), (2) DMA reads from memory (system bus) to PC!, and (3) pro grammed 1/0 (system bus) reads/writes by a CPU from/to the PCI. The T2 bridge chip's data buffering is organized into five 32-byte buffers. Two 32-byte buffers each are al located to the DMA write and
DMA read functions, and one 32-byte buffer is allo cated to the programmed 110 function. Each of the three t ransaction functions contains its own buffer management logic to determine the best use of its available data buffering. Buffer management is especially val uable in situations in which a PCI device is reading data from memory on the sys tem bus. To maintain an even flow of data from bus to bus, the buffer management inside the T2 bridge chip attempts to prefetch more read data from memory while it is moving data onto the PC!.
Buffer management helps the bridges service bus transactions in a way that promotes continuous data flow that, in turn, promotes bus efficiency.
Burst Transactions
Using a bus efficiently also means utilizing as much of the bus bandwidth as possible for "useful" data movement. Useful data movement is defined as that section of time when only the actual data is moving on the bus, devoid of add ress or protocol cycles.
Maximizing useful data movement can be accom pl ished by sending many data beats (data per cycle) per single transfer time. Send ing multiple data beats per single transfer is referred to as a "burst transaction."
All three buses have the ability to perform burst transactions. The system bus can burst as much as
32 bytes of data per transaction, and the PC! and
EISA buses can burst continuously as requ ired.
Data bursting promotes bus efficiency and very high data rates. Each bus bridge in the server is required to support data bursting.
The Bus Bridges
In the previous section, we discussed certain design techniques used to promote efficiency within the server's hierarchical bus structure. The section that fol lows describes the bus bridges in more detail, emphasizing a few interesting features.
2 2 .i Summer 1994 Digital Technical journal
The AlphaServer 2100
110
Subsystem
The T2 Bridge Chip
The T2 bridge chip is a specially designed ASIC that provides bridge functionality between the server's multiprocessing system bus and the primary PC! bus. (See Figures 1 and 2.) The T2 ASIC is a 5.0-volt chip designed in complementary metal-oxide semi conductor (CMOS) technology. I t is packaged in a 299-pin ceramic pin grid array (CPGA).
As stated earl ier, the T2 bridge chip contains a
128-bit system bus interface running at 24 ns and a 32-bit PC! interface running at 30 ns. By using these two interfaces and data buffering, the T2 bridge chip translates bus protoco ls in both directions and moves data on both buses, thereby providing the logical system bus- to-PCI interface (bridge). In addi tion to the previously mentioned bridge features, the T2 bridge chip integrates system functions such as parity protection, error reporting, and CPU-to
PCI add ress and data mapping, which is d iscussed later in the section Connecting the Alpha CPU to the
PC! ancl EISA Buses.
The T2 bridge chip contains a sophisticated DMA controller capable of servicing three separate PCI masters simultaneously. The DMA contro l ler sup ports different-size data bursting (e.g., single, mu lti ple, or continuous) and two kinds of DMA transfers, direct mapped and scatter/gather mapped . Both
DMA mappings allow the T2 bridge chip to transfer large amounts of data between the PCI bus and the system bus, independent of the CPU.
Direct-mapped DMAs use the address generated by the PC! to access the system bus memory directly.
Scatter/gather-mapped DMAs use the add ress gener ated by the PC! to access a table of page frame num bers (PFNs) in the system bus memory. By using the
PFNs from the table, the T2 bridge chip generates a new address to access the data . To enhance the per formance of scatter/gather-mapped DMAs, the T2 bridge chip contains a translation look-aside buffer
(TUl) that contains eight of the most recently used
PFNs from the table. By storing the PFNs in the TLB, the T2 bridge chip does not have to access the table in system bus memory every time it requi res a new
PFN. The TLB improves scatter/gather-mapped DMA performance and conserves bus bandwidth. Each entry in the TLB can be individually inval idated as required by software.
The T2 bridge chip also contains a single
110
data mover that enables a CPU on the system bus to initi ate data transfers with a device on the PC! bus. The
110
data mover supports accesses to all the valid PC! address spaces, includ ing PC!
110
space, PCI mem-
ory space, and PCI configuration space. The T2 bridge chip supports two l/0 transaction types when accessing PC! memory space : sparse-type data transfers ancl dense-type data transfers. Sparse type transfers are low-performance operations consisting of 8-, 16-, 24-, 32-, and 64-bit data tra ns actions. Dense-type transfers are high-performance operations consisting of 32-bit through 32-byte data t ransactions. Dense-type transfers are especially useful when accessing 110 devices with large data buffers, such as video graphics adapter (VGA) con t rollers. A single PC! device mapped into PC! mem ory space can be accessed with either sparse-type operations, dense-type operations, or both.
In addition to accessing the PCI, a CPU can access various T2 bridge chip internal control/status regis ters (CSRs) for setup and status purposes. For maxi mum flexibil ity, al l the T2 bri dge chip's fu nctions are CSR program mable, a l lowing for a variety of optional features. Al l CPU I/O transfers, other than those to T2 bridge chip CSRs, are forwarded to the
PC! bus.
Intel PC!- to-ElSA Bridge Chip Set
The Intel PCI-to-EISA bridge chip set provides the bridge between the PC! bus and the EISA bus.
I
It inte grates many of the common 1/0 functions found in today's ElSA-based PCs. The chip set incorporates the logic for a PC! interface running at a clock rate of 30 ns and an EISA interface running at a clock rate of 120 ns. The chip set contains a DMA con troller that supports direct- and scatter/gather mapped data transfers, with a sufficient amount of data buffering to isolate the PC! bus from the EISA bus. The chip set also includes PC! and EISA arbiters and various other support control logic that pro vide decode for peripheral devices such as the flash read-only memories (ROMs) containing the basic
1/0 system (BIOS) code, real-time clock, keyboard/ mouse control ler, floppy controller, two serial ports, one para l lel port, and hard d isk drive. In the
AJphaServer 2100 system, the PCI-to-EISA bridge chip set resides on the standard 1/0 module, which is discussed later in this paper.
Connecting the Alpha CPU to the PC/ and EISA Buses
In the next section, we discuss several i nteresting design chal lenges that we encountered as we attempted to connect re-oriented bus structures to a high-powered multiprocessing Alpha chassis.
Digital Techr�ical jour,al
Vol. G No. 3 Summer 1994 2 3
AlphaServer Multiprocessing Systems
Address and Data Mapping
When a CPU initiates a data transfer to a device on t h e PC! bus, t h e T2 bridge c h i p m ust first determ i ne the location (address) and amount of data (mask) i nformation for the requested transaction a nd then generate the appropriate PC! bus cycle. This issue is not straightforward beca use the PCJ and EISA buses both support data transfers clown to the byte granu larity, but the Alpha CPU and the system bus provide masking granularity only down to :12 bits of data.
To generate less than 32 -bit addresses and byte masked data transactions on the PC:! bus, the T2 bridge chip needed to i m plement a special decod i ng scheme that converts an Alpha CPU-to-I/O transaction, as it appears on the system bus, to a cor rectly s ized PC! transaction. Tables 1 and 2 give the low-order Alpha address bits and Alpha 32-bit mask fields ancl show how they are encoded to generate the appropriate PC! address and data m asks. By using this encod ing scheme, the Alpha CPL can per for m read and write transactions to a PC! device m apped in either PC! 1/0, PC! memory, or PC! configuration space with sparse-type transfers.
(Sparse- type transfer sizes have 8-, 16-, 24-, 32-, or
64-bit data granu larity)
Another mapping probkm exists when a PC! device wants to move a byte of data (or anything smal ler than 32 bytes of data) i nto the system bus
Table 1 CPU-to- PCI Read Size Encod ing
Transaction
Size
EV _Addr[6:5] EV _Addr[4:3] I nstructions PCI Byte
Enables
(L)
8 bits
1 6
24
bits bits
32 bits
64 bits
0 1
1 0
00
0 1
00
0 1
1 0
1 1
00
00
1 1
00
00
00
1 0
1 0
1 1
1 1
00
0 1
0 1
0 1
LDL
LDL
LDL
LDL
LDL
LDL
LDL
LDL
LDL
LDL
LDQ
1 1 1 0
1 1 01
1 0 1 1
01 1 1
1 1 0 0
1 001
001 1
1 000
0001
0000
0000
0000
0 1
1 0
00
0 1
00
0 1
1 0
1 1
0 0
00
00
PCI_AD[1 :0] Data Returned to Processor,
EV _Data[127:0]
OW_O: [D7: DO]
OW_O:[D1 5:D8] ow _O:[D23: D16]
OW_O: [D31 : D24]
OW_O:[D79:D64] ow _O: [D87:D72] ow
_O: [D95: D80]
OW _1 : [D23: DO]
OW_1 : [D95: D64]
OW_1 : [D95: D64]
OW_1 :[D127: D96]
Table 2 CPU-to-PCI Write Size Encoding
TransEV _Addr[6:5] EV _Addr[4:3] EV _Mask[7:0] (H) action
Size l nstrucPCI Byte PCI_AD[1 :0] Data Retu rned tions Enables to Processor,
(L)
EV Data[127:0]
8
bits
00
0 1
1 0
1 1
16
bits 00
0 1
1 0
24
bits 00
0 1
32 bits 00
64 bits
1 1
00
00
00
00
0 1
0 1
0 1
1 0
1 0
1 1
1 1
00000001
00000001
00000001
00000001
000001 00
00000100
00000100
00010000
00010000
01 000000
1 1 000000
LDL
LDL
LDL
LDL
LDL
LDL
LDL
LDL
LDL
LDL
LDQ
1 1 1 0
1 1 01
1 0 1 1
0 1 1 1
1 1 0 0
1 0 0 1
001 1
1 000
0001
0000
0000
0000
00
0 1
1 0
1 1
00
0 1
1 0
00
01
00
00
OW_O: [D7: DO]
OW_O:[D15:D8]
OW_O: (D23:D16]
OW_O: [D31 : D24]
OW_O:[D79: D64]
OW_O: (D87: D72]
OW_O: [D95: D80]
OW _1 : [D23: DO]
OW_1 : [D95: D64]
OW_1 : [D95: D64]
OW_1 :[D127: D96]
24 Vol. (i No .
.
)
Sitllllll�r
I'J'J·1 Digital Tecbnical journal
The AlphaServer 2100 1/0 Subsystem memory. Neither the system bus nor its memory supports byte granularity data transfers. Therefore, the T2 bridge chip must perform a read-modify write operation to move Jess than 32 bytes of data into the system bus memory. During the read-modify write operation, the T2 bridge chip f irst reads a full
32 bytes of data f rom memory at the address range specified by the PC! device 2 It then merges the old data (read data) with the new data (PC! write data) and writes the ful l 32 bytes back into memory.
/SA Fixed-address Mapping
We encountered a third interesting mapping prob lem when we decided to support certain !SA devices with f ixed l/0 addresses in the AlphaServer
2100 system. These ISA devices (e.g., ISA local area network [LAN] card or an ISA f rame buffer) have f ixed ( hardwired) memory-mapped 110 addresses in the 1-MB to 16-MB address range.
The ISA devices being discussed were designed for use in the f irst PCs, which contained Less than
1 MB of main memory. In these PCs, the 1!0 devices had f ixed access addresses above main memory in the 1-MB to 16-MB address range. Today's PCs have significantly more physical memory and use the
1-MB to 16-MB region as a part of main memory.
Unfortunately, these JSA devices were never redesigned to accommodate this change. There fore, to support these ISA options, the PC designers created 110 access gaps in main memory in the 1-MB to 16-MB address range. With this technology, an access by a CPU in that address range is automati cally forwarded to the !SA device.
To remain compatible with the ISA community, the T2 bridge chip also had ro al low for a gap in main memory at the 1-MB to 16 -MB address range so that these addresses could be forwarded to the appropriate !SA device.
BIOS Caching Compatibility
Today's Microsoft -compatible:: PCs provide another performance-enhancing mechanism. \Ve decided to implement this function inside the T2 bridge chip as well.
During system initialization, MS-DOS-based PCs read several BIOS ROMs from their l/0 space. Once the ROMs are read, their contents are placed in fixed locations in main memory in the 512-kilobyte
(KB)
to 1-MB address range. The software then has the abil ity to mark certain addresses within this range as read cacheable, write cacheable, read noncacheable, or write noncacheable. The basic intention is to mark f requently accessed sections of code as read cacheable but write noncacheable.
In this way, read accesses " hit" in main memory (or cache), and writes update the ROMs directly.
Interrupt Mechanism
No computer system wou ld be complete without providing a mechanism for an l/0 device to send interrupts to a CPU. The 1/0 interrupt scheme on the AlphaServer 2100 system combines familiar technology with custom support logic to provide a new mechanism.
Electrical and architectural restrictions prohib ited the interrupt control logic f rom being directly accessed by either the system bus or the PC! bus.
As a result, the interrupt contro.l logic is physical ly located on a utility bus called the XBUS. The XBUS is an 8-bit slave !SA bus placed nearby the PCI-to-EISA bridge chips.
The base technology of the 1/0 interrupt logic is a cascaded sequence of Intel 8259 interrupt con trollers. The 8259 chip was chosen because it is a standard, accepted, and well-known controller used by the PC industry today. The use of the 8259 interrupt controller translated to low design risk as well. Although the 8259 interrupt control ler is not new, its integration into a high-performance multi processing server, without incurring u ndue perfor mance degradation, required some novel thinking.
The integration of the 8259 interrupt controller into the A.lphaServer 2100 system presented two considerable problems. First, the designers had to satisfy the 8259 interface requirements in a way that would have a minimal impact on the perfor mance of the interrupt-servicing CPU. The 8259 requires two consecutive special-acknowledge cycles before it will present the interrupt vector.
To resolve this problem, we designed a set of handshaking lACK programmable array logic (PAL) devices. These PALs enhance the functions of the
8259 controllers as XBUS slaves. The interrupt servicing CPU performs only a single read to a desig
Juted address that is decoded to the XBUS. The lACK control PAls decode this read and then generate the special, double-acknowledge cycles required to access the vector. The PAL logic also deasserts
CHRDY, a ready signal to the !SA bus, so that the cycle has ample time to proceed without causing a con formance error for a standard !SA slave cycle. When the double acknowledge is complete and the vector is guaranteed to be driven on the bus, the PALs assert the CH RDY ready signal .
Digital Technical journal
Vol. 6 No. 3 Summer 1994 25
AlphaServer Multiprocessing Systems
The second problem involved the location of the interrupt controller. As mentioned earlier, because of electrical and architectural restrictions, the inter rupt controller was located on the XBUS near the
PCI-to-EISA bridge chips. With the interrupt con troller located on the XBUS, an interrupt-servicing
CPU is required to perform a vector read that spans two I/0 bus structures. For this reason and its potential effect on system performance, vector reads bad to be kept to a m inimum, which is not easy in a system that allows more than one CPU to service a pending interrupt request.
Since the AlphaServer 2100 system can have as many as four CPUs, a l l four CPUs can attempt to service the same pending interrupt request at the same time. Without special provisions, each CPU would perform a vector read of the interrupt con troller only to find that the interrupt has already been serviced by another CPU. Requiring each CPU to perform a vector read of the interrupt controller on the XBUS wastes system resources, especially when each vector read spans two bus structures. Of course, this problem could be resolved by assigning only one CPU to service pending interrupts, but this would negate the advantage of having multiple CPUs in a system. To solve this problem, the T2 bridge chip on the system bus implements special "passive release" logic that informs a CPU at the earliest possi ble time that the pending interrupt is being serviced by another CPU This allows the "released " CPU to resume other, more important tasks.
The term passive release typically refers to a vec tor code given to an interrupt-servicing CPU during a vector read operation. The passive-release code informs the CPU that no more interrupts are pend ing. The special passive-release logic allows the T2 bridge chip to return the passive-release code to a servicing CPU on behalf of the interrupt controller.
The T2 bridge chip performs this function to save time and bus bandwidth.
After the designers implemented all the features described above, they needed to address the prob lem of how to deal with all the slow, highly volatile,
" off-the-shelf' parts. To integrate these compo nents into the I/O subsystem, they invented the standard 1!0 module.
The Standard
1/0
Module
As part of the development effort of the 1/0 subsys tem, the engineering team faced the challenge of integrating several inexpensive, low-performance, off-the-shelf, PC-oriented l/0 functions (e.g., TOY clock, keyboard, mouse, speaker) into a high performance Alpha multiprocessing system, with out affecting the higher performing architectural resources. The mu.ltilevel I/0 bus structure served to alleviate the performance issues, but the develop ment of a PC-style 1/0 subsystem with off-the-shelf components involved inherent risk and challenge.
To reduce the risks inherent with using new and unfamiliar devices, such as the PCI-to-EISA bridge chip set, we chose to build an I/O mod u le (called the standard I/0 module) that plugs into the
AlphaServer 2100 system backplane and contains the PCI-to-EISA bridge, associated control logic, con trollers for mouse, keyboard, printer, and floppy drive as well as the integral Ethernet and SCSI con trollers. Without this plug-in module, fixing any problems with the PCI-to-EISA bridge chip set or any of the supporting logic would have required a backplane upgrade, which is a costly and time consuming effort.
The standard 1/0 module is relatively small, inex pensive both to manufacture and to modify, and easily accessible as a field replaceable unit (FRU). As shown in Figure 3, the standard 1/0 module con tains the fol lowing logic
•
PCI-to-Ethernet controller chip
•
PCHo-scsr control ler chip
•
PCI-to-EISA bridge chips
•
Real-time clock speaker control
• 8 - KB, nonvolatile, ElSA-configuration , random access memory
(RAM)
•
•
1-MB BIOS flash ROM
Keyboard and mouse control
•
Parallel port
•
FDC floppy controller
•
Two serial ports
•
•
I2C support: controller, expander, and ROM
I ntel 8259 interrupt controllers
•
Ethernet station address ROM
•
Reset and sysevent logic
•
Fan speed monitor
•
Remote fault management connector
•
External PC! subarbiter
•
3.3-volt and 5.0-volt generation
26 Vol. 6 o.
3 Summer 1994 Digital Technical journal
The AlphaServer 2100 1/0 Subsystem
PCI BUS
ETHE R N ET
ROM
32 BYTES 32
8
SCSI
INTERRUPT
CONTROLLER
,.- - - - -- - - - - - , l
8259A-2 I
I 00000
_ _ _ _ _ _ _ _ _
I
J lACK
CONTROLLER
FAN
ROTATION
MONITOR
SYSTEM
RESET
GEN ERATION
KEYBOARD
MOUSE
- 5 -V
GENERATION
3.3-V
G E NERATION
___.
___.
8242
AND MOUSE
CONTROLLER
1 2C
PARALLEL
PORT
PARALLEL
0
FLOPPY
( 1 }
SER IAL PORT
SE RIAL PORT
0
0
0
0
Figure 3 The Standard
l/0
Module
For the most part, all these functions were gener ated by using integrated, off-the-shelf components at commodity pricing. Solutions known to work on other products were used as often as possible.
The flash memory resides on the EISA memory bus and is control led by the PCI-to-EISA bridge chip.
A simple multiplexing scheme with minima! hard ware enabled the server to address more locations than the bridge chip allowed, as much as a ful l
1
MB of BIOS ROM. The National PC87312, which provides the serial and parallel port control logic, and the floppy disk control ler reside directly on the ISA bus.
The rest of the devices are located on the XBUS (an
8-bit buffered slave ISA bus), with control managed by the PC!-to-ElSA bridge chips.
In addition, the common PC functions are located at typical PC addresses to ease their integra tion and access by software. As expected, hardware changes were required to the standard l/0 module during i ts hardware development cycle. However, the standard 1/0 module, which takes only minutes to replace, provided an easy and efficient method of integrating hardware changes into the AlphaServer
2100 system . We expect the usefulness of the stan dard l/0 module to continue and hope that it will provide an easy and i nexpensive repair process.
D igital Technical journal
Vol. 6 No. 3 Swmner 1994
27
AlphaServer Multiprocessing Systems
Summary
The 1/0 subsystem on the AlphaServer 2 100 system contains a two- level hierarchical bus structure con sisting of a high-performance PC! bus connected to a secondary ElSA bus. The PCl bus is connected to the AlphaServer 2100 system's multiprocessing sys tem bus through the T2 bridge chip. The secondary
1/0 bus is connected to the PC! bus through a stan dard bridge chip set. The 1/0 subsystem demon strated sufficient flexibility to become the 1/0 interface for the small pedestal AJphaServer 2000 and the rackmountable version of the AJphaServer
2100 products.
Acknowledgments
The AlphaServer 2100 l/0 wou ld not be what it is today, without the dedicated, focused efforts of sev eral people. Although not complete, the fol lowing l ist gives credit to those who were the most deeply invo lved . Thanks to Fidelma Hayes for leading the
Sable effort; to Vicky Triolo for the Sable mother board and her support of the T2 bridge chip effort; to Rachael Berman for her unflagging support of the standard 110 modu le; to Lee Ridlon for his much needed early conceptual contributions; to Stephen
Shirron for driving Sable software l/0 issues; to jolm
Bridge for cleaning up the second-pass T2; and to
Tom Hunt and Pau l Rot ker for their contributions to the first-pass T2.
References
1 . F Hayes, ··oesign of the AlphaServer Multiproces sor Server Systems;· Digital Technical journal, vol . 6, no. 3 (Sum mer 1994, this issue): 8-19.
2. PCI Local Bus Speetfication, Revision
2. 0
( Hills boro, OR: P
C
! Special Interest Group, Order No.
281446-001 . April 1 993).
3.
82420182430
PC/set /SA and EISA Bridges (Santa
Clara,
CA:
Intel Corporation, 1993).
4. E. Solari, !SA and EISA, Themy and Operation
(San Diego,
CA:
A.nnabooks, 1992).
28 Vol. 6 No . .! S/1111111er 1994 Digital Technical journal
jeffrey M. Denham
Paula Long james A. Woodward
DEC OSF/1 Version
3.0
Symmetric
Multiprocessing Implementation
The primary goal for an operating system in a symmetric multiprocessing
(SMP)
implementation is to convert the additional computing pawer provided to the sys tem, as processors are added, into improved system pe1jonnance without compro mising system quali�y. The DEC
OSF/1
version
]. 0
operating system uses a number of techniques to achieve this goal. The techniques include algorithmic enhance ments to improve parallelism within the kernel and additional lock-based synchro nization to protect global system state. �ynchronization primitives include spin locks and blocking locks. An optional locking hierarchy was imposed to detect latent symmetric multiprocessor synchronization issues. Enhancements to the ker nel scheduler improve cache usage by enabling soft affinity of threads to the proces sor on which the thread last ran; a load-balancing algorithm keeps the number of runnable threads spread evenly across the available processors. A highly scalable and stable
SMP
implementation resulted from the project.
The DEC OSf/ 1 operating system is a Digital product based in part on the Open Software Fou ndation 's
OSF/1 operating system . 1 One major goal of the DEC
OSF/1 version 3.0 project was to provide a leader ship multiprocessing implementation of the C '-JIX operating system for AJpha server systems, such as the Digital AlphaServer 2100 product. This paper describes the goals and development of this operat ing system feature for the version 3.0 release.
The DEC OSF/1 Version
3.0
Multiprocessing Project
Mul tiprocessing platforms like the AJphaServer
2100 product provide a cost- effective means of increasing the computing power of a server. Addi tional computing capacity can be obtained at a potentially significant cost advantage by simply add ing CPU modules to the system rather than by adding a new system to a more loosely coupled network-server arrangement. An effect ive execu tion of this server-scaling strategy requires signifi cant cooperation between the hardware and software components of the system. The hardware must provide symmetrical (i .e., equal) access to sys tem resources, such as memory and J/0, for all pro cessors; the operating system software must provide for enough paral lel ism in its major subsys tems to allow appl ications to take advantage of the additional crus in the system . That is, the operating system cost of mul.tiprocessing must be kept low enough to enable most of an additional CPU's com puting power to be used by applications rather than by the operating system's efforts to synchro nize simu ltaneous access to shared memory by mul tiple processors.
Regard ing hardware, the AlphaServer 2100 prod uct and the other Alpha mult iprocessing platforms provide the shared memory and symmetric access to the system and l/0 buses desired by the operat ing system designers 2 The design allows all CPUs to share a single copy of the operating system in memory. The hardware also has a load- locked/ store- conditional instruction sequence, which pro vides both a mechanism fo r atomic updates to shared memory by a single processor and an inter processor interrupt mechanism.
Given these hardware features, operating system software developers have a great deal of freedom in developing a multiprocessing strategy. The approach used in DEC OSF/1 version 3.0 is cal led symmetric mu ltiprocessing (SMP), in which all pro cessors can participate fu lly in the execution of operating system code. This symmetric design con trasts with asymmetric multiprocessing
(ASMP),
in which all operating system code must be executed on a single designated "master" processor. Such an
Digital Techuicaljourual
Vol. 6 No . .) Summer 1994
29
DEC OSF/1 Symmetric Multiprocessing a valuable product feature and was a preview of t he effort that would be required to adapt the OSF/ 1 code for the DEC 2000, 4000, and 7000 multipro cessing platforms. Supporting separate preemptive kern els for three versions prior to DEC OSf/ l version 3.0, combined with the developers' experi ence on other multiprocessing systems ( including
ULTRIX version 4 and an advanced development project using M IPS multiprocessing platforms), uncovered the following challenges and problems that the team had to overcome to produce a com petitive multiprocessing product:
•
Supporting two complete sets of kernel binary objects-base and rea l-time-was burdensome for the operating system engineers ancl awk ward for thi rd-p arty developers. Therefore, the
DEC OSF/1 multiprocessing product team had to strive to ship a single, unified set of k ernel bina ri es. This set should encompass the full range of real-time features, including preemption and
POS IX fi xed-priority scheduling. For that to be practica l , the resulting multiprocessing kernel would have to perform as well on a uniproces sor as the non-SNIP ke rnel.
• Diagnosing locki ng problems in the preemptive kernel was expensive in developer t i me . The process required pa instaking inspection of the simple-locking source code, which is often disguised in subsystem-specific macros. Lock ing or unlock ing a spin lock multiple times or not unlocking it at all (usually in code loops) would dis able preemption well beyond the end of a critical section or enable it before the encl.
A coherent lock ing architecture with automated debugging facilities was needed to ship a reliable product on rime. The lock-debugging facility present in the original OSF/1 code \Vas probably inadequate for the task.
•
Experiments with the real-time kernel revealed unacceptable preemption latencies, especially in funneled code paths. This deficiency indi cated that, when moved to a multiprocess i ng platform, the existing kernel would fai l ro use additional processors effectively. nlat is, the kernel would not exhibit adequate parallel ism to scale effectively. Clea rly, major work was requ ired to significantly increase parallelism in the kern el. This task ·would likely involve remov ing most uses of funneling, eliminating some spin locks, and adding other spin l.ocks ro create a finer granularity of locklllg.
Adapting the Base Operating System for Symmetric Multiprocessing
Mak ing the leap f rom a preemptive uniprocessor kernel to an effective SMP implementation , built from a single set of kernel binaries, required con tributions from the OSF/1 version 1 .2 and the DEC
OSF/ 1 version 3.0 projects. Fundamental changes were introduced into the system to support SMP.
The basic approach planned by the SMP project team was f irst to bootstrap the DEC: OSF/1 version
1 . 3 kernel on the existing Alpha multiprocessing platforms. This task was accomplished by funneling all major subsystems to a single processor while sta bilizing the underpinnings of the multiprocessing system (i.e., the scheduler. the v irtual memory sub system, the virtual file system, and the hardware support) in the new environment. This approach allowed the ream to make progress in unde rstand ing the scope of the effort wh ile analyzing the multiprocessing requ i rements of each kernel sub system. The in-depth ana lysis was necessary to identify those su bsystems in the kernel that required modifications to run safely and effi ciently under SMP. As each su bsystem was confirmed to exhibit parallelism or was made paralleL it was unfunnded and thus freed to mn on any processor.
This process was iterative. If incorrectly paral lelized, a subsystem will reveal itself by ( l) leaving data incorrectly unprotected and thus open for cor ruption and ( 2) developing a deadlock, i.e .. a s itua tion in which each of two threads holds a spin lock requ ired by the other th read and thus neither thread can rake the lock ancl p roceed.
The efforts at parallel i zing the kernel fell into two classes of modification: lock-based synchro nization to ensure multiprocessing correctness and algorithmic changes to increase the level of paral lelism ach ieved.
Lock-based Synchronization
The code base on which the DEC OSF/ l product is built, i.e., the Open Software Foundation's OSf'/1 software, provides a strong foundation for SMP. The
OSF further strengthened thi s foundation in OSF! l ve rsions 1 . 1 and 1 .2, when it corrected multiple
Si'vll' problems in t he code base and parallel ized
(and thus unfunneled) additional subsystems. As the mu ltiprocessing bootstrap effort continued , the team analyzed and incorporated the OSF/
I ver sion 1 . 2 S1Y!P improvements into DEC O S F!l version
3.0. As strong as this starting point was, however, some st ructures in the system did not receive the
:) 2
\l,,f. (, No. 3
S11111111er
1994
Digital Technical jour11al
DEC OSF/ 1 Version .). 0 Syrnmetric Multiprocessing Implementation appropriate level of synchronization. The team cor rected these problems as they were uncovered through testing and code inspection.
The DEC OSF/ 1 operating system uses a combina tion of simple locks, complex locks, elevated SPL, and funneling to guarantee synchronized access to system resources and data structures. Sim ple locks,
SPL. and fu nnel ing were described briefly in the earl ier d iscussion of preemption. Complex locks. l i ke elevated SPL, are used in both uniprocessor and m u l tiprocessor enviro nments. These locks are usu ally sleep locks- threads can block while they wait for the lock-which offer additio nal features, i nclud i ng m u l t iple-reader/single-writer access and recursive acquisition.
An example of the use of each synchronization technique fol lows:
•
A simple lock is used to protect the kernel 's cal l out (timer) queue. In an S,\1 P environment, m u l tiple threads can update the ca l lout queue at the same time. as each of them adds a t i mer entry to the queue. Each thread must obtain the call out lock before adding an entry and release the lock when done. The cal lout simple lock is also a good example of SPL synchron ization under multiprocessing because the cal lout queue is scanned by t he system clock l S R . Therefore, before locking the cal lout Jock, a thread m u st raise the SPL to the clock's lPL. Otherwise, the thread holding the cal lout lock at an SPL of zero can be interrupted by the clock I S R , which w i l l in tu rn attempt to take the callout lock. The resu lt is a permanent dead lock.
•
A complex lock protects the file system direc tory structure. A blocking lock is requ ired because the d i rectory lock holder m ust perform
I/O to update the directory, which itself can block. Whenever block ing can occur w h ile a lock is hel d , a complex lock is requi red.
•
Fu nnel ing is used to synchronize access to the
I S O 9660 CD-ROM file syste m . - The decision to fu n nel this file system was .largely due to l i mi ta tions in the DEC OSI'/ 1 version 3.0 schedu le; however, the file system is a good cho ice for fun nel ing because of its general l y slow operation and I ight usage.
To ensure adequate performance and seal ing as processors are added to t he system , an SJ\II P imple mentation must provide fo r as much paral lel ism through the kernel as possible. The granularity of locks placed in the system has a major i mpact on the amount of paral lelism obtained .
During multiprocessing developmen t, locking strategies were designed to
•
•
Reduce the total number of locks per su bsystem
Reduce the number of locks ta ken per subsys tem operation
•
Improve the level of paral lelism throughout the kernel
At t i mes, t hese goals clashed: enhancing paral lelism usually involves add ing a lock to some struc ture or code path. This outcome confl icts with the goal of reducing lock counts. Consequent ly, in prac tice. the process of successfu l l y para l lel izing a sub system involves striking a balance between lock red uction and the resulting increase i n lock granu larit y. Often, benchmarking different approaches is required to fine- tune this balance.
Several general trends were uncovered during lock analysis and tuning. In some cases locks were removed because they were not needed; they were the products of overzealous synchro nization.
For example, a structure that is pr ivate to a thread may require no lock ing at a l l . Moreover, a data ele ment t hat is read atomica l l y needs no locking. An example of lock removal is the getti meofday( ) sys tem cal l, which is used frequent ly by DBMS servers.
The system cal l simply reads the system time, a 64bit quantity, and copies it to a buffer provided by the cal ler. The original OSF/ 1 system cal l . running on a
32-bit architecture, had to take a simple lock before reading the time to guarantee a consistent value. On the Alpha archi tecture, the system call can read the ent ire 64 -bit time value atomical ly. Removing the lock resulted in a 40 percent speed up.
In other cases, analyzing how structures are used revealed that no lock ing was needed. For example, an 1/0 control block cal led the buf structure was being locked in several device drivers while the block was in a state that al lowed only the device driver to access i t . Removing t hese unnecessary locks saved one complex and one simple locking sequence per l/0 operation in these drivers.
Another effective optim ization involved post poning lock i ng until a thread determined that it had actual work to do. This technique was used success fu l l y in a routine frequently cal led in a transaction processing benchmark. The routi ne, which was locking structures in anticipation of fol lowing a rarely used code path, was mod ified ro lock only
Digital Technical jourual
Vol. (J No
> Summer 19')4
33
DEC OSF/1 Synunetric Multiprocessing when the u ncommon code path was needed . This optimization significantly reduced lock overhead.
To improve paral lel ism across the system, the
DEC OSf/1 SMP development team modified the lock strategies in numerous other cases.
Algorithm Changes
In some instances. the effective migration of a sub system to the multi processing environment required significant reworking of its fundamental algorithms. This section presents three examples of this work. The first example involves the rework of the process ma nagement subsystem; the second example is a new technique for a thread to refer to its own state; and the third example deals with enhancements i n translation buffer coherency or
"shootdown:·
J'rfanaging Processes and Process State Early ver sions of the DEC OSF/ 1 software maintained a set of systemwide process lists, most notably proc (static proc structure array), a l lproc (active process l ist), and zomproc (zombie process list). These l ists tend to be fairly long and are normally traversed sequen tial ly. Operations involving access to these lists include process-creation time (fork( ) ), signal post ing, and process termination. The original OSF/1 code protected these process l ists and the i ndivid ual proc structures themselves by means of funnel ing. This meant that virtually every system call that involved process state, such as exit( ). wait( ), ptrace( )
, and sigaction( ), was also forced into a single funnel. Experience with real- time preemp tion indicated that this approach would exact excessive m ultiprocessing costs. Although it is pos sible to protect these l ists with locks, the develop ment team decided that this basic portion of the kernel must be optimized for maximum multi processing performance. The OSF also recognized the need for optimization; they addressed the prob lem in OSF/ 1 version 1 . 2 by adopting a redesign of the process management developed for their
Mu ltimax systems by Encore Compu ter Corpora tion. The DEC OSF/ I team adopted a nd enh:mced this design for hand ling process l ists, p rocess man agement system calls, and signal processing.
The redesign replaces the statical ly sized array of proc structures with an array of smaller process identification (PID) entry structures. Each P[J) entry structure p otentially points ro a dynamically al lo cated proc structure Under this new scheme, find ing the proc structure associated with a user P I D has been reduced to hash ing the P I D value to an index into the PID entry array. The process state associated with that PID (active, zombie, or nonexis tent) is maintained in the PID entry structure. This allows process structures to be al located dynami cally, as needed, rather than statically at boot time. as before. Simple locks are also added to the process structure to allow mu ltiple threads in the process to perform process management system cal ls and sig nal hand l i ng concurrently. These changes allowed process management fu nnel ing to be removed entirely, which significantly improved the degree of paral lelism in the process management subsystem.
Accessing Current Thread State One critical design choice in i mplementing SMP on the DEC OSF/ 1 sys tem concerned how to access the state of the cur ren t ly running thread. This state i ncludes the cmrent thread 's process, task. and virtual memory structures, and the so-called uarea, which contains the pageable UNfX state. Access to this state, which threads require frequently as they run i n kernel context, must have low overhead . Further, because the DEC OSF/1 operating system supports kernel mode preemption, the method for accessing the current thread's state must work even if a context switch to another CPU occurs during the access operation.
The original OSF/ 1 code used arrays i ndexed by the CPU number to look up the state of a runni ng thread . One of these arrays was the U_A D DRESS array, which was used to access the cu rrently active uarea. The U_A D DRESS array was loaded at context switch time and accessed while the thread exe cuted. Before the advent of multiprocessi ng. the
CPU number was a compile-time constant, so that thread-state lookup involved simply reading a global variable to form the pointer to the data.
Ad di ng mu ltiprocessing support meant changing the CPU number from a constant ro the resu lt of the WHAMI ("Who am P ") PALcode call to get the current CPU number. (PAL.code is the operating system-specific privi leged arch itecture library that provides control over interrupts, exceptions. context switching, etc H)
Using such global arrays for accessing the current thread's state presented three shortcomings:
1 . The \\fJ-IAMI PALcode cal l added a minimum over head of 21 machine cycles on rhe AlphaServer
2100 server, nor includ ing further overhead c.lue to cache m isses or instruction stream stalls. The multiprocessing team felt that this was too large a performance price to pay.
Vol. 6 No. 3 Summer 1<)')4 Digital Teclmical]ourrwl
DEC OSF/
1
Version 3. 0 Symmetric il-'lultiprocessing Implementation
2. Allowing multiple crus to write sequ ential pointers caused cache thrashing and extra over head during context switching.
3. Indexing by CPU nu mber was not a safe practice when kernel-mode preemption is enabled.
A thread could switch processors in the middle of an array access, and the wrong pointer wou ld be fetched . Prov iding additional locking to pre vent this had unacceptable performance impli cations because the operation is so common.
These problems convi nced the team that a new algorithm was required for accessing the current thread's state.
The solution selected was modeled on the way the OpenVMS VAX system uses the processor inter rupt stack pointer to derive the pointer to per-CPU state .9 In the OSF/ 1 system, each thread has its own kernel stack . By aligning this stack on a power-of two boundary, a si mple masking of the stack pointer yields a pointer to the per-thread data, such as the process control block (PCB) and u thread structure. Any data item in the per-thread area can be accessed with the fol l owing code sequence: l d a r 1 6 , M A S K b i c s p , r 1 6 , r O l d q r x , O F F S E T ( r Q )
# G e t m a s k v a l u e
# M a s k s t a c k p o i n t e r t o p o i n t t o s t a c k b a s e
# A d d o f f s e t t o b a s e a n d f e t c h i t e m
Accessing thread state using t h e kernel stack pointer solves a l l three problems with CPU-number based indexing. First, this techn ique has very low overhead; accessing the cu rrent thread's data involves only a simple masking operation and a read operat ion. Second . usin g the kernel stack pointer incurs no extra overhead during context switching because the pointer has to be loaded for other uses.
Third, because thread stack areas are pages, no cache confl icts exist between threads runni ng on different processors. Final ly, the data access can be preempted at any point, and the correct poi nter is still fetched . No processor-dependent state is used to access the current thread state.
lnterprocessor Translation Lookaside Buffer
Shootdown
Alpha processors employ translation lookaside bu ffers (TLFls) to speed up the translation of physical-to-virtual mappings. The TLB caches page table entries (PTEs) that contain vi rtual-to physical address mappings and access control infor mation. Unl ike data cache coherency, which the hardware maintains, TLFl cache coherency is a task of the software. The DEC OSF/ 1 system uses an enhan ced ve rsion of the TLB shootdown algorithm developed for the Mach kernel to maintain TLB coherency.
1°
First, a mod ification to the original shootdown algorithm was needed to im plement the Alpha archi tectu re's add ress space numbers
(ASNs) . Second, a synchronization feature of the original algorithm was removed entirely to enhance shootdown performance. This feature provided synchronization for archi tectures in which the hardware can modify PTEs, such as the VAX plat form; the added protection is un necessary for the Alpha architecture.
The final shootdown algorithm is as fo l l ows. The physical map (PMAP) is the software structure that holds the virtu al-to-physical mapping information.
Each task within the system has a PMAP; operating system mappings have a special kernel P;\1AP. Each
PMAP contains a l ist of processors cu rrently using the associated address space. To init iate a virtual- to physical translation change, a processor (the initia tor) first locks the PMAP to prevent any other threads from modifying it. Next, the in itiator updates the PTE mapping in memory and flushes the local TLB. The processor then sends an i nterprocessor in terrupt to all other processors (the responde rs) that are cu rrently active in the same add ress space. Each responder needs to ack nowledge the ini tiator and inval idate its own mapping. Once all responders are accounted for, the i nitiator is free to continue with the knowledge that all TLBs are coherent on the system . The initiator marks nonactive proces sors· ASNs i nactive , spins wh ile it waits for other processors to check in, and then unl ocks the PMAP.
Figure 1 shows this final TLB shootdown algori thm as it progresses from the initiating processor to the poten tial respond ing processors.
Developing the Lock Package
Key to meeting the performance and rel iabi l it y goa ls for the multi processing portion o f the DEC
OSF/ 1 version 3.0 release was the development of a lock pac kage with the fol lowing ch:ll'acteristi cs:
•
Low execution and memory overhead
•
Flexible support for both uniprocessor ancl multiprocessor platforms, with and without real-time preemption
•
Autom ated debuggi ng facil ities to detect in cor rect locking practices at run time
Digital Technical ]our11al \IIJI. 6 No. 3 Summer 1994
35
DEC OSF/1 Synunetric Mult iprocessing
I n itiator:
Lock the PMAP.
Update the translation map (PTE).
I nval idate the processor TLB ent ry.
Send an interprocessor interrupt to all processors that are using the PMAP.
Responders:
Acknowledge the shootdown.
I nvalidate the processor TLB entry.
Ret urn from the i nterrupt.
Mark the nonactive processors' ASNs inactive.
Spin while it waits for other processors to check in.
Un lock the PMAP.
Figure I hanslation Lookaside Bujfer Shootdown Algorithm
•
Statistical facil iries ro track the number of locks used , how many times a lock is taken, and how long threads wa i t to obtain locks
Of course, the overa l l role of the l ock package is ro provide a set of synchron iz:Jtion primitives. that is, the simple and complex Jocks described in earlier sections. To support kernel-mode rhrea<.J preemption, DEC OSF/ 1 version 1 .0 had extended the lock package originally del ivered with OSF/ I version 1 .0. Early in the DEC OSF/1 version 3.0 rroj ect, the development team extended the package again to optimize its performance and to add the desired debugging and statistical features.
As previously noted, a major goa l for D EC OSF/1 version 3 0 was to ship a single version of its kernel objects, instead of the base and rea l-time sets of previous releases. Therefore, simple locks would have to be compi led into the kernel, even for ker nels that wou ld run only on uniprocessor systems.
Achieving this goal required minimizing the size of the lock structure; it would be unacceptable to h;lVe hundreds of kilobytes (KB) of memory dedi cated ro lock structures i n systems that did not use such structures. Further. the simple lock and u nlock invocations required by the multip rocess ing code wou ld have to be present for a l l platforms, which wou ld raise serious performance issues for uniprocessor systems. In fact, i n the original OSf/ I lock package, the CPU overhead cost of compil ing in the lock code was between I and 20 percent.
Compute- intensive benchmarks showe(l the cost to be less than ) percent, but the cost for multiuser benchmarks was greater than 10 percent, which represents an u nacceptable performance degrada tion. To meet the goal of a single set of binaries, the development team had to e n hance the lock package to be configurable at boot time. T hat is, the package needed to be able to tailor itsel f to fi t the configura tion a nd real- rime requirements of the platform on which it wou ld run.
The lock package supplied by the OSF/ 1 system was h1rther deficient in that it did nor support error checking when locks were asserted . This deficiency left developers open to the most com mon ronnen tor of concurrent programmers. i .e. , dead locks.
Without error checking, potential system hangs caused by Jocks being asserted in the wrong order could go undetected for years and be difficul t to debug. A formal locking order or hierarchy for all
Jocks in rhe system had to be establ ished. and the lock package needed the abil ity to check the hierar chy on each lock taken.
These needs were mer by introd ucing the notion of lock mode to the lock package. Developers defined the ti>Howing five modes and associated roles:
•
Mode 0: No lock operations; for production u niprocessor systems
•
Mode I : Lock cou nting only to manage kernel preemption; for prod uction real-time u nipro cessor systems
•
Mocle 2: Locking withm1t kernel preemption; for product ion multiprocessing systems
•
Mode :') : Lock ing with kernel preemption; for production real-time mul tiprocessing systems
•
Mode 4: Full Jock debugging with or without preemption; for any development system
Vol. () No .. ! Still/Iller
1994 Digital Technical journal
DEC 05F/l Version 3. 0 Symmetric kfultiprocessing Implementation
The default uniprocessor lock mode is 0; the mul tiprocessing default is lock mode 2. Both selections favor non-real-time producti o n systems. The sys tem's lock mode, however, can be selected at boot time by a nu mber of mechanisms. Lock modes are implemented through a dynamic lock configura tion scheme that essential ly iiJstal ls the appropriate set of lock primitives for the selected lock mode.
I nstallation is rea lized by patching the compiled-in function calls, such as simple_lock( ), to dispatch to the corresponding lock primitive for the selected lock mode. This technique avoids the overhead of dispatching indirectly to different sets of lock primitives for each call, based on the lock mode.
The compi led-in lock function calls to the lock package are a l l entry points that branch to a call patching routine called simple_lock_patch( ). This routine changes the call i ng machine i nstruction to be patched out (for Jock mode 0) or to branch to the corresponding primitive in the appropriate set of actual primitives, and then branches there (for lock modes 1 through 4). Thus, the overhead for dynamically switching between the versions of sim ple lock primitives occurs only once for each code path. In the case of lock mode 0, calls to simple
Jock primitives are "back patched " out. Under this model, uniprocessor systems pay a one-time cost to invoke the simple lock primitives, after which the expense of executing a lock primitive is reduced to executing a few no-op i nstructions where the code for the lock call once resided.
To address memory consumption issues and to provide better system debug capabi.lities, the devel opers reorgan ized the lock data structures around the concept of the lockinfo structure . This struc ture is an encapsulation of the Jock's ordering (hier archical relationship) with surrounding locks and its minimum SPL requirement. Lock debugging information and the lock statistics were decoupled from the lock structures themselves. To facil itate the expression of a lock hierarchy, the developers introduced the concept of classes and i nstances.
A lock class is a grouping of Jocks of the same type.
For example, the process structure lock constitutes a lock class. A lock instance is a particular lock of a given class. For example, one process structure simple lock is a n instance of the class process struc ture lock. Error checking and statistics-gathering are performed on a lock-class basis and only i e1 lock mode 4.
Decoupling the lock debugging information from the Jock itsel f significantly reduced the sizes of the s imple and complex lock structures to 8 and
32 bytes, respectively. Embedded in both structures is a 16-bit i ndex i nto the Jockinfo structure table for that particular lock class. The lock info structure is dynamically created at system startup in lock mode 4. All classes in the system are assigned a rela tive position in a single u nified lock hierarchy.
A lock class's position i n the lockinfo table is also its position in the lock hierarchy; that is, locks must be taken in the order in which they appear in the table. Lock statist ics are also maintained on a per class basis with separate entries for each processor.
Keeping lock statistics per processor and separat i ng this information by cache blocks eliminates the need to synchronize lock-primitive access to the statistics. Thi s design , which is illustrated in
Figure 2, prevents negative cache effects that could result from sharing this data.
Once this powerful lock package was opera tional, developers analyzed the lock design of their kernel subsystems and attempted to place the locks used into classes in the overal l system lock h ierar chy. The position of a class depends on the order i n which i t s locks are taken and released in relation t o other locks i n the same code path a n d i n the sys tem. At times, this static lock analysis revealed prob lems in existing lock protocols, in which Jocks were taken in varying orders at different points i n the code. Clearly, the lock protocol needed t o be reworked to produce a consistent order that could be codified in the hierarchy. Thus, the exercise of producing an overall lock hierarchy resu lted i n
LOCK INSTANCES
Figure
LOCK CLASS LOCK STATISTICS
L......,
CPU 1
2 Lock Structure
Digital Technical journal
Vol. 6 No. 3 Summer 1994 37
DEC OSF/1 Symmetric Multiprocessing a significant cleanup of the original multiprocess ing code base. To add a new lock to the system, a developer would have to determine the hierarchi cal position for the new lock class and the mi ni m u m SPL at which the lock must be taken.
Running the system in lock mode 4 and exercis ing code paths of interest provided developers with i m mediate feedback on their lock protocols. Using the hierarchy and SPL information stored in the run time lockinfo table, the Jock primitives aggressively check for a variety of locking errors, which include the following:
•
Locking a lock out of h ierarchical order
•
•
Locking a simple lock at an SPL below the required minimum
Locking a s imple lock already held by the cal ler
•
Unlocking an u nlocked simple lock
•
•
Unlocking a simple lock owned by another CPU
Locking a complex lock with a simple lock held
•
Locking a complex lock at interrupt level
•
Sleeping with a simple lock held
•
Locking or unlocking an uninitialized lock
Encountering any of these types of violation results in a lock fault, i.e. , a system bug check that records the information requ ired by the developer to quickly track down the lock error.
The reduction in lock sizes and the major enhancement of the lock package enabled the team to rea lize its goal of a single set of kernel binaries.
Benchmarks that compare a pure uniprocessor kernel and a kernel in lock mode 0 that are both running on the same hardware show a less than
3 percent difference in performance, a cost consid ered by the team to be well worth the many advan tages to returning to a u nified kernel. Moreover, the debugging capabilities of the Jock package with its hierarchical scheme streamlined the process of lock analysis and provided precise and immediate feedback as developers adapted their subsystems to the multiprocessing environment.
Adapting the Scheduler for
Multiprocessing
The normal scheduling behavior, i.e., policy, of the OSF/ 1 system is traditional
UNIX
ti me-sharing.
The system time-slices processes based on a time quantum and adjusts process priorities to favor interactive jobs over compute-intensive jobs. To support the POSIX real-time standard, the DEC OSF/ 1 system incorporates two additional fixed-priority schedu ling poli cies: first in, first out (POLICY _FIFO) and round robin (POLICY _RR).
A time-share thread's priority degrades with CPU usage; the more recent the thread's CPU usage, the more its priority degrades. (Note that OSF/ 1 schedu li ng entities are threads rather than pro cesses.) In contrast, a fixed-priority thread never suffers priority degradation. Instead, a POLICY_RR thread runs until it blocks voluntarily, is preempted by a higher-priority thread, or exhausts a quantum
(and even then, the round robin schedu ling applies only to threads of equal priority). A POLICY _FIFO thread has no scheduling quantum; it runs until it blocks or is preempted. These specialized policies are used by real-time appl ications and by threads created and managed by the kernel. Examples of these kernel threads include the swapper and paging threads, device driver threads, and network protocol handlers. A feature called thread binding, or hard affinity, was added to DEC OSF/1 version 3.0
Binding allows a user or the kernel to force a thread to run only on a specified processor. Binding sup ports the funneling feature used by unparallelized code ancl the bind_to_cpu( ) system cal l .
The goal o f a multiprocessing operating system in scheduling threads is to run the top threads on
N
priority
N
processors at any given time. A simple way to accomp.l ish this would be to schedule threads that are not bound to a CPU in a single, global run queue and sched u le bound threads in a run queue local to its bound processor. When a proces sor reschedules, it would select the h ighest-priority thread available in the local or the global run queue.
Schedu l ing threads out of a global run queue is highly effective at keeping the
N
highest -priority threads running; however, two problems arise with this approach:
1. A single run queue leads to contention between processors that are attempting to reschedu le, as they race to lock the run queue and remove the highest-priority thread .
2. Sched ul ing with a global run queue does not take advantage of the cache state that a thread builds on the CPU where i t last ran. A thread that migrates to a different processor must reload its state into the new processor's cache. This can substantially degrade performance.
To help preserve cache state and reduce wastefu l global run queue contention, the developers
38
Vol. 6 No. 3 Summer 1994
Digital Technical jour11al
DEC OSF/ 1 Version
3.
0 Symmetric Multiprocessing Implementation enhanced the multiprocessing scheduler by adding two new schedul ing models: a soft-affinity sched uling model for time-share threads and a last processor-preference model for fixed-priority threads. Under these models, each processor sched ules time- share and bound threads in its local run queue, and it sched u les unbound fixed-priority threads out of a global run queue.
Fixed-priority threads scheduled from a global run queue are able to run as soon as possible. This behavior is required for h igh-priority tasks like kernel threads and real-time applications. The last processor-preference model gives a fixed-priority tlu·ead a preference for running on the processor where it last ran; if that processor is busy, the thread runs on the next available processor. Each time share thread is softly bound to the processor on which it last ran; that is, the thread shows an affinity for that processor. Unlike funneling or user bind ing, which support hard (mandatory) affinity, soft affinity al lows a thread to run elsewhere if it is advantageous, i.e., if such activity balances the load .
Otherwise, the softly bound thread tries to return to the processor where it last ran and where its recent cache state may still reside.
Under load , however, a soft affinity model used alone can degenerate to a state where one proces sor builds up a large queue of threads, leaving the other processors with little to do and thus dim in ishing the performance of the multiprocessing sys tem. To mitigate these side effects of soft affinity, developers paired the soft affinity feature with the ability to load-balance the runnable threads in the system. To keep the load of time-share jobs spread evenly across processors, the schedu ler must peri odically load-balance the system. In addition to d is tributing threads evenly across the local run queues in the system, this load-balancing activity must
•
Cost no more in processing time than it saves
•
•
Prevent excessive thread movement among processors
Recogn ize and effectively accommodate changes in the job m ix
To implement load balancing, each processor maintains a time-share load average, i.e., the aver age local run queue depth over the last five sec onds. Each processor uptlates its own load average on each system clock tick. Processors also keep track of the time they spend hand l i ng interrupts and running fixed-priority threads, which are not accounted for in the loca l run queue depth. Taking a processor's total potential execution time for a schedu ling period and subtracting from this time the interrupt-processing and fixed-pri ority run times yields the total time available on a processor
(processor ticks available) to run time-share threads.
In the worse case, a processor could be completely consumed by fixed-priority threads and/or inter rupt processing and have no time to run time-share threads. In this extreme case, the scheduler should give no time-share load to that processor.
Adding the time-share load averages of all proces sors determines the aggregate time-share load for the system . Similarly, summing the processor ticks available yields the total time available on the sys tem for hand ling time- share tasks. Using this data, the scheduler calculates the desired load for each processor once per second, as fol lows:
Desired load
Processor ticks available X
System time-share load
System ticks available
Load balancing is cal led for when at least one pro cessor is above and one is below its desired load by a mi nimal amount. I f this condition arises, then those processors under their desired loads are declared to be "out of balance:· The next time an out-of-balance processor reschedules, it will try to take a thread from the local run queue of a proces sor that is above its desired load (" thread stealing").
A processor can declare itself back in balance when its current load is above its desired load or when there are no eligible threads to steal. Figure 3 shows a simpl ified load-balancing scenario, in which a processor below its desired load steals a thread from a processor above its desired load .
To help preserve the cache benefits of soft affin ity, a thread is el igible for stealing only when it has not run on its current processor for some config urable number of clock ticks. After this time has elapsed without a thread running, the chance of it having significant cache state remaining has dimin ished sufficiently that the thread is more l ikely to benefit from migrating to another processor and running im mediately than from waiting longer to run on its current processor.
To demo nstrate that soft affinity with load bal ancing improves multiprocessing performance through cache benefits and the elimination of run queue contention, developers ran a simple test pro gram. The program, which writes 128 KB of data, yields the processor, and then reads the same data back, was run on a four-processor DEC 7000 system.
D igital Technical journal
Vol. G No. 3 Summ.er 1994 39
DEC OSF/1 Symmetric Multiprocessing
CPU 1 CPU 2 CPU N
CURRENT LOAD
(NUMBER OF
THREADS)
DESIRED LOAD
5
4
CPU 1 IS
OUT OF
BALANCE
3
4
4
LOCAL
R U N
H IGHEST PRIOR ITY
THREAD BETWE EN
LOCAL RUN QU EUES
Q U E U E I
AND GLOBAL R U N Q U E U E
WINS THE PROC ESSOR
CPU 2
STEALS
ONE THREAD
FROM CPU 1
LOCAL
R U N
QUEUE
I I
. . .
4
LOCAL
R U N
QUEUE
I
GLOBAL
R U N
Q U E U E
Figure 3 Load Balancing
Tab.le 1 shows the results of running multiple versions of this program with and without soft affin ity and load bal ancing in operation. Performance benefits appear only when mu ltiple copies of the p rogram begin piling up in the run queues at the 16 -job level. Prior to this point, each job keeps running on the same processor, i . e . , the cache it had just filled still had its data cached when the pro gram read i t back-the ideal case. At the 16-job level, the four processors must be time-shared . The jobs that are runni ng with soft affinity now benefit significantly because they continue to r u n on the same processor and thus find some of their cache state preserved from when they last ran. The sys tems that schedu le from a global r u n queue provide no such benefit. jobs take longer to complete, since they are l i kely to run on a different processor for each time slice and find no cache state that they can reuse.
The soft affinity and load-balanci ng features improved many other multiuser benchmarks. For example, a transaction processing benchmark showed a 17 p ercent performance improvement.
Focusing on Quality
The error-checking focus of the lock package is just one example of how the DEC OSF/1 version 3.0 proj ect focused on the quality and stabil ity of the prod uct. Most members of the multiprocessing team had been involved in an SMP development effort prior to their DEC OSF/ 1 effort. Th is past experi ence, coupled with the difficulties other vendors had experienced with their own mu l tiprocessing i mplementations, reinforced the need to have a strong qual ity focus in the SMP project.
Developers took mu ltiple steps to ensure that the Si'vlP solution delivered in D EC OSF/1 version 3.0 would be production quality, includi ng
Table 1 Benefits of Soft Affin ity with Load Balancing (SAILS)
Nu mber of Jobs
Time with SAILB
(Seconds)
Time without
SA/LB (Seconds)
1
4
16
25.9
25.9
1 06.5
26.0
26.0
1 41 .9
Benefit from
SA/LB (Percent)
0
0
25
40 14>1. 6 No. 3 Sumn11!r 1994 Digital Tecbnica/ journal
DEC OSF/
1
Version
3. 0
Sym metric Multiprocessing Implementation
•
Code reviews
•
Lock debugging
•
In-line assenion checking
•
Multithreaded test suite development for SMP qual ification
The base kernel code was reviewed for m u lti processing correctness. D u ring this review phase, checks were made to ensure that the proper level of synchronization was employed to protect global data structures. Numerous defects were uncovered during this process and corrected. Running code with lock checking enabled provided empirical evidence of the incremental improvements of the multiprocessing implementation.
Beyond code reviews and lock debugging, inter nal consistency checks (assertions) were coded into the kernel to verify correctness of operations at key points. Assertion checking was enabled dur ing the development process to ensure that the ker nel was functioning correctly; it was then compiled out for the production version of the kernel.
In parallel with the operating system develop ment effort, new component tests were designed to force as much concurrency as possible through panicular code paths. The core of the test suite is a thread-race l ibrary, which consists of a set of rou tines that can be used to construct multithreaded system-call exercisers. The library provides the ability to commence multiple test instances simul taneously. The individual tests are then combined to form focused subsystem tests and systemwide tests. These tests have been used to u ncover multi ple race cond itions in the operating system .
The UNIX development organization had a four processor DEC 7000 system deployed in its develop ment environment for more than 7 months prior to releasing the S M P product. This system has been extremely stable, with few complaints from the user community. Extensive internal and external field testing produced similar results.
Measuring Multiprocessing
Performance Outcomes
The major functiona lity cleliverecl with SMP is improved performance through application con currency. The goal of the SM P project was to provide leadership performance in the areas of compute and data servers. To gauge success in this effort, several industry-standard benchmarks were utilized. These benchmarks include SPECrate_INT92,
SPECrate_FP92, and
AIM
Suite
I I I .
SMP performance is measured in terms of the incremental performance gained as processors are added to the system. Adding processors by no means guarantees increased system performance. Systems that have l/0 or memory l imitations rarely benefit from introducing additional CPUs. Systems that are compute bound tend to have the largest potential for gain from additional processors. Note that large, monolithic applications tend to see l it tle perfor mance improvement as processors are added because such applications employ little concur rency in their designs.
Performance tuning for SMP was an iterative pro cess that can be characterized as follows:
1 . Collect and analyze performance data.
•
•
•
•
•
CPU utilization across the processors
Lock statistics
1!0 rates
Context swi tch rates
Kernel profil ing
2. Identify areas that require improvement.
3. Prototype changes.
4. Incorporate changes that demonstrate improve ment.
5. Repeat steps 1 through 4.
In reality, the process has two stages for each benchmark. The initial phase was devoted to driv ing the system to become compute bound. The sec ond phase improved cocle efficiencies.
Figures 4 and 5 show that, as expected, the
SPECrate_JNT92 and SPECrate_FP92 benchmarks scale almost l inearly. Both of these benchmarks are compute intensive and make only nominal demands on the operating system.
AIM Suite III is a m ultiuser benchmark that stresses m u ltiple components of an operating sys tem, including the virtual memory system, the scheduler, UNlX pipes, and the 1/0 subsystem.
Figure 6 shows A I M J l l resu lts for one and four pro cessors, with a resulting 3.27 to 4 scaling factor.
This equates to a greater than 80 percent scal ing factor, a figure well within the goals for this bench mark at first multiprocessing release. Efforts to pro duce still better results are under way.
AIM Suite III scaling appears to be gated by a single test in the AJi\1 Suite Ill benchmark, i.e.,
D igital Technical Journal Vol. 6 No. J Summer 1994
4 !
Chandrika Kamath
Roy Ho
Dwight
P.
Manley
DXML: A Highperformance
Scientific Subroutine Library
Mathematical subroutine libraries for science and engineering applications are an important tool in high-jJeJformance computing. By identifying and optimizing frequent�y used, numerically intensive operations, these libraries help in reducing the cost of computation, enhancing portability, and improving productivity The
Digital eXtended Math Library is a set of public domain and Digital proprietmy software that has been optimized for high performance on Alpha systems. In this paper, DXi'v!L and the issues related to library software tecbnolog)l are described.
Specific examples illustrate bow algorithms can be optimized to take advantage of the architecture of Alpha systems. Modem algorithms that effectively e:�ploit the memory hierarchy enable DXi'v!L routines to provide substantial improvements in pe1jonnance.
The Digital eXtended !'vlath Library (DXML) is a set of mathematical subrouti nes, optimized for high performance on Alpha systems. These subroutines perform numerical ly i ntensive subtasks that occur frequently in scientific computing. They can there fore be used as building blocks for the optimization of various science and engineering appl ications in industries such as chemical. aerospace, petro leum, automotive, electronics, finance, and transportation.
In this paper, we discuss the role of mathematical software l ibraries, fo l lowed by an overview of the contents of the Digital eXtended Math Library.
DXML includes optimized versions of both the stan dard
BLAS
and
LAPACK
l i braries as well as l ibraries designed and developed by Digital for signa.l pro cessing and the solution of sparse l i near systems of equations. Next, we describe various aspects of l ibrary software technology, includi ng the design ancl testing of DXJ'vlL subroutines. Using key routines as examples, we illustrate the techn iques used in the performa nce optimization of the l ibrJry.
Final ly, we present data that demonstrates the per formance improvement obtained through the use of DXML.
The Role of Math Libraries
Early mathematical libraries concentrated on sup plemen ting the fu nctiona l ity provided by the
Fortran compilers. In addition to ro utines such as sin and exp, which were included in the run- time math l ibrary, more complicated special functions, l inear algebra algorithms, and Fou rier transform algorithms were included i n the software layer between the hardware and the user application.
Then, in the early 1970s, there was a concerted effort to produce high-quality nu merical software, with the aim of providing end users with implemen tations of numerical a.lgorithms that were stable, robust, and accurate. This led to the development of several math libraries, with the public domain
UNPACK
and
EISPACK
libraries for the solution of l inear and eigen systems, setting the standards for future development of math software. ' - '
The late 1970s and early 1980s saw the avail ability of advanced architectures, i nclud i ng vector ancl parallel computers, as well as h igh-performance workstations. This added another facet to the devel opment of math l ibraries. namely, the implemen tation of algori thms for h igh efficiency on an underlying architecture.
TI1e effort to produce mathematical software thus became a task of building bridges between numeri cal analysts, who devise algorithms, computer archi tects, who design high-pertormance computer systems, and computer users, who need efficient, reliable software for solving their problems. Con sequently, these libraries embody expert knowledge in appl ied mathematics, nu merical analysis, data structures, software engineering, compilers, oper ating systems, and computer architecture and
44 l·b/ (> No. 3 Summer J<J91
Digital Technical journal
DXML: A
Hig1J-pe1jormance Scientific Subroutine libra!)'
are an important progra mmi ng tool in the use of l1igh-performance computers.
Modern superscalar R!SC architectures with floating-point pipelines, such as the AJpha, have deep memory hierarchies. These include floa ting point registers, mu ltiple levels of caches, and virtual memory. The significant latency and bandwidth d if ferences between these memory kve. ls i mply that nume rical algori thms have to be restructured to make effective use of the data brought into any one level. The perfor mance of an algorithm is also sus ceptible to the order in wh ich comp u tations are sched uled as wel l as the h igher cost associated with some operations such as floating-po int square-root and division.
The architecture of the Alpha systems and the tech nology of the Fortran and C compilers usual ly provide an efficient computing environment with adequate performance. However, there is often room for improvement, especia l ly in engineering and science appl ications. where vast amounts of data are processed and repeated operations are per formed on each data element. One way to achieve these improvements is through the use of opti mized subroutine l ibraries.
The Digital eXtended Math Li brary is a collection of rou tines that performs frequently occurring, nume rically intensive operations. By identifying such operations and optimizing t hem for h igh per fo rmance on AJpha systems.
DXM L
provides several benefits to the comp utational scientist.
•
It al lows defin ition of fu nctions at a su fficiently h igh l evel and therefore optimization beyond the capabilities of the compiler.
• It makes the arch itecture of the systems more tra nsparent to the user.
•
•
•
I t improves prod uctivity by providing easy access to highly optimized, efficient code.
It enhances the portabil ity of user software through the support of standard libraries and interfaces.
It promotes good software engineering practice and avo ids dupl ication of work by identifying and optimizing common fu nctions across sev eral appl ication areas.
Overview
of DXML
DXM L contains almost 400 user-callable rou tines, optimized fo r Alpha systems.> This i ncludes both software developed by Digital as we l l as the 13LAS and LAl'ACK libraries. Most ro utines are available in fo ur ve rsions: real single precision, real double p recision, complex single precision. and complex double precision.
DX ML
is available on both Ope nVMS and DE C
OSF/ I operating systems. Its ro ut ines can be cal led from either Fortran or C, provided the difference i n array storage between these languages is taken into account. DXML is ava i l able as a shareable library. with a s imple interface, enabl ing easy access to the rou tines. O n DEC OSF/1 systems,
DX M L
supports the
IEEE floating-point format. On OpenVMS systems, either the IEEE floating-point format or the VAX
F-float/G -float format can be selected .
DX M L routines can be broadly categorized into the following fo ur areas:
•
BLAS. The Basic Linear AJgebra Subroutines include the standard BIAS and Digital enhancements.
•
LAI'ACK.
The Linear Algebra PAC Kage incl udes l i near and eigen-system solvers.
• Signal processing. This includes fast Fourier transforms (FFTs), convolution, and correlation.
•
Sparse linear system solvers. These include direct and iterative solvers.
Of these, the signal-processing l ibrary and the sparse I in ear system solvers are designed . deve l oped, and optimized by Digital. The majority of the
BLAS ro utines and the
l.A.PACK
l ibrary are versions of the public domain standard that were optim ized for the Alpha architecture. By support ing the industry standard interfaces of these I ibraries,
DX ,V!L
p ro vides both portability of user code and high perfor ma nce of the optimized software.
We next provide a brief description of the fimc tiona l ity provided by each subcomponent of DXM L.
Further details are availa ble i n the Digital eXtended
Math Library Reference Manual. s
VL/B
The vector library consists of seven double precision routines that perform operations such as sine. cosine, and natu ral logarithm, on data stored as vectors.
BLAS I
The Basic L inear AJgebra level I subprograms per form low-granu larity operations on vectors that involve o ne or two vectors as input and return either a vector or a scalar as output." Examples of
BLAS I rout ines include dot product, index of the maximum element in a vector, and so on.
Digital Technical journal
I-1JI. 6 .) Sutwl/el' 1')94
4 5
Scientific Computing Optimizations for Alpha
BIAS
1
Extensions (BIAS
1
E)
Digital has extended the functionality of the BLAS l routi nes by including 13 similar operations. These include index of the minimum e lement of a vector, sum of the elements of a veer or, and so on.
BIAS
1
Sparse (BIAS
1
S)
DX M L also includes nine rou tines that are sparse extensions of t he BLAS 1 routines. Of these, six are from the sparse BLAS 1 standard and three are enhancements.- These rou tines operate on two vectors, one of which is sparse and stored in a com pressed for m . As most of the elements in a sparse vector are zero, both compu tational time and mem ory are reduced by storing and operating on only the nonzeros. ULAS lS routines include construc tion of a sparse vector from the specified elements of a dense vector, dot prod uct, and so on.
BIAS 2
The BLAS level 2 rou tines perform operatio ns of a h igher granul arity than the Ievel l routines .s These include ma trix-vector operations such as matrix vector product, ra nk-one and rank- two upda tes, and solutions of triangular systems of equations.
Va rious storage schemes are supported, in cluding genera l, symmetric, banded, and packed.
BIAS 3
The BLAS level .1 routines pe rform m atrix-matrix operations, which are of a h igher granularity than the B L A S 2 operations. Tl1ese rou t ines i nclude matrix-matrix product, rank-k updates, solution of triangular systems with multiple right -hand sides. and multiplication of a matrix by a triangular matrix.
Where appropriate, these operations are defined for matrices that may be general, sym metric, or tri angular 9 The functionality of the public domain
BLAS
3
library has been en hanced by th ree addi t iona I rout ines for m atrix addition, su btraction, and transpose.
LA PACK
DXM L includes the standard Linear Algebra
PACKage, LAPAC K, which supersedes the U NPAC K and EISPACK packages by extending the fu nctio nal ity, using algorithms with higher accuracy, and improving the performance through the use of the optimized flLAS I i brary m LA PACK can be used for solving many co m mo n l inear algebra prob lems, including solution of l i near systems, l inear least -squares problems, eigenva lue problems, and singu lar value problems. Various storage schemes are su pported, i ncl uding general, band, trid iagonal, sym metric positive definite, and so on.
Signal Processing
The signal- processing subco mponent of DX M L includes l'FTs, convolutions, a n d correlations .
A comprehensive set of Fourier transforms is provided, incl uding
• fFTs in one, two. and three di mensions
• FFTs in fo rward and inverse d irections
•
M u ltiple one-dimensional transforms
There is no I imit on the numbe r of elements being transformed , though the performance is best when the data length is a powe r of 2. Popu lar storage for mats for the input and output data are supported, a l l owing for possible sym metry i n the output data and consequent red uction in the storage required .
Further efficiency is provided th rough the use of the three-step F FT, which separates the process of setting up and deallocating the in ternal data stru ctures from the actual application of the FFT.
This resu Its i n sign ificant p erfo rmance ga i n when repeated application of FFTs is requ ired.
The convolution and correlation rou tines in
DXNI L support both periodic (circular) and nonperi oclic (l i near) definition. A discrete summ ing tech nique is used for calculation. Special versions of t he routines a l low control of output options such as the range of coefficients computed, sca ling of the output, ancl addition of the output to an array.
A l l FFT, convolution , and correlation routines are avai.lable in both single and double precision and support both real and complex data.
sparse Iterative Solvers
DXJ'vl L inclu des a set of routines for the iterative solu tion of sparse l inear systems of equations using pre conditioned, conj ugate-grad ienr-like methods. 1 1 · 1l
A flexible user interface, based on a matrix-free for mu lation of the solve r, al lows a choice among vari ous solvers, storage schemes, and preco nditioners.
This formulation permits the user to defi ne his or her own preconcl itioner ancl/or storage scheme for the matrix. It also al lows the user to store the m atrix using one of the storage schemes defined by OX.vtL and/or use the preconditioners provided .
A d river routine provides a si mple interface to the iterative solvers when the DX M L storage sche mes and preconditioners are use d .
46 vbl. 6 No
.
. J Summer 1')')4
Digital Tech11ical journal
DXML: A
High-performance Scientific Subroutine Library
The different iterat ive methods provided are
( 1 ) conjugate gradient,
(2)
least-squares conj ugate grad ient, (3) biconjugate grad ient, grad ient squared, and
(5)
(4)
conjugate generalized minimum residual. Each method supports various appl ica tions of the preconditioner: left, right, split, and no precond itioning.
The matrix can be stored in the symmetric diago nal storage scheme, the unsymmetric diagonal stor age scheme, or the general storage (by rows) scheme. Three preconditioners are provided for each storage scheme: diagonal, polynomial (Neumann), and incomplete LU with zero diagonals added.
A choice of four stopping criteria is provided, in addition to a user-defined stopping criterion.
The iteration process can be control led by setting various input parameters such as the maximum number of iterations, the degree of polynomial pre conditioni ng, the level of output provided, and the tolera nce for convergence. These solvers are avail able in real double precision only.
Sparse Skyline Solvers
The sparse skyline solver library in
DXML
includes a set of routines for the direct solution of a sparse linear system of equations with the matrix stored using the sky line storage scheme. 13·14 The fol lowing functions are provided .
•
•
•
LDU factorization, which includes options for the evaluation of the determinant and inertia, partial factorization, statistics on the matrix, and options for handl ing small pivots.
Solve, which i ncludes mul tiple right-hand sides and solves systems involving either the matrix or its transpose.
Norm evaluation, including 1-norm, infinity norm, Frobenius norm, and the maximum abso lute value of the matrbc
•
Condition number esti mation, which includes both the 1-norm and the i nfinity norm.
•
Iterat ive refinement, including the componen t wise relative backward error and the estimated forward error bound for each solu tion vector.
•
Simple and expert drivers.
This fu nctionality is provided for each of the fol lowing storage schemes:
•
For symmetric matrices:
- Profile-in storage mode
- Diagonal-out storage mode
•
For unsymmetric matrices:
- Profile-in storage mode
- Diagonal-out storage mode
- Structural ly sym metric profile-in storage mode
These solvers are available in real double precision on ly.
Software Considerations
As with any software effort, many software engi neering issues were encou ntered during the design and development of DXML. Some issues were spe cific to math l ibraries such as the numerical accu racy and stabil ity of the routines, while others were more general such as the design of a user i nterface, testing of the software, error checking, ease of use, and portability. We next discuss some of these key design issues in further detail.
Designing the Interface
The first task in creating a l ibrary was to decide the fu nctional ity, fol lowed by the design of the inter face. This included both the naming of the subrou tines as wel l as the design of the parameter l ist. For each subcomponent in DXML, the call ing sequence was designed to be consistent across al l routines in that subcomponent. In the case of the BLAS and
LAPACK
li braries, the public domain interface was maintained to enable portability of user code.
For the rou tines added by Digital, the routine names were chosen to indicate the function being performed as we l l as the precision of the data.
Furthermore, the parameter l ists were chosen to provide a simple interface, yet allow flexibil ity for the sophisticated user. For example, the sparse solvers require various real and integer parameters.
By using arrays instead of scalar variables, a more concise i n terface that did not vary from routine to routine was obtained. I n addition, all solver routines have arguments for real and integer work arrays, even if these are not used in the code. This not only provides a uniform interface but also acts as a placeholder for work arrays, should they be required in the future.
Accuracy
The numerical accu racy of the routines i n
DXML
is dependent on the problem size as wel l as the algo rithm used, which may vary within a routine. Since performance optimization often changes the order in which a computation is performed, identical results between the DXML routines and the public
Digital Technical journal
Vol. 6 No. 3 Summer 1994 47
Scientific Computing Optimizations for Alpha domain RLAS and LAPACK rout ines may not occur.
The accuracy of the results obtained is checked hy ensuring that the op timized versions of the llLAS and LAPACK rou ti nes pass the public domain tests to with i n the specified tolerance.
Error Processing
Most of the rou t i nes i n OX ML trap usage errors and provide sufficient inform ation so that the user can identify and fix the problem. Th e low- level. fine grained computational rou tines, such as the llLAS level I . do not provide this fu nction because the overhead of testing and erro r trapping wo u l d seri ously clegracle the performance.
In the case of B L A S 2, BLAS 3. and LA PACK, the pub l ic domain error-reporting mechanism has been maintained . If an input argu ment is i nval i d , such as a negative value for the order of the matrix, the rou tine prints out an error message and stops. If a fail ure occu rs in the course of the algorithm, such as a matrix being singular to wo rking prec ision, an error flag is set and control is retu rned to the call i ng program.
The signal-processing rou tines report success or failure using a status function value. Further infor mation on the error can be obtained by using a user callable routine t hat prints ou t an error message a nd an error flag. The user documentation indicates the actions to be taken to recover from the error.
In th e case of the sparse solvers, error is incli cated by setting an error flag and printing an ap pro priate message if the printing option is enabled.
Control is always returned to the ca l l i ng program.
Testing
OXML rou ti nes are rested for correctness and accu racy using a re g r e ss ion rest su ite. This includes both test code developed by D igital, as we l l as the publ ic domain test codes for llLAS and I.APACK.
These codes are usecl n o t o n ly during the i m ple mentation and performance o p t i m ization of the routines, but also d uring the build ing of the com pl e t e lib r ary from each of the s u b c o mponent s
.
The test codes check each rou tine extensively, inclu ding checks for error exits, accuracy of the resu lts obtained, invariance of read-only data and t he correctness of all paths through the code. As the complete r e gre ss i o n tests take over 20 hours to exec ute, two i n p ut data sets are used: a short one that tests each routine and can be used to make a qu ick check that a l l su bcomponents comp i led and b u i l t correctly, and a l ong data set that rests each path through a routine a nd is thus more exhaust ive
Many of the ro utines, such as the FFTs aml l3LAS .), are tested using random i n p u t data. However, some routines, such as the sparse solvers, operate on spe cific data structures or matrices with specific prop erties. These have been tested using mat rices generated from the finite d i ffe rence discretization of partial d i ffe ren tial equations o r using t he matri ces in the Harwei i-Boeing test suite. "
Anorher aspect to the OX 1vl L regress i o n test pack age is the in clusion of a performance rest gauge .
This software tests the performance o f key rou tines in each component of DX,\H. and is u sed to ensure that the performan ce of 0X ML rou tines is not adversely affected by changes in compilers or the operating svstems.
Pe1jormance Trade-ojfs
The design and optim i zation of the routines i n
DX:vi L often prompted a trade- off between perfor mance on one hand, and accu racy and gen er al i ty on the other. A l t hough every effort bas been made not to sacrifice accuracy for performance. the reordering of computations d u ring performance optimization m ay lead to resu l ts before optimiza tion that are nor bit-for-bit ident ical to the resu l ts after optim ization. J n other cases, performance has been sacrificed to ensure genera l i ty of a routine.
For example, a l though the matrix-free formu lation of the iterative solvers permits the use of any sparse matrix storage scheme. it could resu lt in a sl ight degradation in pe rformance due to less efficient use of the instruction cache and the i nabi l ity to reuse some of the data in the registers.
Per:formance Optimization
DXM L rou t i nes have been designed to provide high performance on t he Alpha systems. 11' These rou t ines are ta i lored to take advan tage of the sys tem characterist ics such as the nu mber of floa t i ng point registers. the size of the primary and secondary data caches, and the page si ze. This opti m ization i nvolves cha nges to data structures and the use of new a lgorithms as well as the restructur i ng of computation to effectivel y manage the mem ory hierarchy.
Several ge neral tech niques are used across all
DXML subcomponents to improve t he perfor mance. '- These include the f oll owing tech n iqu es:
•
Unro l l ing loops to make better use of the f l oati ng-point p ipel i nes
• Reusing data in registers and caches whenever possible
48 Vol. (>
No
.
.i S/1111111<'1' I'J'J4
Digital Technical journal
DXML: A Higb-pe1jormance Scientific Subroutine Librm:y
•
•
Managing the data caches effect ively so that the cache hit ratio is maxim ized
Accessing data using strideI computation
• Using algorithms that exploit the memory h ierar chy effectively
•
Reordering compu tations to minimize cache and translation bu ffe r thrash i ng
Al though many of these opti mizations are done by the comp i ler. occasionally, for example i n the case of the skyl ine solver. the data structu res or the i m plementation of the a lgorithm are such that they do not lend themselves to optim ization by the com pi ler. In these cases, expl icit reordering of the com putations is requ ired.
We next d iscuss these optimization techniques as used in specific examples. Al l performance data is for the DEC :)000 Model 900 system using the DEC
OSF/ 1 version 3.0 operating system. This work station uses the Alpha 2 1 064A chip, running at 275 megahertz (J'viHz). The on-ch i p data and instruction caches are each 1 6 kilobytes
(Kn)
i n size, and the secondary cache is 2 megabytes (MB) in size.
In the next section, we compare the perfor m a nce of DXM L BLAS and
LAPACK
routi nes with the corresponding pu bl ic domain rou ti nes. Both ver sions are written in standard Fortran and compiled using identical compi ler options.
Optimization of BLAS
1
BLAS 1 rou tines operate on vector and scalar data only. As the opera ti ons and data stru ctures are sim ple, there is l i ttle opportu n i ty to use advanced data blocking and register re use techn iques. Neverthe less. as the p lots in Fig ure I demonstrate, it is pos sible to optim ize the
BLAS
l rou tines by careful coding that ta kes advantage of the data prefetch features of the Alpha 2 1064A chip and avoids data path- rel ated stal I s.
Jh. J H
General ly, the DXML rou tines are 10 percent to 15 percent faster than the corresponding public domain rou t i nes. Occasional ly, as i n the case of
DDOT for very short, cache- resident vectors, the benefits can be much greater.
The shapes of the plots in Figure 1 rather dramat ica l l y demonstrate the benefi ts of data caches. Each plot shows very high performance for short vectors that reside in the
16-KH,
on-chip data cache, much lower pe rforma nce for data vectors that reside in the 2-MB, on-board secondary data cache, and even lower performa nce when the vectors reside com pletel y in memory.
250
200
(f) 1 50 a::
:2 1 00
I
\
- -
o L_�--��--�������
6 8 1 0 1 2 1 4 1 6 1 8 20 22
VECTOR LENGTH (AS POWER OF 2)
K E Y :
-B L A S DAXPY
- - - - DXML DAXPY
· · · · · · · · · ·
· -
BLAS DDOT
DXML DDOT
Figure I PeJ.formance of BLAS I Routines
DDOT and DAXP Y
Optim.ization of BLAS 2
BLAS 2 rou t ines operate on matrix, vector. and scalar data. The data st ructu res are larger and more complex than the BLAS 1 data structures and the operations more compl icated . Accordi ngly, these rou tines lend themse lves to more sophisticated optim ization techniques.
Optimized DX.,\1l BLAS 2 routines are typica l ly 20 percent to 100 percent faster than the public domain rou ti nes. Figure 2 il lustrates this performance improvement for the matrix-vector mu ltiply routine.
DGE.\1V. and the triangu lar solve routine. DTRSY.H
The DXML DGEMY uses a data-blocking technique that asymptotical l y performs two floati ng-poi nt operat i o ns for each mem ory access, compared to the publ ic domain ve rsion, which performs two floating-poi n t operations for every three memory accesses.
19
This tec hnique is designed to m i n i mize translation bu ffer and data cache misses and m axi m ize the use of floating-point registers u'
IH 2o
The same data prefetch considerations used on the BLAS
1 routines are also used on the BLAS 2 rou ti nes.
The DXJ\'IL version of the DTRSY rou tine partitions the problem such that a sma.l l triangu lar solve oper ation is performed. The resu l t of this solve opera tion is then used in a DGEM V operation to u pdate the remainder of the vector. The process is repeated unt i l the final triangu lar update comp letes the operation. Thus the DTRSV rout ine rel ies heavily on the optim izati o ns used in the DGEMY rou tine.
Digital Technical jounwl
Vol.
() No . .l Summer 1994
49
Scientific Computing Optimizations for AJpha
1 40
1 20 I
1 00
(f)
0::
80
0 u.
2 60
40
20
'
'
'
'
'
'
- . '
.._
"'
.
-
.
0 200 400 600 BOO
ORDER OF V ECTORS/MATRICES
KEY:
-BLAS D G E M V
- - D X M L D G E M V
BLAS DTRSV
· · · · ·
-
· - · -
DXML DTRSV
- - -
.
-
..,. _ _ _ _
1 000
1 80
1 60
1 40
I
I
I
I
/ - -
1 20
(f)
0:: 1 00
�
80 I
60
40
20
0 200
KEY:
-- BLAS D G E M M
- - - - DXML DGEMM
- · -
·
BLAS DTRSM
- DXML DTRSM
400 600
O R D E R OF MATRICES
800 1 000
Figure 2 Performance of BLAS 2 Rou tines
DGEMV and DTRSV
As with BLAS 1 routines, BLAS 2 rout ines be nefit greatly from data cache. Although the effect is less dramatic for the BLAS 2 routines, Figure 2 clearly shows the three-step profile observed in Figure 1 .
Best performance is achieved when both matrix and vector fit in the primary cache. Performance is lower but flat over the region where the data fits on the secondary board level cache. The final per formance plateau is reached when data resides entirely in memory.
Optimization of BLAS 3
BLAS 3 rou tines operate primarily on matrices. The operations and data structures are more compl i cated that those of
Typically, BLAS
BLAS 1 and BLAS 2 routines.
3 routines perform many computa tions on each data element. These routines exhibit a great deal of data reuse and thus naturally lend them selves to sophisticated optimization techniques.
DXML BLAS
3
rou t i nes are general ly two to ten times faster than their public domain counterparts.
The plots in Figure 3 show these performance dif ferences for the ma trix-matrix mul tiply routine,
DGEMM, and the triangular solve routine with multi ple right -hand sides, DTRSM 9
Al l performance optimization techniques used for the on the
DXML B LAS 1 and BLAS 2 routines are used
DXM L BLAS 3 routines. In particul ar, data blocking techniques are used extensively. Portions
Figure 3 Performance of BIAS 3 Routines
DGEMM and DTRSM of mat rices are copied to page-aligned work areas where secondary cache and translation bu ffer misses are eliminated and primary cache misses are absolutely minimized.
As a n example, within the primary compute loop of the DXJVIL DGEMM rou tine, there are no transla tion bu ffer misses, no secondary cache misses, and, on average, only one pri mary cache miss for every
42 floating-point operations. Performance within this key loop is also enhanced by carefu l ly using floating-point registers so that fou r floating-point operations are performed for each memory read access. Much of the DXML BLAS 3 performance advantage over the publ ic domain routi nes is a direct consequence of a greatly improved ratio of floating-point operations per memory access.
The DXML DTRSM routine is optimized in a man ner similar to its BLAS 2 counterpart, DTRSV. A small triangu lar system is solved . The resu l ti ng matrix is then used by DGEMM to update the remainder of the right-hand-side matrix. Consequently, most of the DXML DTRSM performance is directly attrib utable to the DXML DGEMM routine. In fact, the tech n iques used in DGEMM pervade DXM L BLAS
3
routines.
Figure 3 i l lustrates a key feature of DXML BLAS 3 routines. Whereas the performance of public domain rou tines degrades sign ificantly as the matrices become too large to fi t in caches, DXJ\1 L
50 Vol. 6 No. 3 Summer 1994 Digital Technical journal
DXML: A High-performance Scient1jic Subroutine Library routines are relatively insensit ive to array size, shape , or orientation."
9
The performance of a DXML
BLAS 3 routine typical ly reaches an asymptote and remains there regard less of problem size.
Optimization of LAPACK
The LAPACK subroutine library derives a large part of its h igh performance by using the opti mized BLAS as build ing blocks w The DXML ver sion of LAPACK is largely unmod ified from the publ ic domain ve rsion. However, in the case of the factorization routine for general matrices,
DGETRF, we have introduced changes to the algorithm to improve the performance on Alpha systems.
For example, while the original public domain
DGETRF routine uses Crout's method to factor a matrix, the DXML version uses a left - looking method
. 1 1
Left - looking methods make better use of the secondary cache and translation bu ffers than the Crout method. Furthermore, the public domain version of the DLASWP routine swaps a single matrix row across an entire matrix . This is a very bad technique for RISC machines; it causes severe cache and translation bu ffer thrashing. To avoid this, the DXML version of DLASWP p erforms all swaps within colu mns, which makes much better use of the caches and the translation bu ffer and resu lts in a much improved performance of the
DXJ\IIL DGETRF routine.
The DGETRS routine was not modified. Its perfor mance is solely attributable to use of optimized
DXM L routines.
Figure 4 shows the benefits of the optimizations made to DG ETRF and the BLAS routines. DGETRF makes extensive use of the BLAS 3 DGEMM and
DTRSM routines. The performance of DXML DGETRF improves with increasing problem size largely because DXML BLAS 3 romines do not degrade in the face of larger problems.
The plots of Figure 4 also show the performance ofDGETRS when processing a single right-hand-side vector. In this case, DTRSV is the dominant BLAS rou tine, and the performance differences between the public domain and DXML DGETRS routines reflect the performance of the respec tive DTRSV routines. Final ly, although not shown, we note that the performance of DXM L DGETRS is much better than the public domain version when many right hand sides are used and DTRSM becomes the domi nant BLAS routine.
1 40
1 20
1 00
(f) a: g
80 lJ._
::;;
60
I
I
I
I
/
· '
'
40
20
'
0 200 400 600 800
O R D E R OF VECTORS/MATR ICES
KEY:
--
BLAS DGETRF
- - - - DXML
DGETRF
- · -
· · ·
BLAS DGETRS
-
DXML DGETRS
Figure 4 Performance of LA PACK Routines
DGETRF and DGETRS (LDA
=
N + 1)
Optimization of the
Signal-processing Routines
1 000
We illustrate the techniques used in optimizmg the signal-processing routines using the one dimensional, power-of-2, complex FFT.21 The algo rithm used is a version of Stockham's autosorting algorithm, which was origina l ly designed for vector compu ters but works wel l, with a few modifica tions, on a RISC architecture such as AJpha 22·2.1
The main advantage in using an au tosorting algo rithm is that it avoids the initial bit-reversal permu tation stage characteristic of the Cooley-Tukey algorithm or the Sande-Tukey algorithm. This stage is implemented by either precalculating and load ing the permutation indices or calculating them on- the-fly. ln either case, substantial amounts of integer mul tipl ications are needed . By avoiding these mu ltipl ications, the autosorting algorithm provides better performance on Alpha systems.
This algorithm does have the disadvantage that it cannot be done in-place, resulting in the use of a temporary work space, which makes more demands on the cache than an algorithm that can be clone i n-place. However, this disadvantage is more than offset by the avoidance of the bit-reversal stage.
The implementation of the FFT on the Alpha makes effective use of the hierarchical memory of the system, specifically, the 31 usable floating-point
Digital Technicaljow·nal Vol. 6 No. 3 Summe1· 1994 5 1
Scientific Computing Optimizations for AJpha registers, which are at the lowest, and therefore the fastest, level of this hierarchy. These registers are util ized as much as possible, and any data brought into these registers is reused to the extent possible.
To accomplish this, the FFT rou tines i mplement the largest radices possible for all stages of the power of-2 FFT calculation. Radix-8 was used for all stages except the first, util izing 16 registers for the data and 14 for the twidd le factors 21 For the first stage , as all twiddle factors are I , radix-16 was used.
Figure 5 i l lustrates the performance of this algo rithm for various sizes. Although the performance is very good for small data sizes that fit into the pri mary, 16-KB data cache, it drops off qu ickly as the data exceeds the primary cache. To remedy this, a blocking algorithm was used to better util ize the primary cache.
The blocking algorithm. which was developed for computers with hierarchica l memory systems, decomposes a large FFT into t wo sets of smal ler
FFTs 2·i The algorithm is implemented using four steps:
I. Compute NI sets of FFTs of size N2.
2. Apply twid d le factors.
3. Compute N2 sets of FFTs of size N l .
4 . Transpose the N l b y N2 matrix into a n N 2 b y Nl matrix.
200
1 80
1 60
(f)
1 40 g 1 20 lL
:2 1 00
80
60
1 6
SIZE OF FFT (AS POWER OF 2)
KEY:
AUTOSO RTING
- - - - BLOCK I N G
Figure 5 Pet.Jornumce of 1-D Complex FFJ
In the above, N = Nl X N2. Steps ( I ) and (3) use the autosorting algorithm for small sizes. Jn step (2), instead of precomputing all N twiddle factors, a table of selected twiddle factors is com puted and the rest calculated using trigonometric identities.
Figure 5 compares the performance of the block ing algorithm with the autosorti ng algorithm. Due to the added cost of steps (2) and (4), the maximum computation speed for the block ing algorithm
( 1 15 m i l l ion floating-point operations per second
[Mflops] at N = 2 12) is tower than the maximum computation speed of the autosorting algorithm
( 192 Mftops at N= 2�). The crossover point between the two algorithms is at a size of ap proxi mately 2K. with the aurosorting a lgorithm perform ing better at smatter sizes. Based on the length of the FFT. the ox:vt L routine automatically p icks the faster algorithm. Note that at N= 2 10, as the size of the data and workspace exceeds the 2-:'-•!B secondary cache, the performance of the blocking algorithm drops off.
Optimization of the Skyline Solvers
A sky l ine matrix (Figure 6) is one where only the elements within the envetopt: of the sparse matrix are stored. This storage scheme exploits the fact that zeros that occu r before the first nonzero ele ment in a row or col u m n of the matrix, remain zero d uring the facto rization of t he matrix. pro vided no row or column interchanges are made. l-l
Thus, by sroring the e nvelope of the matrix, no additional storage is required for the fi l l-in that occurs du ring the factorization. Though the sl-.-y t ine storage scheme does not exploit the sparsity within the envelope. it allows for a static data structure, and is therefore a reasonable compro mise between organizational simplicity and com putational efficiency
In the sky l ine solver, the system, Ax= b, where A is an N by N matrix, and b and x are tV-vectors, is solved by first factorizing A as A = l.DU, where L and
U are unit lower a nd u pper triangular matrices, and
D is a diagona l matrix. The solution x is then calcu lated by solv i ng in order. Ly = b, Dz = y, and Ux = z, where y and z are tV-vectors.
In our discussion of performance optimization, we concentrate on the factorization routine as it is often the most time-consuming part of an appl ica tion. The algori thm implemented in DXML uses a tech n ique that generates a column (or row) of the
52 Vol. o
No
.
.l SIIIIIIJ/l!l' /')')4 Digital Teclmicaljourrwl
DXML: A Highperformance Scientific Subroutine Library
COLUMN i COLUMN j
+ +
-----·
n
- - - - - -
U
)
LENGTH OF THE
I N N E R PRODUCT
FOR THE
EVALUATION
OF ELEM ENT (i. j)
- Row ;
Figure 6 Skyline Column Storage of a Symmetric Matrix factorization using an inner product fo rmulation.
Specifically, fo r a symmetric matrix A, Jet
A
=
(ur w r
�)
�) (�/ 0 d
) cu.\1 w)
0
I
where the symmetric factorization of the leading
(N - I) by (N - 1) leadi ng principal submatrix M has already been obtained as
M
= v:; D.H U.u
Since t he vector u, of length
(N - I),
s are known, the vector w, of length (N
-
1 ) and the scalar d can be determined as and
The defi nition of w indicates that a column of the factorizati o n is obtai ned by tak ing the inner prod uct of the appropriate segment of that colu m n with one of the previous columns that has al ready been calculated. Referring to Figure 7, the value of the element in location (i,j) is calcul ated by taking the inner product of the elements in col u m n j above the element in location (i,j) with the corre sponding elements in column i. The entire colu mn
Figure 7 Unoptimized Skyline
Computational Kernel j is thus calculated starting with the first nonzero element in the column and moving down to the d iagonal entry.
The optimization of the sky line fac torization is based on the fol lowing two observations
25 26:
•
The elements of column
j,
used in the evalua tion of the element in location
(i,j),
are a lso used i n the evaluation of the element in location
(i+ l ,j).
•
The elements of column i, used in the evalua tion of the element in location (i,j), are a lso used i n the eva luation of the element i n locat ion
(i,j + l) .
Therefore, b y unrol l i ng both t h e inner loop on i and the outer loop on), twice, we can generate the entries in locations (i,j), (i+ l,j), (ij+ 1 ) , (i+ 1 ,)+ 1 ) a t the same time, as shown i n Figure 8 . These fou r elements are generated using o n l y h a l f the memory references made by the standard algorithm. The memory references can be reduced further by i ncreasing the level of u nrol li ng. This is, however, l imited by two factors:
•
The number of floating-point registers required to store the elements being calcu lated and the elements in the columns.
•
The length of consecutive columns i n the matrix, which should be close to each other to derive fu ll benefit from the unro l l i ng.
Based on these factors, we have unrolled to a depth of 4, generating 16 elements at a time.
D igital Technical journal
Vol. 6 No. 3 Summer 1994 53
Scientific Computing Optimizations for Alpha
54
)
LENGTH OF THE
I N N E R PRODUCT
FOR THE PARTIAL
EVALUATION OF
ELEMENTS (i,
(i + 1 , and
(i +
1, j
+ j) j),
(i, j +
1)
1 ) �
ROW i
ROW ( i + 1 )
Figure 8 Optimized Skyline
Computational Kernel
A similar technique is used in optimizing the for ward elimination and the backward substitution.
Table 1 gives the performance improvements obtained with the above techniques for a symmet ric and an unsym metric matrix from the Harwell
Boeing collection .1" The characteristics of the matrix are generated using DXJ.\1l routines and were included because the performance is dependent on the profile of the sky line. The data presented is for a single right-hand side, which has been generated using a known random solution vector.
The resu lts show that for the matrices u nder con sideration, the technique of reducing memory references by unroll i ng loops at two levels leads to a factor of 2 improvement in performance.
Summary
In this paper, we have shown that optimized mathe matical subroutine libraries can be a useful tool in improving the performance of science and engi neering applications on Alpha systems. We have
Table 1 Performance I mprove ment in the Solution of Ax =b, Using the Skyl ine Solver on the
DEC 3000 Model 900 System
Example 1 Example 2
Harweii-Boeing matrix1s
Description
BCSSTK24
Stiffness matrix of the Calgary
Olympic Saddledome Arena
Symmetric diagonal-out
ORSREG1
Jacobian from a model of an oil reservoir
Unsymmetric profile-in Storage scheme
Matrix characteristics
Order
Type
3562
Symmetric
Condition number estimate
Number of nonzeros
Size of skyline
Sparsity of skyline
Maximum row (co lumn) height
Average row (column) height
RMS row (column) height
Factorization time (in seconds)
Before optimization
After opti mization
Solution time (in seconds)
Before opti mization
After optimization
Maximum component-wise relative error in solution
(See equation below.)
6.37E + 1 1
81736
2031722
95.98%
3334
570.39
1 1 35.69
66.80
35.02
0.82
0.43
0.1 6 E - 5
2205
Unsymmetric with stru ctu ral symmetry
1 .54E + 4
1 4 1 33
1 575733
99.1 0%
442 (442)
357.81 (357.81 )
395.45 (395.45)
23.1 2
1 3.02
0.32
0.17
0.50 E - 1 0 max I x(i) - x(i) I
I x(i) I where x(i) i s the i-th component of the true solution, and x(i) i s the i-th component of t h e calcu lated solution.
Vol. 6 No 3 Summer 1994
Digital Technical journal
DXML: A High-performance Scientific Subroutine Library
described the functionality provided by DXML, discussed various software engineering issues and illustrated techniques used in performance optimization.
Future enhancements to DX!\1L i nclude symmet ric multiprocessing support for key routines, enhancements in the areas of signal processing and sparse solvers, as wel l as further optimization of rou tines as warranted by changes in hardware and system software.
Acknawledgments
DXML is the joint effort of a number of individuals over the past several years. We would like to acknowledge the contribu tions of our col leagues, both past and present. The engineers: Luca Broglio,
Richard Chase, Claudio Deiro, Laura Farinetti, Leo
Lavin, Ping-Charng Lue, Joe O'Connor, Mark
Schure, Linda Tel la, Sisira Weeratunga and John
Wilson; the technical writers: Cheryl Barabani,
Barbara Higgins, Marl! McDonald, Barbara Schott and Richard Wolanske; and the management: Ned
Anderson, Carlos Baradello, Gerald Haigh, Buren
Hoffman, Tomas Lofgre n, Vehbi Tasar and David
Vel ten. We would also like to thank Roger Grimes at
Boeing Computer Services for making the Harwell
Boeing matrices so readily available.
References
1 . W Cowell, ed . ,
Sources and Development of
Mathematical Software
(Englewood Cliffs,
NJ: Prentice-Hall, 1984) .
2 . D. Jacobs, ed . , Numerical Software-Needs and Availability (New York: Academic Press,
1978).
3. ]. Dongarra, ]. Bunch, C. Moler, and G. Stewart,
LINPACK Users' Guide (Philadelphia: Society for Industrial and Applied Mathematics
[SIAM ] , 1979).
4. B. Smith et a!,
Matrix Eigensystem
Routines-EISPACK Guide
(Berlin: Springer
Verlag, 1976).
5.
Digital eXtended Math Library Reference
Manual
(Maynard, MA: D igital Equipment
Corporation, Order No. AA-QOMBB-TE for
VMS
and AA-QONHB-TE for OSF/ 1 ) .
6. C. Lawson, R. Hanson, D. Kincaid, and
F Krogh, " Basic Linear Algebra Subprograms for Fortran Usage,"
ACM Transactions on
Mathematical Software,
vol. 5, no. 3 (Septem ber 1979) : 308-323.
7. D. Dodson, R. Grimes, and ]. Lewis, "Sparse
Extensions to the FORTRAN Basic Linear Alge bra Subprograms,"
ACM Transactions on
Mathematical Software,
vol . 17, no. 2 (June
1991) : 253-263.
8. ]. Dongarra, ]. DuCroz, S. Hammarl ing, and
R. Hanson, "An Extended Set of FORTRAN
Basic Linear Algebra Subprograms,"
ACM
Transactions on Ma thematical Software,
vol. 14, no. 1 (March 1988): 1-17.
9. ]. Dongarra, ]. DuCroz, S. Hammarling, and
I . Duff, "A Set of Level 3 Basic Linear Alge bra Subprograms,"
ACM Transactions on
Mathematical Software,
vo l. 16, no. 1 (March
1990): 1-17.
10. E. Anderson et at. ,
LAPACK Users' Guide
(Philadelphia: Society for Industrial and
Applied Mathematics [SIAM] , 1992).
1 1 . ] . Dongarra, I . Duff, D. Sorensen, and H . van der Vorst, Solving Linear Systems on Vector and Shared Memory Computers (Philadel phia: Society for Industrial and Applied Math ematics [SIAM ] , 1991 ).
12. R. Barrett et at. ,
Templates fo r the Solution of
Linear Systems: Building Blocks for Iterative
Methods
(Philadelphia: Society for Industrial and Applied Mathematics [SIAM ] , 1993).
13. C. Felippa, "Solution of Linear Equations with
Skyline Stored Symmetric Matrix," Computer and Structures, vol. 5, no. 1 (April 1975): 13-29.
14. I . Duff, A. Erisman, and]. Reid,
Direct Methods for Sparse Matrices
(New York: Oxford
University Press, 1986).
15. I. Duff, R. Grimes, and ]. Lewis, " Sparse
Matrix Test Problems,"
ACM Transactions on
Mathematical Software,
vol. 15, no. 1 (March
1989): 1- 14.
Digital Tech11iCtll jour11al
Vol. 6 No. 3 Summer 1994 55
Scientific Computing Optimizations for Alpha
16.
Alpha AXP Architecture and Systems,
Digital
Technical]ournal,
vol. 4, no.
4
(Special Issue
1992).
17.
K. Dowd,
High Performance Compu ting
(Sebastopol, CA: O'Reilly & Associates, Inc . ,
1993).
18. DECcbip 21064-AA Microprocesso r-Hard ware Reference Manual (Maynard, tvlA:
Digital Equipment Corporation, Order No.
EC-N0079-72, October 1992).
19. ].
Dongarra and S. Eisenstat, "Squeezing the
Most Out of an Algorithm in CRAY FORTRAN,"
ACM
Transactions on Mathema tical Soft ware,
vo l.
10,
no. 3 (September
1984):
219-230
20.
R. Sites, ed.,
Alpha Architecture Reference
Manual
(Burl ington, 1YlA: Digital Press,
1992).
2 1 . H. Nussbaumer, Fast Fourier Transforms and
Convolu tion Algorithms, Second Edition
(New York: Springer Ve rlag, 1982) .
22.
D. Bai ley, "A H igh-performance FFT Algorithm for Vector Supercomputers,"
The interna tional journal of Supercomputer Applica tions,
vol . 2, no. 1 (Spring
1988): 82-87.
23.
P. Swarztrauber, " FFT Algorithms for Vector
Computers,"
Parallel Computing,
vol. 1, no.
1
(August
1984): 45- 63.
24.
D. Bailey, "FFTs in External or Hierarchical
Memory,"
The journal of Supercomputing,
vol .
4,
no. 1 (March
1990): 23-35.
25.
0. Storaasli, D. Nguyen, and T. Agarwal,
"Para llel-Vector Solution of Large-Scale Struc tural Analysis Problems on Supercompu ters,"
American Institute of Aeronau tics and
Astronautics
(Ju ly
(AIAA) jo urnal,
vol.
28,
no.
7
1990): 1 2 1 1 - 1216.
26.
H . Samukawa, "A Proposal of Level
3
Interface for Band and Skyline Matrix Factorization Sub rou tine,"
Proceedings of the 1993 ACM In ter national Conference on Super Computing,
Tokyo, japan (July
1993): 397- 406.
56
Vr!l. 6 No
.
.3 Summer 1994
Digital Tecbnica/ journal
Robert H.
Kuhn
Bruce Leasure
Sanjiv
M.
Shah
The KAP Parallelizer for
DEC
Fortran and
DEC C
Programs
The KAP preprocessor optimizes DEC Fortran and DEC C programs to achieve their best performance on Digital Alpha systems. One key optimization that KAP per forms is the parallelization of programs for Alpha shared memory multiprocessors that use the new capabilities of the DEC OSF/1 version 3. 0 operating system with
DECthreads. The heart of the optimizer is a sophisticated decision process that selects the best loop to parallelize from the many loops in a program. The preproces sor implements a robust data dependence analysis to determine whether a loop is inherently serial or parallel. In engineering a high-quality optimizer, the designers specified the KAP software architecture as a sequence of modular optimization passes. These passes are designed to restructure the program to resolve many of the apparent serializations that are artifacts of coding in Fortran or C. End users can also annotate their DEC Fortran or DEC C programs with directives or pragmas to guide KAP's decision process. As an alternative to using KAP's automatic parctl lelization capabilit]\ end users can explicitly identify parallelism to KAP using the emerging indust1:v-stcmdard X3H5 directives.
The KAP preprocessor developed by Kuck &
Associates, Inc. (KAI) is used on Digital Alpha sys tems to increase the performance of DEC Fortran and DEC C programs. KAP accomplishes this by restructuring fragments of code that are not effi cient for the Alpha architecture. Essentially a super optimizer, K AP performs optimizations at the source code level that augment those performed by the DEC Fortran or DEC C compilers . 1
To enhance the performance o f DEC Fortran and
DEC C programs on Alpha systems, KAI engineers selected two challenging aspects of the Alpha archi tecture as KAP targets: symmetric multiprocessing
(SMP) and cache memory. An additional design goal was to assist the compiler in optimizing source code for the reduced i nstruction set computer
(RISC) instruction processing pipeline and multiple functional units.
This paper discusses how the KAP preprocessor design was adapted to parallelize programs for SMP systems running under the DEC OSF/1 version 3.0 operating system. This version of the DEC OSF/ 1 system contains the DECthreads product, Digital 's
POSJX-compliant multithreading l ibrary. The first part of the paper describes the process of mapping paral lel programs to DECthreads. The paper then discusses the key techniques used in the KAP design. Final ly, the paper presents examples of how
KAP performs on actual code and mentions some remaining challenges. Readers with a compiler background may wish to explore Optimizing Super compilers for Supercomputers for more details on
KAP's techniques.
2
I n this paper, the term directive is used inter changeably to mean directive, when referring to DEC
Fortran programs, and pragma, when referring to
DEC C programs. The term processor generally rep resents the system component used in parallel pro cessing. In discussions in which it is significant to d istinguish the operating system component used for parallel processing, the term thread is used.
The Parallelism Mapping Process
Figure 1 shows the input modes and major phases of the compilation process. Parallelism is repre sented at three levels in programs using the KAP preprocessor on an Alpha SMP system. The first two are input to the KAP preprocessor; the third is the
Digital Technical journal Vul. 6
No.
3 S11mmer 1994 57
Scientific Computing Optimizations for Alpha
� - - - - - - - - -,
I I M PLICIT PARALLELISM
I
I
I
I
DEC C PROGRAM
L - - - - - - - -
�
I
I
I
I
I
I
I
I
I
I
- -
-
- -
-
-
-
-
I
EXPLICIT HIGH-LEVEL PARALLELISM
KAP G U I DING
D IR ECTIVES
-
KAP ASSERTIONS
-
-
-
-,
I
I
I
I
I
- - - J
KAP PARALLELISM
DETECTION AND
OPTIM IZATION
-
_ _ _ _ _ _
-
KAP-OPTIMIZED
FORTRAN OR
C OUTPUT FILE
-
_ _
-
I
I
I
I
I
L
EXPLICIT LOW-LEVEL PARALLELISM
_ _ _
I
_j
I
I
1
I
DEC FORTRAN
OR DEC C
COMPILER
DEPENDENCE
ANALYSIS
KAP PRE PROCESSOR
Figure 1 Parallelism Mapping Process representation of parallelism that KAP generates.
The three levels of parallel ism are
1 . Implicit parallelism. Starting from DEC Fortran or DEC C programs, KAP automatically detects parallel ism.
2. Explicit high-level parallelism. As an advanced feature, users can provide any of three forms:
KAP guiding directives, KAP assertions, or X3H5 directives. KAP guiding directives give KAP hints on which program constructs to parallelize . KAP assertions are used to convey information about the program that cannot be described in the DEC
Fortran or DEC C language. This information can sometimes be used by KAP to optimize the pro gram. Using X3H5 directives, the user can force
KAP to parallelize the program in a certain way.3
58 Voi. 6 No. 3 Summer 1994 D igital Technical journal
The KAP Parallelizer for DEC Fortran and DEC C Programs
3. Explicit low- level parallelism. KAP translates eitl1er of the above forms to DECthreads with the help of an SMP support l ibra ry. (The user could specify paral lelism directly, using DECthreads; however, KAP does not perform any optimiza tion of source code with DECtlU"eads. Therefore, the user should not mix this fo rm of parallelism with the others.)
Because the user can employ para l lelism at any of the three levels, a discussion of the trade-offs involved with using each level fol lows.
From DEC Fortran or DEC C Programs
The
KAP
preprocessor accepts DEC Fortran and DEC
C programs as input. Although starting with such programs requires the compilers to intell igently utilize a h igh-performance
SMP
system, there are several reasons why this is a natural point at which to start.
• Lots of software. Since DEC Fortran and DEC C are de facto standards, there exists a large base of applications that can be parallelized relatively easily and inexp ensively.
• Ease of use. Given the high rate at which hard ware costs are decreasing, eveqr workstation may soon have multiple processors. At that point, it will be critical that programming a multiproces sor be as easy as programming a single processor.
•
Portabil ity. Many software developers with access to a m u ltiprocessor already work in a het erogeneous networking environment. Some sys tems in such an environment do not support expl icit forms of para l lelism (either X3H5 or
DECthreads). The developers would probably l ike to have one version of their code tllat runs we ll on all their systems, whether uniprocessor or mul tiprocessor, and using DECthreads would cause their uniprocessors to slow down.
•
Maintainability. Using an intricate programming model of para] lei ism such as X3H5 or DECthreads makes it more difficult to maintain the software.
KAP
produces KAP-optimized DEC Fortran or DEC
C as output. This fac t is imponant for the fol lowing reasons:
•
Performance. Users can leverage optimizations from both Digital's compilers and
K AP.
•
Integration. Users can employ all of Digital 's per formance tools.
•
Ease of use. Expert users like to " tweak" the output of
KAP
to fine-tune the optimizations performed.
With KAP Guiding Directives, KAP
Assertions, or X3H5 Directives
Although the automatic detection of parallelism is frequently within the range of
KAP
capabili t ies on
SMP
systems, i n some cases, as described below, users may wish to specify the parallelism.
•
In the SMP environment, coarse-grained paral lelism is sometimes important . The higher in the cal l tree of a program a preprocessor (or com piler, as wel l) operates, the more diffic u lt it is for a preprocessor to paral lel ize automatically.
Even though the KAP preprocessor p erforms both inlining and interprocedural analysis, the higher in the call tree KAP operates, the more l i kely it is that KAP will conservatively assume that the parallel ization is inval id.
•
Sometimes information that is available only at run time precludes the preprocessor from auto matically finding paral lelism.
•
Occasionally, experts can fine-tune the paral lelism to get the highest efficiency for programs that are run frequent ly.
•
For software that is more portable between sys tems, it is sometimes important to get repeatable parall el performance or to indicate where paral lelism has been appl ied . In such cases, explicit paral lelism may be preferable.
Three mechanisms are available to the user for directing K AP to parallelism. The first mechanism uses KAP guiding directives to guide KAP to the preferred way to parallel ize the program. The sec ond mechanism uses K AP assertions. The t hird mechanism uses X3H5 -compl iant d irectives to directly describe the parallelism. The first two mechanisms differ significantly from the third. With the first two, K AP analyzes the program for the feasi bility of parallelism. With the third, KAP assumes that parallelism is feasible and restricts itsel f to man aging the details of implementing paral lelism . In particular, the user does not have to be concerned with either the scoping of variables across proces sors, i.e., designating which are private and which are shared, or the synchronization of accesses to shared variables 4 K AP guiding directives will not be
Digital Tecl:mical Journal Vol. 6 No. 3 Summer 1994 59
Scientific Computing Optimizations for Alpha discussed in this paper. KAP assertions and how they are implemented are discussed later in the section
Advanced Ways to Affect Dependences. A descrip tion of the X3H5 d irectives follows.
The X3H5 model of parallelism is wel l struc tured ; all operations have a begin operation-end operation format. The paral lel region construct ident ifies the fork and join points for paral lel processing. Parallel loops iden tify units of work to be d istribu ted to the available processors. The critical section and one processor section con structs are used to synchronize processors where necessary Table 1 shows the X3H5 directives as implemented in KAP.
To the DEC OSF/1 Operating System with DECthreads
Although KAP does not optim ize programs that use
DECthreads directly, there may be some benefits to specifying parallelism expl icitly using DECthreads.
•
DECthreads al lows a user to construct almost any model of paral lel processing fairly efficien tly
The high-level approaches described above are limited to loop-structured para l lel processing.
Some applicati ons obtain more paral lelism by using an unstructured model . It can even be argued that for some cases, unstructured paral lelism is easier to understand and maintain.
•
A user who invests the time to analyze exactly where paral lelism exists in a program may wish to fo rego the benefits mentioned above and to capture the parallelism in detail with DECthreads.
In that manner, no efficiency is lost because the preprocessor misses an optimization.
•
The POSJX threads standard to which DECthreads conforms is available on several platforms.
Because this standard is broad ly adopted and language independent, it is only sl ightly less portable than implicit parallelism.
The KAP preprocessor translates a program in which KAP has detected implicit parallelism or a pro gram in which the user expl icitly d irects parallelism to DECthreacls. KAP p erforms this trans lation in two steps. First, it translates the internal representation into ca lls to a paral lel SMP support l ibrary. Second , the support l ibrary makes cal ls to DECthreads.
The SMP support I ibrary implements various aspects of X3H5 notation, as can be seen by com paring Tables 1 and 2.
In t h e paral lelism translation phase, K A P signifi cantly restructures a program by moving the code in a paral lel region to a separate subroutine. A cal l to the SMP support library replaces the paral lel region. This cal l references the new subroutine.
KAP examines the scope of each variable used in the parallel region and, if possible , converts each variable to a local variable of the new subroutine.
Otherwise, the variable becomes an argu ment to the subrou tine so that it can be passed back out of the parallel region.
Converting variables to local variables m akes accessing these variables more efficient. A variable that is referenced outside the paral lel region cannot be made local and must be passed as an argument.
Shared Memory Multiprocessor
Architecture Concerns
Given its paral lelism model, the KAP preprocessor requ ires operating system and hardware support from the system for efficient paral lel execution .
There are three areas of concern: t hread creation and sched ul ing, synchron ization between threads, and data caching and system bus bandwidth.
Table 1 X3H5 Directives As Implemented in KAP
Function
To specify regions of parallel execution
To specify parallel loops
To specify synchronized sections of code such that all processors synchronize
To specify that all processors execute sequential ly
To specify that only the first processor executes
X3H5 D irectives
C*KAP* PARALLEL REGION
C*KAP* END PARALLEL REGION
C*KAP* PARALLEL DO
C*KAP* END PARALLEL DO
C*KAP* BARRIER
C*KAP* CRITICAL SECTION
C*KAP* END CRITICAL SECTION
C*KAP* ONE PROCESSOR SECTION
C*KAP* END ONE PROCESSOR SECTION
60 Vol. (, No . .) Sumn1er 1994 D igital Technical journal
The KAP Parallelizer for DEC Fortran and DEC C Programs
Table 2 KAP SMP Support Libra ry
Fortran
Name C Entry Point Name
__ k m p_ e n t e r _ c s e c
__ k m p_ e x i t
c s e c
__ k m p_ f o r k m p p e c s m p p x c s m p p f r k m p p f k d
__ k m p_ f o r k a c t i v e
__ k m p_ e n d m p p e n d
__ k m p _e n t e r _o n e p s e c
__ e x i t _o n e p s e c m p p b o p m p p e o p
__ k m p_b a r r i e r m p p b a r
Function
To enter a critical section
To exit a critical section
To fork to several threads
To inq u i re if already parallel
To join threads
To enter a single processor section
To exit a single processor sect ion
To execute a barrier wait
OSF/1 DECthreads
Subroutines Used p t h r e a d _ m u t e x _ l o c k p t h r e a d _m u t e x _ u n l o c k p t h r e a d _ a t t r_ c r e a t e , p t h r e a d_ c r e a t e
( n o n e ) p t h r e a d _ j o i n , t h r e a d d e t a c h p t h r e a d _m u t e x _ l o c k , p t h r e a d _m u t e x _ u n l o c k p t h r e a d _ m u t e x _ l o c k , p t h r e a d _m u t e x _ u n l o c k p t h r e a d _m u t e x _ L o c k , p t h r e a d _ c o n d _w a i t , p t h r e a d _m u t e x_ u n l o c k
Thread Creation and Scheduling
Thread cre ation is the most expensive operation. The X3H5 standard minim izes the need for creating threads through the use of paral lel regions. The SMP sup port l i brary goes further by reusing threads from one parallel region to the next. The SMP support library examines the value of an environment vari able to determine how many threads to use. The appropriate sched ul ing of threads onto hardware processors is extremely important for efficient execution. The support l i brary relies on the
DECthreads implementation to achieve this. For the most efficient operation, the l i brary should schedule at most one thread per processor.
Synchronization between Threads
In the KAP model of parallelism, threads can synchronize at
•
A point where loop iterations are scheduled
•
A point where data passes between iterations
(for col lection of local reduction variables only)
•
A barrier point leaving a work-sharing construct
•
Single processor sections
Two versions of the SMP support library have been developed: one with spin locks for a single-user environment and the second with mu tex locks fo r a multiuser environment. Either l ibrary works in either environment; however. using the spin lock version in a single-user environment yields the most efficient para llelism .
Using spin locks i n a mu ltiuser environment may waste processor cycles when there are other users who could use them. Using mu tex locks for a single user environment creates un necessary operating system overhead. In practice, however, a system may sh ift from single-user to mul tiuser and back again in the course of a single run of a large pro gram. Therefore, KAP supports all lock-environment combinations.
Data Caching and System Bus Bandwidth
Multiprocessor Alpha systems support coherent caches between processors. s
To use these caches efficiently, as a policy, KAP localizes data as much as possible, keeping repeated references within the same processor. Loca li zing data reduces the load on the system bus ancl reduces the chances of cache thrashing.
When all the processors simultaneously request data from the memory, system bus bandwidth can limit SMP performance. If optimizations enhance cache locality, less system bus bandwidth is used , and therefore SMP performance is less likely to be l imited .
KAP
Technology
This section covers the issues of data dependence analysis, preprocessor architecture, and the selec tion of loops to paralleli ze.
Digital Tech11ica/ journal .3 Summer 1994 61
Scientific Computing Optimizations for Alpha
Data Dependence A nalysis-The Kernel of Parallelism Detection
DEC Fortran and DEC C have standard ru les for the order of execution of statements and expressions.
These rules are based o n a serial model of program execution. Data dependence analysis allows a com piler to see where this serial order of execution can be modified without changing the meaning of the program.
Types of Dependence KAP works with the four basic types of dependence:6
1 . Flow dependence, i.e., when a program writes a variable before it reads the variable
2 . Antidependence, i . e . , when a program reads a variable before it writes the variable
3.
Output dependence, i . e . , when a program writes the same variable twice
4. Control dependence, i.e., when a program state ment depends on a previous conditional
Because dependences involve two actions on the same variable, for example, a write and then a read,
KAP uses the term dependence arc to represent information flow, in this example from the write to the read.
Since these dependences can prevent paralleliza tion, KAP uses various optimizations to eliminate the different dependences. For example, an optimi zation called scalar renaming removes some but not all antidependences.
Loop-related Dependences When dependences occur within a loop, the control flow relations are captured with d irection vector symbols tagged to each dependence arc.
2
The transformations that can be applied to a loop depend on what depen dence direction vectors exist fo r that loop. The symbols used in KAP and their meanings are
The dependence occurs within the same loop iteration.
> The dependence crosses one or several itera tions.
< The dependence goes to a preceding iteration of the loop.
The dependence relation between iterations is not clear. or a combination of the above, for example,
< > The dependence is known not to be on the same iteration.
1
2
3
4
When a dependence occurs in a nested loop, KAP uses one symbol fo r each level in the loop nest. A dependence is said to be carried by a loop if the cor responding direction vector symbol for that loop incl udes < , > , or *
In the fol lowing program segment f o r ( i
= t e m p
1 ; i < = n ; i + + ) { a [ i J ; a [ i J b [ i ] b [ i ] ;
= t e m p ; } there is a flow dependence from statement 2 to statement 4. There is an antidependence from state ment 2 to statement 3 and from statement 3 to statement 4. There are control dependences from statement 1 to statements 2, 3, and 4 because exe cuting 2, 3, and 4 depends on the i <
= n condition.
Al l these dependences are on the same loop itera tion; their direction vector is
= .
Some dependences i n this program cross loop iterations. Because temp is reused on each itera tion, there is an output dependence from statement
2 to statement 2, and there is an antidependence from statement 4 to statement 2. These two depen dences are carried by the loop in the program seg ment and have the direction vector > .
Data Dependence Analysis The purpose of depen dence analysis is to bu ild a dependence graph, i . e . , the collection o f all the dependence arcs i n the pro gram. KAP builds the dependence graph i n two stages. F irst, it bu ilds the best possible conservative dependence graph 7 Then, i t applies filters that identify and remove dependences that are known to be conservative, based on special circumstances.
What does the phrase "best possible conserva tive dependence graph " mean' Because the values of a program's variables are not known at prepro cessing time, in some situations it may not be clear whether a dependence actually exists. KAP reflects this situation in terms of assumed dependences based on impe rfect information. Therefore, a dependence graph must be conservative so that
K AP does not optimize a program incorrectly. On the other hand , a dependence graph that is too con servative results in insufficient optim ization.
In building the best possible dependence graph,
K AP uses the following optimizations: constant propagation, variable fo rward substitu tion, and
62 Vol. 6 No. 3 Summer 1994
Digital Tech nical journal
The KAP Parallelizer for DEC Fortran and DEC C Programs
scalar expansion.
KAP
does not, however, leave the program optimized in this manner unless the opti mizations will improve performance.
Advanced Ways to Affect Dependences
When there are assumed dependences in the program,
KAP
may not have enough information to decide on parallelism opportunities.
KAP
implements two techniques to mitigate the effects of imperfect i nformation at preprocessing time: assertions and alternate code sequences.
Assertions, which are similar to directives in syn tax, are used to provide information not otherwise known at preprocessing time.
KAP
supports many assertions that have the effect of removing assumed dependences. Table 3 shows
KAP
assertions and their effects 8.9 When the user specifies an asser tion, the information contained in the assertion is saved by a data abstraction cal led the oracle. When an optimization requests that a data dependence graph be built for a loop , the dependence analyzer inquires whether the oracle has any information about certain arcs that it wants to remove.
When accurate information is not known at com pile time , a few
KAP
optimizations generate two versions of the source program loop : one assumes that the assumed dependence exists; the other assumes that it does not exist. In the latter case,
KAP
can apply subsequent optimizations, such as para!lelizing the loop.
KAP
applies the two-version loop optimizations selectively to avoid dramatically increasing the size of the program. However, the payback of parallel izing a frequently executed loop warrants their use.
For example, the KAP C pointer disambiguation optimization is employed in cases in which C point ers are used as a base address and then incremented in a loop . Neither the base address of a pointer nor how many times the pointer will be incremented is usually known at compile time. At run time, how ever, they can be computed in terms of a loop index. KAP generates code that checks the range of the pointer references at the tail and at the head of a dependence. If the two ranges do not overlap, the dependence does not exist and the optimized code is executed.
KAP Preprocessor Architecture
A controversial control architecture decision in
KAP
is to organize the preprocessor as a sequence of passes, generally one for each optimization per formed . This design decision was controversial because of the fol lowing concerns:
•
Run-time inefficiency would occur in process ing programs because each pass would sweep through the intermediate representation for the program being processed, causing some amount of virtual memory thrashing.
Table 3 KAP Assertions
Assertion
[ N O ] A R G U M E N T A L I A S I N G
[ N O J B O U N D S V I O L A T I O N S
C O N C U R R E N T C A L L
D 0 (<specifier>)
D 0 P R E F E R (<specifier>)
[ N O J E Q U I V A L E N C E
H A Z A R D
[ N O J L A S T V A L U E
N E E D E D (<specifier>)
P E R M U T A T I O N
(<specifier>)
N O R E C U R R E N C E
(<specifier>)
R E L A T I 0 N (<specifier>)
Specifiers
S E R I A L , C O N C U R R E N T
S E R I A L , C O N C U R R E N T
Variable names for which [no] last value is needed
Names of permutation variables
Names of recurrence variables
Relation loop index known to be true
N O S Y N C
Primary Effect
Removes assumed dependence arcs
Removes assumed dependence arcs
Removes assumed dependence arcs
Guides selection of loop order strongly
Guides selection of loop order loosely
Removes assumed dependence arcs
(Fortran only)
Tunes the parallel code and sometimes removes assumed dependences
Removes assumed dependence arcs
Removes assumed dependence arcs
Removes assumed dependence arcs
Tunes the parallel code which is produced
Digital Technicaljout-nal Vol. 6 No. 3 Summer 1994
63
Scientific Computing Optimizations for Alpha
•
Added software development cost wou ld be incurred because the KAP code that loops through the intermediate representation wo uld be repeated in each pass.
The second concern has been dispelled. The added modularity of KAP, provided by its multipass structure, has saved development time as KAP has grown from a moderately complex piece of code to an extremely complex piece of code.
The KAP preprocessor uses more than 50 major optimizations. The pass structure has helped to organize them. In some cases, such as cache man agement, one optimization is broken into several passes. KAP performs some basic optimizations, e.g. , deadcode elimination, more than once in dif ferent ways. In some cases, such as scalar expan sion, KAP performs an optimization to uncover other optimizations and then performs the reverse optimization to tighten up the program again.
The run-time efficiency issue is sti l l of interest.
There is always some benefit to making the prepro cessor smaller and faster.
Selecting Loops to Parallelize
Paral lelizing a loop can greatly enhance the perfor mance of the program. Testing whether a loop can be paral lel ized is actua l ly quite simple, given the data dependence analysis that KAP performs. A loop can be parallelized if there are no dependence arcs carried by that loop. The situation, however, can be more comp l icated. If the program contains severa.l nested loops. it is important to pick the best loop to parallel ize. Add itional ly, i t may be possible not on ly to parallelize the loop but also to optimize the loop to enhance its performance. Moreover, the loops in a program can be nested in very complex structures so that there are many d ifferent ways to parallelize the same p rogram. In fact, the best option may be to leave a l l the loops serial because the overhead of para l lel execution may outweigh the performance improvement of using mu ltiple processors.
The K AP preprocessor optimizes programs fo r para l lelism by searching for the optimum program in a set of possible configurations, i.e., ways in which the original program can be transformed for parallel execution. (In this regard, KAP optimizes programs from a classical definition of numerical optimization.) There is an object ive fu nction for evaluating each configuration. Each member of the set of configurations is cal led a loop order. The optimum program is the loop order whose objec tive hmction has the highest performance score, as discussed later i n this section.
Descriptions of loop orders, the role of depen dence analysis, and the objective hmction, i.e., how each program is scored, fol low.
Loop Orders A loop order is a combination of loop transformations that the KAP preprocessor has performed on the program . The loop transforma tions that KAP performs wh ile searching fo r the optimal paral lel form are
•
Loop distribution
•
Loop fusion
•
Loop interchange
Loop distribution splits a loop into two or more loops. Loop fusion merges two loops. Loop fusion is used to combine loops to increase the size of the paraUel tasks and to reduce loop overhead .
Loop interchange occurs between a pair of loops.
This transformation takes the inner loop ou tside the outer loop, reversing their relation. If a loop is triply nested , there are three factorial (31), i .e . . six, differ ent ways to interchange the loops. Each order is arrived at by a sequence of pairwise in terchanges.
To increase the opportu nit ies to interchange loops, KAP tries to make a loop nest into one that is perfectly nested . This means that there are no exe cutable statements between nested loop state ments. Loop distribu tion is used to create perfectly nested loops.
KAP examines a l l possible loop orders for each loop nest. Each loop nest is treated independently because no transformations between loop nests occur at this phase of optimizat ion.
For example, an LU factorization program con sists of one loop nest that is three deep and not per fectly nested. Figure 2 shows the loop orders. Loop order (a) is the origina l LU program. The KAP pre processor first d istributes the ou ter loop in loop orders ( b) and (c). Next, K AP performs a loop inter change on the second loop nest which is two deep, as shown in loop order (d ). Then, KAP interchanges the third loop nest in loop orders (e) through (i).
Note that KAP eliminates some loop orders, ( i) for example, when the loop- bound expressions cannot be interchanged . As explained above, there are six d ifferent loop orders because the nest is triply
64 Vol. 6 No. 3 Summer 1')94
Digital Tecbnical journal
The KA P Parallelizer for DEC Fortran and DEC C Programs
(a)
ORIGINAL LU (OUTLINED): do i = 1 , n
/ * I n v e r t E l i m i n a t o r * / d o k = i + 1 , n
/ * C o m p u t e M u l t i p l i e r s * / e n d d o d o j = i + 1 , n d o k = i + 1 , n
/ * U p d a t e M a t r i x * / e n d d o e n d d o e n d d o
(d) FOR SECOND NEST INTERCHANGE
SECOND do i LOOP: do k = 1 , n d o i = 1 , k - 1
/ * C o m p u t e M u l t i p l i e r s * /
(b) DISTRIBUTED do i LOOP d o i = 1 , n
/ * I n v e r t E l i m i n a t o r * / e n d d o d o i = 1 , n d o k = i + 1 , n
/ * C o m p u t e M u l t i p l i e r s * / e n d d o d o j = i + 1 , n d o k = i + 1 , n
/ * U p d a t e M a t r i x * / e n d d o e n d d o e n d d o
(c)
DISTR IB UTE do i LOOP AGA I N : d o i = 1 , n
/ * I n v e r t E l i m i n a t o r * / d o i = 1 , n d o k = i + 1 , n
/ * C o m p u t e M u l t i p l i e r s * / d o i = 1 , n d o j = i + 1 , n d o k = i + 1 , n
/ * U p d a t e M a t r i x * /
REEXAM I N E LOOP ORDERS
(e) THROUGH (i)
(e) FOR THIRD NEST
I NTERCHANGE do i AND do j d o j = 1 , n d o i = 1 , j - 1 d o k = i + 1 , n
/ * U p d a t e M a t r i x * / l
(f) FOR THIRD NEST I NTERCHANG E do i AND do k :
L o o p O r d e r R e j e c t e d
-
N e w b o u n d s s p l i t L o o p . d o j = 1 , n d o k = 2 , j d o i = 1 , k - 1
/ * U p d a t e M a t r i x * / d o k = j , n d o i = 1 , j - 1
/ * U p d a t e M a t r i x * /
(g) FOR THIRD NEST
INTERCHANGE do j AND do k: d o i = 1 , n d o k = i + 1 , n d o j = i + 1 , n
/ * U p d a t e M a t r i x * /
(h)
FOR THIRD NEST
INTERCHANGE do i AND do k: do k = 1 , n d o i = 1 , k - 1 d o j = i + 1 , n
/ * U p d a t e M a t r i x * /
(i) FOR THIRD NEST INTERCHANGE do i AND do j:
L o o p O r d e r R e j e c t e d
-
N e w b o u n d s s p l i t l o o p . d o k = 1 , n d o j = 2 , k d o i = 1 , k - 1
/ * U p d a t e M a t r i x * / d o j = k , n d o i = 1 , k - 1
/ * U p d a t e M a t r i x * /
Figure 2 Loop Ordersjo1' LV Factorization
D igital Tecbnical]ow·naf
Vol. 6 No. 3 Summer 1994
65
Scientific Computing Optimizations for Alpha nested. Since the loop nest in (d) was originally nested with the triply nested loop at the outermost do loop, K AP will reexamine these six loop orders after the interchange in (d).
Dependence Analysis for Loop Orders
Before a loop order can be evaluated for efficiency, KAP deter mines the val idity of the loop order. A loop order is valid if the resulting program would produce equ iva lent behavior. KAP tests validity by examining the dependences in the dependence graph according to the transformation being applied.
For example, the test for loop interchange validity involves searching for dependence d irection vec tors of a certain type. The direction vector (<, >) indicates that a loop interchange is invalid. The direction vectors ( <,*), (*, >), or (*,*), if present, also indicate that the loop interchange may be invalid.
Evaluation of a Loop Order
After the KAP prepro cessor determines that a loop order is valid, it scores the loop order for performance. KAP consid ers two major factors: ( 1 ) the amou nt of work that will be performed in parallel and (2) the memory reference efficiency
The memory reference efficiency of a loop order can degrade performance so much that it out weighs the performance gained by executing a loop i n paral lel . On an SMP, if a processor refer ences one word on a cache l ine, it should reference all the words contiguously on that line. In Fortran, a two-dimensional array reference, A(
i
J)
, should be parallelized so that the j loop is parallel and each processor references contiguous columns of mem ory. If a loop order indicated that the i loop is paral lel, this reference would score low. If a loop order indicated that the j loop is parallel, it would score high. The score for the loop order is the sum of the scores for all the references, and the highest scoring loop order is preferred .
The score for a loop order depends on which loops in the order can be parallel ized . For a given loop nest, there may be several (or no) loops that can be parallelized. The first step is to determine if any loops can be parallelized . If multiple loops can be parallelized, KAP selects the best one. KAP chooses at most one loop for parallel execution.
KAP tests loops to determine whether they can be executed in parallel by analyzing both the state ments in the loop and the dependence graph. The loop may contain certain statements that block concurrentization. 1/0 statements or a call to a function or subroutine are examples. (Users can code
KAP assertions to flag these statements as parallel iz able.) Second, data dependence conditions may preclude parallelization. ln general, a loop that car ries a dependence is not paralleli zable. (In some cases, the user may override the data dependence condition by al lowing synchronization between loop iterations.) Fina lly, the user may give asser tions that indicate a preference for making a loop parallel or for keeping it serial.
Barring data dependence conditions that would prevent parallelization, the amount of work that will be performed in parallel determines the score of par al lel izing a loop. (The user can also specify with a directive that loops should not be parallelized unless they score greater than a specified value.) In this manner, KAP prefers to parallelize outer loops or loops that are interchanged to the outside because they contain the most work to amortize the over head of creating threads for parallelism.
The actual paral lelization process is even more complex than this d iscussion indicates. KAP applies a number of optimizations to improve the quality of the parallel code. If there is a reduction operation across a loop, KAP paral lelizes the loop. Too much loop distribution can decrease program efficiency, so loop fusion is run to try to coalesce loops.
Performance Analysis
How does the KAP preprocessor perform on real applications' The answer is as complex as the soft ware written for these applications. Consider the real-world example, DYNA3 D , which demonstrates some KAP strengths and weaknesses.
DYNA3D is nonlinear structural dynamics code that uses the finite element analysis method. The code was developed by the Lawrence Livermore
National Laboratory Methods Development Group and has been used extensively for a broad range of structural analysis problems. DYNA3 D contains about 70,000 lines of Fortran code in more than
700 subroutines.
When KAP is being used on a large program, i t i s sometimes preferable t o concentrate on the compute-intensive kernels. For example, KAP devel opers ran six of the standard benchmarks for
DYNA3D through a performance profiling tool and isolated two groups of three subroutines that account for approximately 75 percent of the run time in these cases. This data is shown in Table 4.
KAP's performance on some of these key subrou tines appears in Table 5. KAP paral lelized all the
66 Vol. 6 No. 3 Summer 1994 Digital Technical journal
The KAP Parallelizer for
DEC
Fortran and
DEC C
Programs
Table
4
Performance Profiles of Six DYNA3D Problems
Problem Profile (First Two I n itials of the
Subroutine and Percent of Run Time)
NIKE2D
Example
Cylinder Drop
Bar I m pact
I mpacted Plate
Single Contact
Clamped Beam
ST 19%, FO 1 5%, FE 1 2 % , PR 10%, HG 7%, HR 5%
ST 20%, FO 15%, FE 1 1 %, PR 10%, HG 7%, H R 5%
WR 17%, ST 7%, FE 6%
SH 22%, TN 16%, TA 16%, YH 14%, BL 7%
YH 24%, SH 21 %, TN 7%, TA 7%, BL 6%
EL 1 2%, SH 1 2 % , TN 8%, TA 8%, BL 6%
·call Sequences
(a)
ST is called; ST calls PR; and then FE is called.
(b) HR is called; HR calls HG; and then FO is called.
(c) BL calls SH, then
TA, and then
TN.
Table 5 KAP's Performance on Key Subroutines
Subroutine Number of
Loops
Number of Loops
Parallel ized
STRAIN
PRTAL
FELEN
FORCE
H RGMD
HGX
9
5
4
5
9
6
5
9
6
9
5
4
1
2
Maximum
Nest Depth
1
Key Call
Sequences*
(a) and (b)
(a) and (b)
None of interest
(c)
(c)
(c)
1
2
3
Number of Loops after Fusion
3 loops in these subroutines. Since DYNA3D was designed for a CRAY-1 vector processor, it is perhaps to be expected that the K AP preprocessor would pe rform well . KAP, however, is intended for a shared memory multiprocessor rather than for a vector machine. For this reason, KAP does more than parallel ize the loops. The entries in the col umn labeled " Nu mber of Loops after Fusion" show how KAP reduced loop overhead by fusing as many loops together as it could. KAP fused the five loops in subroutine STRAIN into three loops and fused all nine loops in subroutine PRTAL
Another example of KAP's optimization for an
SMP system is that in the doubly nested loop cases, such as subroutine FORCE (see Figure 3), the
KAP preprocessor automatical ly selects the outer loop for parallel execution. In contrast, a vector machine such as the CRAY-1 prefers the inner loop.
Because the kernels of DYNA3D code span multi ple subroutines, cross compilation optimization is suggested. There are three ways to do this: inl ining, interprocedural analysis, and directives specifying that the inner subroutines can be concurrentized. s u b r o u t i n e d o
F 0 R
� OUTER LOOP
PARALLELIZED
6 0 n l e n i O i 1
=
=
=
=
1 , n n c l c z c
+ n
+ n h 1 2 1 i a ( l n c ) i a ( l c n
+
1 ) 1 c d i r $ i v d e p d o 5 0 i
= i O , i 1 e ( 1 , i x ( i ) ) e ( 1 , i x 1 ( i ) ) + e p 1 1 ( i )
5 0 c o n t i n u e
6 0 c o n t i n u e
Figure 3 Parallel Loop Selection
Using KAP's inlining capabi l ity gives KAP the most freedom to optimize the program because in this manner KAP can restructure code across subroutines.
Figure 4 shows part of the cal l sequence of sub routine SOLDE. (Subroutine SOLDE conta ins ca l l
Digital Tech11ical ]our11al Vol. 6 No 3 Summer 1994 67
Scientific Computing Optimizations for Alpha s u b r o u t i n e S O L D E c a l l s u b r o u t
:
H R G M D ------------... i n e
H R G M
D �
WHOLE CALL
SEQUENCE c a l l f a l l
H G X
-----;
I N L I NED
F O R C E
.____/
Figure 4 lnlining a Kernel
sequence (b) of Table 4.) Subroutine SOLDE calls subroutine HRGMD which calls subrou tine HGX .
Then subroutine SOLDE calls subroutine FORCE.
KAP supports inlin i ng to an arbitrary depth.
Inl ining in KAP can be automatic or control led from the com mand l i ne. In this case, we did not want to enable inlining au tomatically to depth two of subroutine SOLDE because it contains cal ls to many other subrou tines that are not in the kernel. Here, the user specified the subroutines to inl ine on the command l ine. When the user specified inl ining,
KAP fused all the loops in subroutines H RGMD, HGX, a nd FORCE to minim ize loop overhead, and then it paral lel ized the fused loop.
I n some cases, the user can make simple restruc turing changes that improve KAP·s optimizations.
Figure 5 shows a case in which fu sion was blocked by two scalar statements between a pair of loops.
The first loop does not assign any values to the vari ables used to create these scalars, so the user can move the assignments above the loop to enable KAP to fuse them.
Finally, the user can elect to specify the paral lel ism directly. Figure 6 shows subroutine STRAIN with X3H5 directives used to describe the s u b r o u t i n e
?
o 5 i
:
=
S T R A I N l f t
, l l t e n d d o d t 1 d 2 c r h o d o
=
6 i
=
. 5 * d t 1
. 0 6 2 5 * r h o ( l f t )
L f t , l l t e n d d o s u b r o u t i n e d t 1 d 2 c r h o d o 5 i
=
=
S T R A I N
. 5 * d t 1
. 0 6 2 5 * r h o ( L f t ) l f t , l l t e n d d o d o 6 i l f t , l l t e n d d o
Figure 5 Assisted Loop Fusion
68
ALL c"kap·
STATEMENTS
ARE X3H5 EXPLICIT
PARALLEL DIRECTIVES. s u b r o u t i n e S T R A I N c * k a p * p a r a l l e l r e g i o n c * k a p * & s h a r e d ( d x y , d y x , d 1 ) c * k a p * & L o c a l ( i , d t 1 d 2 ) c * k a p * p a r a l l e l d o d o 5 i
=
L f t , l l t d y x ( i )
5 c o n t i n u e
= .
.
.
c * k a p * e n d p a r a l l e l d o c * k a p * b a r r i e r d t 1 d 2
= .
.
.
c * k a p * p a r a l l e l d o d o 6 i
=
L f t , l l t d 1
=
6 c o n t i n u e d t 1 d 2 * C d x y ( i ) + d y x ( i ) ) c * k a p * e n d p a r a l l e l d o c * k a p * e n d p a r a l l e l r e g i o n
Figure 6 X3H5 £y,.plicit Parallelism
Vol. 6 No. j S111nm�r 1994
Digital Tecb nical journal
The KAP Parallelizer for DEC Fortran and DEC C Programs parallelism. In this case, the user elected to keep the same unfused loop structure as in the original cod e. This case is not dramatically less efficient than the fused version because the parallel region causes KAP to fork threads only once.
A very sophisticated example of KAP usage occurs when a user inputs a program to KAP that has already been optimized by KAP. This is an advantage of a preprocessor that does not apply to a compiler because a preprocessor produces source code out put. I n this case, the statements shown in Figure 6 were generated by KAP to illustrate X3H5 paral lelism. A user may want to perform some hand opti mization on this output, such as removing the barrier statement, and then optimize the mod ified program with KAP again.
Challenges That Remain
Although the KAP preprocessor is a robust tool that performs wel l in a production software develop ment environment, several challenges remain.
Among them are adding new la nguages, further enhancing the optimization technology, and improving KAP's everyday usability.
As the popular program ming languages evolve,
KAP evolves also. KAJ will soon extend KAP support for DEC Fortran to Fortran 90 and is developing C++ optimization capabilities.
In optimization technology, KAJ's goal is to make an SMP server as easy to use as a single-processor workstation is today. "Automatic Detection of Par allelism: A Grand Chal lenge for High-Performance
Computing" contains a leading-edge analysis of par allelization technology. 10 The research reported shows that further develop ing current techniques can improve optimization technology. These tech niques frequently involve the grand chal lenge of compiler optimization-whole program analysis.
In a much more pragmatic direction, the KAP preprocessor should be integrated with Digital's compiler technology at the intermediate represen tation level . Such integration would increase pro cessing efficiency because the compiler would not have to reparse the source code. In addition, inte gration would increase the coord ination between
KAP and the compiler to imp rove performance for the end user.
Increasing the usability of the KAP preprocessor, however, benefits the end user directly. KAP engineers frequ ently talk to beta users and encour age feedback. The following are examples of user comments:
•
Optimizing programs is difficult when no sub routine in the program takes more than a few percent of the run time. As its usability in this area improves, KAP will become a substantial pro ductivity aid . If a program is general ly slow, opti mizing repeated usage patterns will allow the programmer to use a comfortable programming style and stil l expect peak system performance.
•
Increasing feedback to the user would improve
KAP's usabil ity. When KAP cannot pe rform an optimization, often the user can help in several ways (e.g. , by providing more information at compile time, by changing the options or direc tives, or by making small changes to the source code). KAP does not always make it clear to the user what needs to be done. Providing such feed back would improve KAP's usability.
•
Integration with other performance tools would be useful. Alpha systems have a good set of per fo rmance monitoring tools that can provide clues about what to optimize in a program and how. The next release of the KAP preprocessor will provide some simple tools that a user can employ to integrate KAP with tools l ike prof and to track down performance differences.
On a final note, the fact that KAP does not speed up a program should not always be cause fo r disap pointment. Some programs already run as fast as possible without the benefit of a KAP preprocessor.
Acknawledgments
We wish to acknowledge the Lawrence Livermore
National Laboratory Methods Development Group and other users for providing applications that give us insight into how to improve the KAP preproces sor. We would li ke to thank those at Digital who have been instrumental in helping us del iver
KAP on the DEC OSF/ 1 platform, especially Karen
DeGregory, John Shakshober, Dwight Manley, and
Dave Velten. Everyone at Kuck & Associates partici pated in the making of this product but of special note are Mark Byler, Debbie Carr, Ken Crawford,
Steve Healey, David Nelson, and Sree Simhadri.
References
I . D . Bl ickstein e t a l , "The GEM Optimizing
Compiler System," Digital Technical journal, vol . 4, no. 4 (Special Issue 1992): 121-136.
Digital Techrticnl journal
Vol. 6 No. 3 Swn-mer 1994 69
Scientific Computing Optimizations for Alpha
2.
M. Wol fe,
Optimizing Supercompilers for
Supercompu ters
(Cambridge, MA: MIT Press,
1989).
3. Parallel Processing Model Jot· High Level Pro gramming Languages, ANSI X3H5 Document
Number X3H5/94-SD2, 1994.
4. P Tu and D. Padua, "Automatic Array Privatiza tion," Proceedings of the Sixth Workshop on
Languages and Compilers fo r Parallel Com puting, vo l. 768 of Lecture Notes in Com puter Science (New York: Springer-Verlag,
1993): 500 - 521 .
5. B. Maskas et a ! . , " Design and Performance of the DEC 4000 AXP Departmental Server Com puting System," Digital Technical journal, vol. 4, no. 4 (Special Issue 1992): 82 -99.
6.
R. Allen and K. Kennedy, "Automatic Transla tion of FORTRAN Programs to Vector Form,"
ACM Transactions on Programm ing Lan guages and Systems,
vo l.
9,
no.
4
(October
1987): 491 - 542.
7
8.
9.
U. Banerjee, Dependence Analysis for Super computing ( Norwel l, MA: Kluwer Academic
Publishers, 1988).
KAP for DEC Fo rtran for DEC OSF/1 AXP User
Guide
(Maynard, MA: Digital Equipment
Corporation,
1994).
KAP
fo r C for
DEC
OSF/ 1
AXP
User Guide
(Maynard, MA: D igital Equipment Corpora tion,
1994).
10. W Blume et a!., "Automatic Detection of Paral lelism : A Grand Chal lenge for High-Perfor mance Compu t ing," CSRD Report No. 1 348
(Urbana, IL: Center for Supercompu ting
Research and Development, University of
Illinois at Urbana-Champaign, 1994) .
70
Vol. 6 No. 3 Summer 1.994
Digital Technical journal
I
Further Readings
The
D igital Technical Journal
publishes papers that explore the technological foundations of Digital's major products.
Each Journa lfocuses on at least one product area and presents a compila tion of refereed papers written by the engineers who developed the products. The con tent for the
Jo urnal
is selected by the journal Advisory Board.
Digital enginee1·s who would like to cont1·ibute a paper to the
journal
should contact the edito1· at RDVAX::BLAKE.
Topics covered in pre v i
o
u s issues of the
Digital Technical journal
are as fol l
o
w s:
Alpha
AXP
Partners-Cray, Raytheon,
Kubota/DECchip 21071/21072 PCI Chip
Sets/DLT2000 Tape Drive
Vol. 6, No. 2, Spring 1994, EY-F947E-TJ
High-performance Networking /OpenVMS
AXP
Vol.
System Software/Alpha
AXP
PC Hardware
6, No 1, Winter 1994, EY-QOl l E-TJ
Software Process and Quality
Vol. 5, No. 4, Fall 1993, EY-P920E-DP
Product Internationalization
Vol. 5, No. 3, Summer 1993, EY-P986E-DP
Multimedia/Application Control
Vol. 5, No. 2, Spring 1993, EY-P963E-DP
DECnet Open Networking
Vol. 5, No. 1, Winter 1993, EY-M770E-DP
Alpha AXP Architecture and Systems
Vol. 4, No. 4, Special Issue 1992. EY-.J886E-DP
NV
AX-microprocessor VAX Systems
Vol. 4, No. 3, Summer 1992, EY-j884E-DP
Semiconductor Technologies
Vol. 4, No. 2, Spring 1992, EY-L521 E-DP
PATHWORKS:
PC
Integration Software
Vol. 4, No. 1, Winter 1992, EY-J825E-DP
Image Processing, Video Terminals, and
Printer Technologies
Vol. 3, No. 4, Fall 1991, EY-H889E-DP
Availability
in
VAXcluster Systems/Network
Performance and Adapters
Vol. 3, No. 3, Summer 1991, EY-H890E-DP
Fiber Distributed Data Interface
Vol. 3. No. 2, Spring 1991, EY-H876E-DP
Transaction Processing, Databases, and
Fault-tolerant Systems
Vol. 3, No. I, Winte1· 1991, EY-F588E-DP
VAX 9000 Series
Vol. 2, No. 4, Fall l990, EY-E762E-DP
DECwindows Program
Vol. 2, No. 3, Summer 1990, EY-E756E-DP
VAX 6000 Model 400 System
Vol.
2,
No. 2, Spring 1990, EY-C 197E-DP
Compound Document Architecture
Vol. 2, No. 1, Winter 1990, EY- C 196E-DP
Distributed Systems
Vol. 1, No. 9. ]une 1989, EY-Cl79E-DP
Storage Technology
Vol. 1, No. 8, February 1989, EY-C l66E-DP
CVAX-based Systems
Vol. 1, No. 7, A ugust 1988, EY-6742 E-DP
Software Productivity Tools
Vol. 1, No. 6, February 1988, EY-8259£-DP
VAXcluster Systems
Vol. I, No. 5, September 1987, EY-8258E-DP
VAX 8800 Family
Vol. 1, No. 4, February 1987, EY-67 1 1 E-DP
Networking Products
Vol. 1, No. 3, September 1986, EY-6715 E-DP
MicroVAX
II
System
Vol.
1,
No. 2, March 1986, EY-3474 E-DP
VAX 8600 Processor
Vol.
I,
No 1, August 1985, EY-3435E-DP
Digital Technical journal
VtJI. 6 No
.
.3 Summer 1994 71
Further Readings
Subscriptions and Back Issues
Subscriptions to the Digital Technical journal are available on a prepaid basis. The subscription rare is $40 00 (non-US $60 00) for four issues and $75.00 (non-US $ 1 1 5.00) for eight issues.
Orders shou ld be sent to Cathy Phi l l ips. D igital
Equipment Corporation. 30 Porter Road l.,I02/D 10,
Littleton. Massachusetts 01460, U.S.A .
.
Telephone:
(508) 486-2538, FA X: (508) 486 -2444. Inquiries can be sent electronical ly to [email protected].
Subscriptions must be paid in US dollars, and checks shou ld be made payable to D igital
Equ ipment Corporation.
Single copies and past issues of the Digital
Technical journal are available for $ 16.00 each by call ing DECd irect at 1-800-DJGITAL
(1-800-344-4825). Recent back issues of the journal are available on the Internet at http:// w�w
.d igital.com/info/DTJ/home.btml.
Complete D igital Internet l istings can be obtained by sending an electronic mail message to info@digital .com.
Digital Research Laboratory Reports
Reports publ ished by Digital·s research labora tories can be accessed on the Internet through the World Wide Web or FTP. For access informa tion on the electronic or hard-copy versions of the reports, see http:/ /gatekeeper.dec.com/ hypertext/info/era. reports.html .
Digital Product Information
Readers of the journal can keep up-to-date on
Digital's products and services by subscribing to the Digital Reference Sen •ice. To receive cur rent information on a l l Digital 's products and services on a regu lar basis. contact the Digital
Reference Service, PO. Box 6464, Hol l iston, MA
0 174 6. Withi n the United Stares, call (800) 494-4377 or (508) 429-5515, extension 765. Outside the United
States, cal l (508) 429-30 1 5 or send a facsimile to
(508) 429-692 1 .
Technical Books and Papers by Digital Authors
Applications of Petri Nets in Manufacturing
Systems fl!lodeling, Con trot, ct nd Perforrnance
A na�)'sis, Alan A. Desrochers and Robert Y. Al-jaar,
IEEE Press, New York. 1994
(ISfiN
0-87942-295-5)
This practicaL h ighly special ized book presents theory and examples that clearly show how to use the Petri net approach to model, contro l. and then analyze the performance of these complex systems.
This book a lso brings together newly documented applications of Petri nets in Japan and Europe and makes them available to practitioners worldwide.
From this book. the reader will learn how to model complex manufacturi ng systems using Perri nets; analyze the performance of the overall manu factur ing system in terms of production rates, machine uti I ization, average in-process inventory, and other measures; generate control software from the Petri net model of an au tomated manufacturing system; and synthesize Petri net models for large automated manufacturing systems.
Applications of Petri Nets in Man ufacturing
Systems Modeling. Control, and Perforrncmce
Analysis will be of particular interest to researchers in manufacturing systems engineering and individ uals i nvolved in production planning and control, plant layout and design. and schedul ing of manu facturing operations.
R. Abugov and K. Zinke, "Prioritization of Defect
Reduction Activity by Yield Impact ,' ' 1994 SEN/ICON
Ultraclea n LVJanufacturing Co nference (October
1994)
M. Ackerman and R. Buckland. '' Multiple Matrices for a Marketing
QFo;·
Fiftb S)•mposium on Quality
Function Deployment (June 1994).
M. Ackerman and R. Buckland. "Successful QFD
Appl ication at Digital: Unique Approaches and
Fifth Symposium on Quality Function
Depl oy ment (
.
J une 1994)
H. Al i, J Steele, ). Bosco, and c;. Bartlett, " Electro mechanical Stu ely of 'No-Clean· Flux Corrosivity,"
Proceedings of the Eighth Electronic Materials and Processing Congress (August 1993).
R . Al lmon, "Design of Portable Systems," Pro ceedings of the IEEE Custorn In tegrated Circuits
Conference (May 1994).
P Anick, "Adapting a Full-text Information Retrieval
System to the Computer Trou bleshooting Domain,"
Proceedings of the Set'enteenth A n nual Interna tional ACM-51GIR Conference on Researcb and
Development in Infomwtion Retrieml (July 1994).
72 1-l'!/. 6 .Vo. j Sllll/111<!1' 1991 Digital Technical jo urnal
N. Arora and B. Doyle, "Model ing the 1-V Charac teristics of Fully Depleted SOl MOSFETs Including
Self Heating," IEEE International Silicon -on
Insulator Conference Proceedings (October 1994).
N. Arora, B. Doyle, and D. Krakauer, ·· s P IC E Model and Parameters for Fu lly Depleted SOl MOS F ETs,
Including Self-heating;· IEEE Electron Device Letters
(October 1994)
N. Arora, R. Rios, and C. Huang, " Impact of Poly silicon Depletion Effect on Circuit Performance for 0.35tL CMOS Technology;· Proceedings of the
Twenty-fourth European Solid State Deuice
Research Conference (September 1 994).
D. Bhavsar and ). Edmondson, "Testability Strategy of the Alpha AXP 2 1 164 Microprocessor," Proceed ings of the IEEE International Test Conference
(October 1994).
S. Bilotta and D. Proctor, ·' Development of a Manu facturable Low Pressure
ROX NOX
Oxidation Pro cess." Adl'anced Semiconductor Manufacturing
Conference and Workshop Proceedings
(November 1994)
C. Brench and B. Archambeault, " Proposed Stan dard EMI Modeling Problems." IEEE ln te·rnational
Symposium on Electromagnetic Compatibility
(August 1994).
C. Brench, " Heatsink Radiation as a Function of
Geometry,'' IEEE International Symposium o n
Electromagnetic Compatibility (August 1994).
C. Brench, "Shield Degradation in the Presence of External Conductors,'' IEEE International
Symposium on Electrom.agnetic Compatibility
(August 1994) . j. Clement and A. Enver. "Modeling EJectromigration induced Stress Buildup Due to Nonuniform Tem perature," Materials Reliability in Microelectronics
IV Symposium Proceedings (April 1994).
W Cronin, ]. Hutchison, K . Ramakrishnan, and
H . Yang, "A Comparison of High-speed LANs,"
Proceedings of the IEEE Nineteenth Conference on
Local Computer Networks (October 1994).
W Dubie, " Networds: The Impact of Electronic
Text-Processing Uti l ities on Writing," journal of
Social and Euolutionary Systems (November 1994).
). Edmondson and P Rubinfeld, "An Overview of the 21164 Alpha
AXP
Microprocessor," Hot Chips VI
Symposium (August 1994).
I
T. Fox, "The Design of H igh-Performance Micropro cessors at Digital," Thirty-first Design Automation
Conference Proceedings (June 1994).
]. Grodstein, E. Lehman, H. Harkness, and
W Grundmann, "Optimal Latch Mapping and
Retiming within a Tree," JEEE/ACM Interna tional Conference on Computer-aided Design
(November 1994).
C. Gross, "Method for Selecting Semiconductor
Equipment Using Empowered Teams," Advanced
Semiconductor Manufacturing Conference and
Workshop Proceedings (November 1994).
T. Guay, "CASE-based Reasoning for Knowledge
Acquisition Suggestions," International journal of Artificial Intelligence Tools (July 1994).
S. jong, " Exploring Paths toward Quality Infor mation," Fortyjirst A nnual Society for Technical
Communication (May 1994).
C. juszczak and D. Lebel, ·' NFS Version 3-Design and Implementation," Summer 1994 USENIX
Technical Conference (June 1994).
D. K.rakauer and K. Mistry. "C ircuit Interactions
During Electrostatic Discharge," IEEE Electrical
Ouer Stress/Electrostatic Discharge Symposium
Proceedings (September 1994).
]. Lloyd , "Electromigration Failure of Narrow A I
Alloy Conductors Containing Stress Voids,''
Materials Reliability in Microelectronics IV
Symposium Proceedings (April 1994).
.J. Lloyd, "Electromigration Failure in Thin Film Con ductors," Materials Reliabili�J' in Microelectronics
IV Symposium Proceedings (April 1994).
P Martino and G. Freedman, " Predicting Solder
Joint Shape by Computer Modeling:' Proceedings of the Forty-fourth Electronic Components and
Technology Conference (May 1 994).
T. Moore, "A Test Process Optimization and Cost
Modeling Tool,'' Proceedings of the
IEEE
Interna tional Test Conference (October 1994).
Digital Technical journal VrJI. 6 No
.
.J Summer 1994 73
Further Readings
D. Morin, T. Comard, M . Joshi, and K. Sprague,
"Calculating Error of Measurement on High-speed
Microprocessor Test," Proceedings of the
IEEE
International Test Conference (October 1994).
G. Papadeas and D. Gauthier, "An On-line Data
Collection and Analysis System for VLSI Devices at Wafer Probe and Final Test," Proceedings of the
IEEE International Test Conference (October 1994).
M. Piasecki, K. Orvek, R. jones, and S. Dass, " Deep
UV Technology for 0.35J.Lm Lithography," 1994 IEEE
Lithography Workshop (September 1994).
K. Ramakrishnan and P Biswas, " Performance
Benefits of Nonvolatile Caches in Distributed File
Systems," Concurrency: Practice and Experience
(July 1994).
K. Ramakrishnan and H. Yang, "The Ethernet
Capture Effect: Analysis and Solution," Proceed ings of the
IEEE
Nineteenth Conference on Local
Computer Networks (October 1994).
K. Ramakrishnan and H. Yang, " FIFO Design for a High-speed Network Interface," Proceedings of the
IEEE
Nineteenth Conference on Local
Computer Networks (October 1994)
R. Razdan and K. Brace, " PRJSC Software Accelera tion Techniques,"
IEEE
International Conference on Compute1· Design: VLSI in Computers and
Processors (October 1994).
A. Sathaye, "Appl ication of Supervisor Synthesis for Controlled Time Petri Nets to Real-time Data base Systems,"
(June 1994)
1994 A merican Control Conference
S. Sathaye, "Conventional and Early Token Release
Scheduling Models for the IEEE 802.5 Token Ring," journal of Real-Time Systems (May 1994).
S. Sathaye, "A Real-time Schedu l ing Framework for Packet-switched Networks," Fourteenth Inter national Conference on Distributed Compu ting
Systems (June 1994).
C. Schiebl, "Application of EDX Spectroscopy to Accurate Nondestructive Measurement of
CoSi Film Thicknesses during Semiconductor
Processing," Twenty-eighth Annual Microbeam
Analysis Society Meeting (August 1994).
C. Schiebl, "Continuous Fluorescence Correction
Factor for Layered Specimen, .
.
Twenty-eighth
Annual Microbeam A nalysis Society Meeting
(August 1994).
C. Schiebl, "Secondary Depth Distribution Gener ated by Characteristic Fluorescence in Multilayer
Samples for Use in Quantitative EPMA," Twenty eighth A nnual Micmbeam Analysis Society
Meeting (August 1994).
). Seyyedi, "Soldered joint Reliability for Interstitial
Pin Grid Array Packages," journal of Surface Mount and Related Technologies Group (October 1994).
H. Soleimani, "An Investigation of Phosphorous
Transient Diffusion in Sil icon below the Solid Solu bility Limit and at a Low I mplant Energy," journal of the Electrochemical Society (August 1994).
K. Steeples and D. Chang Kau, " Multiply Charged,
Channeled , Ion I mplantation," Tenth International
Conference on Ion Implantation Technology
(June 1994)
K. Steeples, D. Chang Kau, M. Andreoli , and
Process," Tenth International Conference on
Ion Implantation Technology (June 1994).
N. Sul l ivan and S. Arsenault, "SEM/ EDS Analysis
Method for Bare Silicon Particle Monitor Wafers,"
Advanced Semiconductor Manufacturing
Conference and Workshop Proceedings
( November 1994).
N. Sullivan and R. Newcomb, "Critical Dimension
Measurement in the SEM : Comparison of Backscat tered vs. Secondary Electron Detection," Proceed ings of the International Society of Photo-Optical
Instrumentation Engineers
(SPIE):
Integrated Cir cuit Metrology, Inspection, and Process Control
VIII (February 1994).
B. Thomas, "Open VMS 1/0 Concepts Kernel
Processes;' Digita! Systemsjournal (July 1994).
B. Thomas, "Open
VMS 110
Concepts: Software,"
Digital Systems journal (July 1994)
B. Thomas and K. Morse, "Open VMS AXP 1/0 Con cepts," Digital Systems journal (June 1994).
74 Vol. 6 No. 3 Summer /994 Digital Technical]ourrral
A. Torabi, M . Mallary, and S. Marshall, "The Effect of Rise Time and Field Gradient on Nonl inear Bit
Shift in Thin Film Heads," The Sixth joint
MMM
Intermag Conference (June 1994).
A. Torabi, M. Mallary, S. Marshall, S. Batra, and
S. Ramaswamy, " Performance Evaluation of D iffer ent Pole Geometries in Thin Film Heads," The Sixth joint M M M
-
Inte rm ag Conference (June 1994).
M. Tsuk, " FASTHENRY: A Multipole-accelerated
3-D Inductance Extraction Program," IEEE Trans actions on Microwave Theory and Techniques
(September 1994)
M. Tsuk and R. Evans, " Modeling and Measure ment of the Power Distribution System of a High performance Computer System,"
IEEE
Topical
Meeting on Electrical Performance of Electronic
Packaging (October 1993).
R . Ulichney, " Halftone Characterization in the
Frequency Domain," The Society for Imaging
Science and Technology's (IS&T's) Forty-seventh
Annual Conference (May 1994).
R . Ulichney, "The Void-and-cluster Method for
Dither Array Generation," Proceedings of the
International Society of Photo-Optical Instru mentation Engineers
(SPIE)
(September 1993).
M. Utt, "A System for Discovering Relationships by Feature," Proceedings of the Seventeenth
Annual International ACM-SIGIR Conference on Research and Development in Information
Retrieval (July 1994).
). Vicente, " Network Capacity Planning,"
CMG '93
Conference (December 1993).
A. Villani, "Cohesive Mechanical Behavior of
Adhesive Materials," Proceedings of the 1993 ASME
International Electronics Packaging Conference:
Advances in Electronic Packaging 1993 (October
1993).
). Yang, " Reliability Performance of an R3000-Based
MCM for Desktop Workstations," International Elec tronics Packaging Conference (September 1993).
I
W Zahavi, " Modeling the Performance Budget,"
Computer Measurement Group Proceedings
(CMG '93) (September 1993).
Digital Techntcaljourtwl Vol. 6 No. 3
Summer
1994 75
I
Recent Digital US. Patents
The following patents were recently issued to Digital Equipment Corporation. Titles and names supplied to us by the US. Paten t and Trademark Office are reproduced exact(y as they appear on the original pub lished patent.
4, 592,072 R. E. Stewart
5,210,834 W Beach and ). Zurawski
5,210,874
5,212,650 D. Hooper and S. Kundu
5,212,783
5,214,770
5.216,413
5,230,071
5,230,079
B. Newman
R. Grondalski
P Bannon,
L. Seiler, ). Pappas, and
R. Rose
5,218,684 D. Hayes and V Triolo
5,218,712 D. Bhandarkar,
D. Cutler, D. Orbits. and
R. Witek
W Cardoza,
5,220,468 M. Sidman
5,220,674 W Morgan, D. Cobb, G Bel l . and A . Carlson
5,221 ,422
5,222,029
S. Das and ). Khan
D. Hooper and S. Kundu
5,222,223 R. Hetherington, D. Webb,
T. Fossum, ). Murray, and
D. Manley
5,222,224 S. Arnold, S. Delahunt,
M . Flynn, T. Fossum ,
R . Hetherington , and
D. Webb
5,226, 170 P Rubinfeld
5,228, 129
5,230,067
5,230,072
P Karger
S. Sherman
R. Ramanujan, and S. Steely
S. Bryant and M .
B . Buch
Harwood
D. Smith and K. O ' Rourke
Decoder for Self-Clocking Serial Data Communications
High Speed Transfer of I nstructions from a Master to a Slave Process
Cross-Domain Ca l l System in a Capabil ity Based Digital
Data Processing System
Procedure and Data Structure for Synthesis and Trans formation of Logic C i rcuit Designs
System Which Directional ly Sums Signals for Identifying and Resolving Tim ing Inconsistencies
System for Flushing Instruction-Cache Only When
Instruction-Cache Address and Data-Cache Address Are
Matched and the Execution of a Return-from-Execution or-Interrupt Command
Apparatus and Method for Specifying Windows with Priority
Ordered Rectangles in a Computer Video Graphics System
Memory Configuration System
Providing a Data Processor with a User-mode Accessible
Mode of Operations in Which the Processor Performs
Processing Operations without Interruption
Disk Drive with Constant Bandwidth Automatic Gain Control
Local Area Print Server for Requesting and Storing Required
Resource Data and Forwarding Printer Status Message to
Selected Destination
Lithographic Technique Using Laser Scanning for Fabrication of Electronic Components and the Like
Bitwise Implementation Mechanism for a C i rcuit Design
Synthesis Procedure
Method and Apparatus for Ordering and Queueing M u ltiple
Memory Requests
Scheme for Insuring Data Consistency between
System a
Plurality of Cache Memories and the Main Memory in a Multiprocessor
Interface between Processor and Special Instruction
Processor in Digital Data Processing System
Synchronous Communication Interface for Reducing the
Effect of Data Processor Latency
Bus Control C ircuit for Latching and Maintaining Data
Independently of Timing Event on the Bus Until New
Data Is D riven Onto
Method of Control I ing the Variable Baucl Rate of Peripheral
Devices
System for Managing Hierarchical Information in a Digital
Data Processing System
Massively ParalleJ Array Processing System with Processors
Selectively Accessing Memory Module Locations Using Address in Microword or in Address Register
76 Vol. 6 No
. .
i Su/1/mer 19')4 Digital Technical journal
5,235,693
5,237 ,574
5,239,635
5, 247,398
5,249, 187
5, 249,293
5,251 ,322
5,255,367
5,261 , 1 13
5,266,409
5, 267, 175
5, 276,892
5,278,840
5,280,617
5, 287.463
5,291 , 581
5, 297,283
5, 301 ,329
5, 303,380
5, 305,462
5,313, 467
5,317,7 17
]. Lynch, K. Chinnaswamy,
P Goodwin, ]. Tcssari, and
M. Gagl iardo
L Weng
R.E. Stewart, T E. Leonard, and ST Lee
M. Sidman
W Bruckert and T Bissett
B. Schreiber, C. Cockcroft,
M. Ozur, R. Bismouth, and
D. Doherty
P Doy le, ]. El lenberger,
E. jones, D. Carver, S. D iPirro,
B. Gerovac, W Armstrong,
E. Gibson, R. Shapiro,
K. Rushforth, and WC. Roach
W Bruckert, T Bissett,
D. Mazur, and ]. Mu nzer
N. P jouppi
Method and Apparatus for Red ucing Bu ffer Storage in a Read-Modify-Write Operation
Error-resilient I nformation Encoding
Virtual Address to Physical Address Translation Using Page
Tables in Virtual Memory
Automatic Correction of Position Demodulator Offsets
Dual Rail Processors with Error Checking on l/0 Reads
Computer Network Providing Transparent Operation on a Compute Server and Associated Method
Method of Operating a Computer Graphics System Includ ing
Asynchronously Traversing Its Nodes
P H . Schmidt and j.C. Angus
D. Hooper
AS Olesin and R . M . Supnik
Fau l t Tolerant, Synchronized Twin Computer System with
Error Checking of
1/0
Communication
Apparatus and Method for a Single Operand Register Array for Vector and Scalar Data Processing Operations
Hydrogenated Carbon Compositions
Database Access Mechanism for Ru les Utilized by a Synthesis
Procedure for Logic C ircuit Design
Destination Control Logic for Arithmetic and Logic Unit for
D igital Data Processor
Apparatus and Method for Data Induced Condition Signall ing I) Bhandarkar, W Cardoza,
D. Cutler, D. Orbits, and
R. Witek
R.F. Brender and B.R. Brett
RC Frame and FA. Zayas
D. Bhandarkar, W Cardoza,
D. Cu tler, D. Orbits, and
R. Witek
D. Cut ler, J Kel ly, and
F. Perazzoli
RC Frame and FA. Zayas
B. Foster, G. Brown, J Piazza,
]. Tenny, B. Nel son,
W Van Roggen, and
P Anagnostopoulos
R. Grondalski
Automatic Program Code Generation in a Compiler System for an Instant iation of a Generic Program Structure and Based on
Forma l Parameters and Characteristics of Actual Parameters
Method and Apparatus for Transferring Information over a Common Para l lel Bus Using a Fixed Sequence of Bus
Phase Transitions
Apparatus and Method fo r Synchronization of Access to Main Memory Signal Groups in a Multiprocessor Data
Processing System
Object Transferring System and Method in an Object Based
Computer Operating System
Double Unequal Bus Timeout
System for Processing Data to Facil itate the Creation of Executable Im ages
G. Varghese, M . Fine, A. Smith, and R. Szmauz
D. Bhandarkar, W Cardoza,
D. Cut ler, D. Orbits, and
R. Witek
Mechanism for Broadcasting Data in a Massively Parallel
Array Processing System
Integrated Communication Link Having Dynamical ly
Allocatable Bandwidth and Protocol for Transmission of Allocation Information over the Link
Apparatus and Method fo r Main Memory Unit Protection
Using Access and Fault Logic Signals
I
Digital Techuical journal
Vol. 6 No 3 Summer 1994 77
Call for Authors from Digital Press
Digital Press has become an imprint of Butterworth-Heinemann, a major inter national publisher of professional books and a member of the Reed Elsevier group. Digital Press remains the authorized publisher for Digital Equipment
Corporation: the two companies are working in partnership to identify and pub lish new books under the Digital Press imprint and create opportunities for authors to publish their work.
Digital Press remains committed to publishing high-quality books on a wide variety of subjects. We would like to hear from you if you are writing or thinking about writing a book.
Contact: Frank Satlow
Publisher
Digital Press
313 Washington Street
Newton, MA 02158
Tel: (617) 928-2649
Fax: (617) 928-2640 [email protected]
ISSN 0898-901X
Prin� in U.S.A. BY.S7991!-TJ/9j 01 14 14.� Copyright 0 Digital Equipment Corporation. All Rights Reserved.
Download
Advertisement
Key features
Multiprocessing architecture
PC-style I/O subsystem
Support for multiple generations of Alpha processors
High performance
Support for Windows NT, DEC OSF/1, and OpenVMS operating systems
Flexible configuration
High availability
Cost-effective
Scalable
High performance I/O
Frequently asked questions
Both systems implement a PC-style I/O subsystem that uses both PCI and EISA buses. They are designed to be general-purpose servers that can run a variaty of applications. They support multiple generations of Alpha processors. The systems are designed to be highly scalable and reliable.
Both systems support Windows NT, DEC OSF/1, and OpenVMS operating systems.
These systems leverage a multiprocessing architecture to achieve high performance. They support multiple processors and are designed for high-performance computing tasks.