Digital VAX 6000 Model 400 VAX 6000 system, VAX 6000 Model 400 VAXvector 6000 processor, 3100 DECstation Technical Journal
விளம்பரம்
விளம்பரம்
VAX 6000 Model 400 System
Digital Technical Journal
Digital Equipment Corporation
Volume 2 Number 2
Spring 1990
Cover Design
Our cover depicts some of the common equations and terminology used in vector processing, which is one of the featured topics in this issue. The v.-IXl'ector processor extends the VAX
6000 family to address the compul ing need� (if numerically intensiue applications.
The VAX
6000
Model
•iOO systern is a bright star in Digital's family of midrange multiprocessors and this issue's main product thenU!.
The couer was des(rt,ned by David Comberg and Randy Ziegler of the
C01porate Design Group.
Editorial
Ja n e C. Blake. Editor
Rarhara Lindmark, Associate Ediwr
R i char d
W.
H c ane, M ana g i n g Editor
Circulation
Catherine M. Phillips, Administrator
Suzanne
,I. Babineau.
Sec r e tar y
Production
Hden
L. l'allcrson. Production Edi to r
N a n cr Jones. T ypo gr a p h e r
Rebecca A.
Barker, Typographer l'ct<:r
Woodhury,
Illustrator �nd De sign er
Advisory Board
S:unud H. F u ller
.
Chairman
Robert M. Glorioso
John
W.
M cCr e di e
Mahendra R. Patel
F
(;rant Saviers
Robt:rt K. S p it z
William D. Strecker
Victor A. Vyssotsky
The Ui!iila/ Tecbnica/joumal is p uh li she d quart e r ly by D i g i t a l
Equipment Corporation,
J46
M:tin Street MLOI-5JB68. Maymrd.
M as sa chus e tt s
01754-2571.
Su b sc ri p t i on s
10 the Journal are
S·iO.OO for four issues and must he prepaid in li.S. funds. Univer si t y and college p r o f e s sor s and Ph. 0. stu d e nt s in the electrical engin t: e ring an d c o m pu t e r sc ien ce f i el ds receive complimen tarr su h scr ip t io ns upon request. Orders. inquiries. and a ddre s s ch a n ge s should be sent to The Di!!ita/'lecbnical}oumal at the publ i s h e d
b y address. Inquiries on also be se nt e l e ctro ni cally on
NI:ARNET to [email protected]. S i n g l e copies and back issues are
3vaibhk for S 16.00 each from Digit:! I P re s s of Digital Equipmc:nt
Corporation, 12 C r o sb y
Drive, Hcdlord. MA
01-_�0- 149.'1.
D i g it a l em p l o y e e s may send s u b sc r i p t io n orders on the ENET 10
RDVAX JOliRNAL or by inter o f f i c e mail to ma ilst o p MLO 1- 5/ll68.
Orders sh o uld i nc lu d e ba d g e number, cost center, site location code and address, and group name. l l s. e n g in e er s in Enginen ing an d Ma nu f ac turing receive complimt:mary s u bscripti ons; enginc:er� in tht:se organizations in countries outside the: lJ.S. sho u l d com act the journal office to receive: their complimemary subscr i ptions
.
All e m pl oyee s must advise of changes of address.
Commt·nts on the content of any paper are wdcomed and may be sem
10 th e editor at the p u b l ish e d
b\
' or nc:t work address.
Copyright' 1990 Digital E qui p m ent C o rp orati o n
.
C opy ing without fcc is p erm i tJ e d p ro v id e d that such copies are made for use in educational institutions by faculty member s and are not disJributed for comml'rcial ad,
· a nt a g e
.
A bs tra c tin g "'ith credit of Digital Equ i pm e nt
C orporati on
' s a u th o r sh ip is permitted
.
All rights r e s e rv e d
.
The information in this journal is s u h j e c t 10 change without no tice and should not be construed as a commitment by
Digital
Equi p m e nt Corporation. Di g ital Equipmem Corporation assumc:s no res po ns ib i l ity for any errors that may appear in this journal.
ISSN 0
898
-
9 0
I X
Documentation Number EY-<:1971:-0P
The f o l lowi ng are trademarks of D i g it a l Equipment Corpora tion: CJ, O EC n e t , DECst:uion .'1100, OH:wimlows, Digital, the
Di g i tal logo, HSC. Mi c r o VAX
VAX 6000, VAX
8700,
VAX
, llZ'i'),
Thi n
W i r e , VAX, VAX-11/780.
HHOO,
VAX 9000, VAXBJ, VAXcluster,
VAX FORTRAN, VAXveclOr(J000, V.\1S, !ILTRtX, XMI.
!IN IX is a reg i s ter ed trademark of American T e lt: p hon e
Telegraph Compa ny
.
& dbx is a r e g i s tere d trademark of dbx, Inc
,\Ill'S is a trademark of MIPS Compu te r S y s t e ms
. I n c .
Book production was d on e by D ig it al
' s Educational
Media Comm u nic a ti on s Group in Hedfortl, MA.
Services
I
Contents
9
Foreword
Pauline A. Nist
VAX 6000 Model400 System
11
Vector Processing on the
VAX vector 6000 Model400
Debra L. Slater, David M. Fenwick, D. John Shakshober, and Douglas D. Williams
27
The
VAX
6000 Model400 Scalar Processor Module
Patrick Sullivan, Michael A. Callander, Sr. ,James R. Lundberg, Rebecca L. Stamm, and William). Bowhill
36
An Overview of the
VAX
6000 Model400 Chip Set
W. Hugh Durdan, William). Bowhiii,John
F
Brown, William V. Herrick,
Richard C. Marcello, Sridhar Samudrala, G. Michael Uhler, and Nicholas Wade
52 VAX
6000 Model400 Physical Technology john T. Bartoszek, Robert). Hannemann, Stephen P. Hansen, Robert]. McCarty, and john C. Sweeney
64 VAX
6000 Model 400 CPU Chip Set Functional Design Verification
Richard
E.
Calcagni and Will Sherwood
73
Test and Qualification of the
VAX
6000 Model400 System
John
W
Croll, Larry T. Camilli, and Anthony]. Vaccaro
84
Development of the DECstation 3100
Thomas C. Furlong, Michael). K . Nielsen, and Neil C. Wilhelm
89
Compiler Optimization in RISC Syst ems
Larry B. Weber
2
I
Editor's Introduction
jane
C. Blake
Editor
This Spring 1990 issue marks the second issue to be published on the new quarterly schedule of the
Digital Technicaljournal.
This is also the ftrst year that the Journal is available by subscription- a service our readers have asked for and which we are glad to be able to offer.
The journal will continue to focus each issue on a product theme. In fact, two products are featured in this issue. The main theme is the latest addition to the V�'l: 6000 family, the Model400. With its multi processing capabilities, this midrange family of systems provides a highly configurable and expand able computing environment. Because the same cabinet, buses, and power systems are used by aU family members, systems can easily be upgraded to achieve higher levels of performance. Papers in this issue describe VAX 6000 Model 400 innovations and additions, including a new vector processor and a higher performance scalar processor module, chip set design and verification, physical technology advances, and system test. The second theme com prises two papers related to Digital's workstation development, specifically the DECstation 3100, and compiler optin1ization in RISC systems.
Opening this issue is a paper on one of Digital's first vector processors. Dave Fenwick, john
Shakshober, Debra Slater, and Doug Williams review the design alternatives for the VAXvector
6000 Model 400 processor and describe its function units. They then give examples of how the units combine to deliver high performance for computa tionally i ntensive applications.
The Model 400 also has a new scalar processor, with nearly twice the performance of its prede cessor, the Model 300. In their paper, Pat Sullivan,
Mike Callander, Jim Lundberg, Rebecca Stamm, and
Bill Bowhill discuss the module design and give particulars on how difficult signal integrity prob lems were resolved.
The five system chips that reside on the module are the topic of our next paper by Hugh Durdan, Bill
Bowhill, John Brown, Bill Hl:rrick, Rich Marcello,
Sri Samudrala, Mike Uhler, and Nick Wade. From their d iscussions of the chip designs, we learn how the best features of the VAX 8700 ECL-based, pipelined system and of previous VLSJ designs were incorporated i n the chip set, which achieves a cycle tinle of
28 nanoseconds.
This fast cycle time was one of several require ments that drove a significant design effort for the physical technology. Joh n B a rtoszek, Rob
Hannemann , Steve Hansen, Bob McCarty, and
John Sweeney describe the technological advances achieved in a number of areas, including tape automated bonding, semicustomized ceramic single-chip package design, and testability.
The two papers that close this collection of papers on the VAX 6000 Model 400 address chip design verification and system test. Rick Calcagni and
Will Sherwood explain the engineers' multipronged approach to design verification, an approach neces sitated by the complexity of the chip set. Then,
John Croll , larry Camilli, and Tony Vaccaro present a paper on the methods and tools designed to com pletely test the i nteraction of VAX 6000 Model 400 system's hardware and software.
In the last rwo papers, the topic turns to work stations. Tom F u rlong, Mike N ielsen, and Neil
Wi.lhelm provide an overview of the successful project undertaken to bui.ld a fast, competitively priced, RJSC-based, ULTRIX workstation, called the
DECstation 3 100. The Journal is fortunate also to have a related paper on compiler optimization in
RISC systems by Larry Weber, vice president, MIPS
Systems, I nc. MIPS Systems built the RJSC chip set incorporated in the DECstation 3100 workstation.
I thank Steve Holmes of the Midrange Systems
Business Group for his help in selecting topics on the Model 400, and Gillian Scholes of Digital and joanne Hasegawa of MIPS Systems, Inc., for their help in obtaining the workstation and RISC papers in this issue.
Biographies
I
John T. Bartoszek john Bartoszek currently manages the Physical Technology
Group within the SDE that is responsible for physical technology applications and product designs. John previously managed the PTG physical technology pro gram that spawned the physical technology used on the VAX 6000 Model 400
CPU module. He joined Digital in 1981 and has a B.S. in nuclear engineering from
Lowell Technological Institute. He holds two patents for thermal control devices. john has authored several papers on spacecraft thermal control, solar energy, and electronics cooling and interconnect technologies.
William). Bowhill
As a principal engineer with the Semiconductor Engineer ing Group, William Bowhill is project leader for an engineering team that is designing an execution unit section of a large CMOS-based microprocessor. He has applied for two patents for his design work on the vector interface and backup control chip on the VAX 6000 Model 400 system. Bill also holds a patent for his work in relation to the Model 400's floating poinr accelerator chip. He joined Digital in 1985. Bill was educated in Great Britain and received a B.Eng.
(honors) in electronic engineering from Liverpool University.
John F. Brown
After receiving an M. S.E.E. from Cornell University in 1980,
John Brown joined Digital's engineering staff. At present, he is a principal engi neer and design team manager for the instruction decode section of the next generation CMOS-based VAX microprocessor. John's previous responsibilities include technical contributions to both the VAX 6000 Model 400 and the Model
200 chip sets. He was also the hardware engineer for the extended floating point data type enhancement to the VAX-11/780 system. John currently holds one patent, has two patent applications pending, and has authored technical papers for several publications.
Richard E. Calcagni A member of the Semiconductor Engineering Group's
VLSI microprocessor verification group, Richard Calcagni has contributed to the design of several VLSI microprocessors in the areas of microcode, verification, and prototype system debugging. He has published several papers on CPU design, modeling, and verification. Prior to his work with microprocessors, he worked on module test process development for Digital's Customer Services
Manufacturing organization. Before joining Digital in 1979, Rick worked for
Burroughs Corporation. He received a B.S. (1976) in electrical engineering from the University of Rhode Island.
Michael A. Callander, Sr. Michael Callander is a principal engineer in Digital's
Semiconductor Engineering Group. At present, he is responsible for the archi tecture of a future VAX CPU module. Mike led the VAX 6000 Model 400 system's
REXMI chip set design project. His previous experience with Digital includes the design and verification of the CPU module for the VAX 8200 system and the VAX 8300 system. Mike received his B.S.E.E. degree from the University of
Massachusetts in 1982. He joined Digital upon graduation.
3
Biographies
Larry T. Camilli Since joining Digital in 1979, Larry Camilli has been a member of several product development projects. He is currently a software cnginL"ering supervisor and project leader for the architecture verification software development project, which focuses on the development and maintenance of test software for current and future architectures. Larry's previous responsi bilities include the development of components for a microcode compiler, and a software trace and analysis package. He holds a B.S.E.E. from Clarkson
University and is a student in the M.S.C .S. degree program at Worcester Poly technical Institute. john Croll John Croll is a principal software engineer in the Midrange Systems
Engineering Group. In this position, he is responsible for developing systems test tools for a future hardware product. John was a team leader in tht: Systems
Integration Group for the VAX 6000 Model 400 system. He was also a project leader for the development of the VAX 6000 Model 400 console software. John joined Digital in 1978, and his previous responsibilities include the development of device drivers and other system software. He received a B.S. E. E. ( 1978) from
Drexel University and is a member of
IEEE and ACM .
W. Hugh Durdan A graduate of the Rensselaer Polytechnical Institute, Hugh
Durdan is a consulting engineer in the Semiconductor Engineering Group. In this position, his major responsibility is management of chip development. Hugh became involved in the VAX 6000 Model 400's chip design at its earliest begin nings in 1984 . He led the behavioral modeling effort and the cache controller chip design project. He also managed the development of the custom chip set for the VAX vector procL"ssor and aFChitected and specified the vector interface bus.
Hugh joined Digital in 19HO and worked on the chip design of the VAX 8200 processor.
David M . Fenwick The architect of the VAX vector 6000 Model 400 processor,
David Fenwick joined Digital's United Kingdom office in 1980 and transferred to the Cnited StatL"s in 1985. He is a principal engineer for the Low End Midrange
Systems Group. His previous experience with Digital includes field service regional support in the 'nited Kingdom and project leadership for the DMB32 communications controller for the European engineering office. He also worked on the VAXBI and
XMI programs in the United States. Dave received a
B.Sc.
(Honors) from Loughborough University of Technology in England.
Thomas C. Furlong The development of Digital's RISC-based workstations is the responsibility of Engineering Manager Thomas Furlong. It was Tom's Palo
Alto-based design group that brought the MIPS RISC technology into Digital and developed the OECstation 3100 workstation . Tom has been with Digital for ten years. In addit·ion to his work with RISC-based workstations, he was a member of the start-up te:tm for the VAXstation group. Tom is originally from Detroit,
Michigan, and earned a B.S.E.E. from Michigan State University. He holds two patents related to graphics workstation design and has three patents pending on the next-generation products.
4
Robert J. Hannemann
Robert Hannemann is group manager of the
SCIT
Design and Engineering Physical Technology Group. His group is responsible for the delivery of
IC packaging, module, physical design and test technology for Digital's microprocessor-based systems. A senior consultant engineer, Rob joined Digital in 1978. His prior experience includes positions at Bell Telephone
Laboratories and the University of Maryland, where he was a member of the faculty. Rob holds the Sc.D. degree in mechanical engineering from MIT. He also holds two patents and has published several papers on heat transfer engineering and electronics packaging.
Stephen P. Hansen
Senior Manager Stephen Hansen manages the SDE/PTG
Technical Office. In this position, he coordinates programs related to cost effective packaging for future CMOS-based products, defines technical solutions for the next generation of semiconductor packaging, and provides technical coordination for external packaging and interconnect-related activities. In his twelve years with Digital, Steve has developed several packages and assembly processes for internally manufactured CMOS products, including the tape automated bonding process. He holds two patents in the areas of tape-automated bonding and packaging.
William V. Herrick
Senior Consultant Engineer William Herrick is currently managing a new generation VAX chip design project in the Semiconductor
Engineering Group. He joined Digital in 1977 and has been a member of many
ZMOS- and CMOS-based product development projects, including the PDP-11123 system and the
VAX
8200 system. Before coming to Digital, Bill worked for
Raytheon and
GTE
Sylvania. He has coauthored several papers on solid-state physics and
MOS chip design. Bill received a
B.S.E.E.
( 1969, magna cum laude) from Tufts University, and an
S.M.E.E.
(1971) and
E.E.
(1971) from MIT and is a member of Tau Beta Pi, Eta Kappa Nu, and Sigma Xi.
James R. Lundberg
James Lundberg joined Digital in 1985. Initially, he was a product engineer with the MOS Product Engineering Group and worked on many projects, including
BIIC and
CQBIC.
Jim is currently a senior engineer in
Digital's Semiconductor Engineering Group. He was responsible for the signal integrity on the VAX 6000 Model 400 system and is now working on the signal integrity for an advanced CMOS CPU chip set and module. Before coming to
Digital, Jim operated his own business. He is a member of Tau Beta Pi and Phi
Kappa Phi. He received a
B.S.E.E.
(1985, honors) from the University of Illinois.
Richard C. Marcello A contributor to the design of the VAX 6000 Model 400 system, Richard Marcello is currently an engineering manager for a new generation VAX chip design project. Rich was involved with reliability analysis of semiconductor devices before moving into design five years ago. He worked for
Fairchild Semiconductor before joining Digital in 1981 . Rich coauthored the paper "System, Process and Design Implications of a Reduced Supply Voltage
Microprocessor," which he recently presented at the
ISSCC .
He received
B.B.A. and B.S.E.E. (1980) degrees from the University of Notre Dame and a M.S.C.S.
(1985) degree from Boston University.
I
5
Biographies
Robert j. McCarty
Since joining Digital in 1982 , Robert McCarty has hn:n involved in several major product development efforts. He managed the
VAX 6000 Model 400 system's physical design project and M31 console and instrumentation module development, and led the PDP-11184 system engineer ing project. Before coming to Digital, Bob worked for AM International as the project leader for a laser-based document printer that was part of a document communication system. He holds a B.S. in electrical engineering from the
University of Michigan and an M.B.A. in marketing from the University of
Chicago. He is a member of Tau Beta Pi and Eta Kappa Nu.
Michael j. K. Nielsen
A consultant engineer in the Workstation Systems
Engineering Group, Michael Nielsen is presently completing his responsibilities as architect and chief designer of the DECstation 5000 Model 200 workstation base platform. Mike joined Digital in 1 984. Among the many projects he has worked on since that time are the DECstation 3100 workstation, for which he was the architect and chief designer, and the VAXstation 3520/3540, for which he was a member of the architecture and design team. Mike holds B.S.E.E., M.S.E.E. , and Ph . D.E.E. degrees from Stanford University. He is a member of Tau Beta Pi and Phi Beta Kappa.
Sridhar Samudrala
Currently acting as project leader for a floating point unit, Sridhar Samudrala is a principal hardware engineer in the Semiconductor
Engineering Group. Sri joined Digital in 1977. Since that time, he has worked on testing and diagnostics, as well as the VAX 8200 system microcode, and floating point architecture and design. He holds two patents for his work in floating point design. Sri has an M.Sc. (Technology) from Andhra University in India and an M.S. E. E. from the University of Wisconsin.
D. john Shaks hober
John Shakshober is a senior hardware engineer in the
Low End Midrange Systems Group. Previously involved in the hardware design of M31, a VAX parallel processor, John is now a member of the VAX 6000
Model 400 hardware group, where his particular focus is vector processor design. He joined Digital in 1984 after receiving a B.S. in computer engineering from the Rochester Institute of Technology. John received a M.S.E.E. from
Cornell University in 1988. He is a member of IEEE and Tau Beta Pi. John's latest published technical paper, "Parallel Algorithms for Super Performance," was presented at SuperComputing 89.
Will Sherwood
As a software consulting engineer, Will Sherwood manages the Semiconductor Engineering Group's
VLSI microprocessor verification group.
Previously, he managed the DECSlM logic simulation group. Will joined Digital in
1975 after receiving B.S.E.E. and M.S.E.E. degrees from Carnegie-Mellon
University. In addition to his contribution to the
Digital Technicaljoumal, he has published 15 technical papers and is a contributing author to three books.
Will is a member of the I FlPS 10.2 working group on computer hardware descrip tion languages and has served as publicity chairman and program committee member for several international conferences.
6
Debra L. Slater
In her position as principal software engineer, Debra Slater leads a group that provides performance modeling and analysis support to hard ware development teams. She and her group were integral members of the
VAX
vector 6000 Model400 vector processor design team. Prior to joining Digital in 1987, Debra worked for the Montreal Engineering Company, initially as a pro grammer/analyst and later as an independent consultant. She received a B.Sc. in mathematics and computer science in 1980 from Bishop's University in Quebec.
In 1981, Debra received a master's degree in applied mathematics from the
University of Waterloo in Ontario .
Rebecca L. Stanun
Rebecca St amm is a principal hardware engineer in the
Semiconductor Engineering Group. She is currently leading the design of the backup cache, bus interface, and pin bus for a new VAX CPU chip. Rebecca was the architect of the backup cache controller chip for the
VAX
6000 Model 400 system. She has also worked on design and verification of a RISC microprocessor.
Rebecca joined Digital in 1983. She is a member of Eta Kappa Nu and I EEE, holds one patent, and has coauthored several technical papers for the ISSCC. She received a
B.A.
(1979) in history from Swarthmore College and a
B.S.E.E.
( 1983) from
M IT.
Patrick Sullivan
The project leader for the
VAX
6000 Model400 CPU module development, Patrick Sullivan is a hardware consultant engineer in Digital's
Semiconductor Engineering Group. Pat is now managing a new CPU module development project. Before his work on the
VAX
6000 Model 400, he led the group effort that brought
MCA emitter-coupled logic (ECL) into the corporation.
He is also responsible for the development of a number of Digital's main memory products and participated in the development of several 36-bit CPUs. Pat holds a patent for a memory controller interface. He received his
B.S. and
M.S. degrees from Northeastern University. john C. Sweeney Currently a principal engineer, John Sweeney is working on the testability and test process development for a future Digital product. His previous experience includes being part of the test process development for the
VAX
8600 and
VAX
8800 systems . .John was an application engineer for
GENRAD before joining Digital in 1981. He has authored several papers on boundary scan and fault isolation, and has one patent pending in relation to the
VAX
6000 Model
400 system's test structures. john received a
B. S. E. E.
(1980) from Rensselaer Poly technical Institute and has taken graduate courses at Northeastern University.
G. Michael Uhler
G. M ichael Uhler is a consulting engineer in the Semi conductor Engineering Group, where he is currently leading the architectural definition for a new CPU. As the CPU architect for the
VAX
6000 Model 400 system, M ike was responsible for the CPU architecture, performance evaluation, and CPU microcode. Since joining Digital in 1978, he has also worked on the development of symmetric multiprocessing in the TOPS-10 operating system, and on the microcode and hardware development for PDP-10 CPUs. Mike received a
B.S.E.E. (1975) and M.S.C.S. ( 1977) from the University of Arizona and is a member of
I EEE, ACM,
Tau Beta Pi, and Phi Kappa Phi.
I
7
Biographies
Anthony J. Vaccaro
A senior engineer in the Midrange Systems Evaluation
Group, Anthony Vaccaro is at present responsible for several evaluation projects.
These projects include the FV64A VAX 6000 Model 400 vector processor and
KDM70 mass storage controller. Tony joined Digital in 1976 as a field engineer.
Some of his earlier project responsibilities include product evaluations for the
CIBCA-B VAXcluster Cl adapter and KA825 VAX processor. He was also a member of the team that developed a certification process for new VAXBI adapters. Tony holds a B.S. (1975, cum laude) from Suffolk University and is studying for an
M.S.C.S. at Rivier College.
Nicholas Wade
The implementation of the backup cache control for the next generation VAX CPU chip is being led by Nicholas Wade. Nick is a senior engineer in the Semiconductor Engineering Group. He joined Digital in 1986 and has worked on several projects, including the VAX 6000 Model 400 chip set. He performed the engineering evaluation and debugging for the system support chip on the VAX 3500 and VAX 6000 Model 200 systems. Nick was also a member of the behavioral design and implementation feasibility project for a CPU-XMI interface. He holds B.S. (1985) and M.S. (1986) degrees from Cornell University and is a member of IEEE.
Larry B. �ber
As vice president of software development for MIPS Computer
Systems, Inc., Larry Weber is responsible for the development, quality assurance, and integration of all systems software products. Larry is one of M IPS first employees, having joined the company in 1984. Prior to joining MIPS, he worked for Dialogic Systems and for IBM. Larry helped develop a PASCAL compiler for both the IBM mainframe and IBM RISC project. He has authored and coauthored a number of articles on compilers and languages. Larry holds a B.S. in mathematics from the State University of New York and an M.S. in computer science from the
University of Colorado.
Neil C. Wilhelm
Neil Wilhelm, a senior consultant engineer, is responsible for the development and maintenance of the Workstation Systems Engineering
Group's CAD system and the design engineering of a low-cost workstation. Neil also designed Digital's first RISC-based system. Neil brought an extensive tech nical background to Digital when he joined the company in 1982. He has worked for Hewlett-Packard and Xerox Corporation, founded Ridge Computers, and taught at the University of Rochester. Neil holds a B.S. (1970) in engineering from
Harvey Mudd College, and an M.S. (1971) and a Ph . D. (1973) in electrical engi neering from Stanford University.
Douglas D. Williams MIT graduate Douglas Williams is a principal engineer in the Midrange Systems Engineering Group. He worked on the architectural definition of the VAXvector 6000 Model 400 processor. He also supervised performance modeling and vector control chip development efforts for that pro cessor. Among the many other projects on which Doug has worked since joining
Digital in 1981 are the RISC processor development, memory interconnect design, and V LSI design. He holds an S.B. and S.M. in electrical engineering and is a member of Eta Kappa Nu and Tau Beta Pi. Doug holds a number of patents and has several patent applications pending.
8
I
Foreword
Pauline A. Nist
Group Engineering Manager
Midrange Systems Business
Because microprocessor-based computer systems are complex, the work to design and architect cus tom chips must be initiated long before module and systems work begins.
Looking back at recent history, Digital intro· duced the VAX 6000 family of computers in April
1988 with the Model 200 series, which utilized the ftrst generation of the CMOS-based VAX micropro cessor. The Model 200 was fabricated in Digital's
CMOS-1 (complementary metal oxide semiconduc tor) process. A single-processor Model 210 pro vided 2.8 times the performance of a VAX-11/780 system. One to four processor configurations pro vided up to a total of 1 1 times the performance of a
VAX-11/780 system. (Papers discuss these chips and systems in the August 1988 issue of this journal.)
In January 1989, Digital introduced the second generation of the VAX 6000 family, the Model 300 series. The Model 300 increased single-processor performance from 2.8 to 3.8 times the VAX-11/780 system and total performance for a six-processor system to 22 times the VAX-11/780 system. The
30 percent increase in single-processor perfor mance was made possible by a direct shrink of the die from Digital's 2.0 micron CMOS-1 process to
Digital's
1.5
micron CMOS-2 process. The new pro cess supported a 25 percent reduction in lateral and key vertical dimensions and a 78 percent improve ment in circuit density. Together, these changes improved chip performance by approximately
30 percent.
However, a simple shrink of the existing die did not permit full exploitation of the new circuit den sity. Newly architected and designed parts had to be tailored to take full advantage of the density and speed available with the CMOS-2 process.
The VAX 6000 Model 400 series was formally introduced in July 1989, a mere SLX months after the
Model 300 was introduced. The Model 400 utilized this newly architected chip set to provide single processor performance of 7 times that of the
VAX-11/780 system and up to 36 times the
VAX-11/780 system for six-processor systems. The performance of the Model 400 was over twice the single-processor performance and more than three times the multiprocessor performance of the
Model 200 series, which had been announced only a short
15
months earlier.
To support such aggressive product introduction cycles, advanced development work on the new generation of CMOS-2 chips began in mid-1984, within months of the start of the CMOS-I designs.
Actual design work began approximately a year later. During this period, Digital made a major deci sion to formally extend the VAX architecture to incorporate full support for vector processing into the base instruction set for aU future VAX proces sors. To provide this support, the chip designs already under way had to be modified to incorpo rate the new instructions.
The scalar chip set developed consists of ftve cus tom VLSI parts. They are the
CPU chip, the floating point accelerator chip, the vector/cache controller chip, the system support chip, and the clock chip.
The development of the custom chips required a team of over 40 people, including logic, circuit, and layout designers, and verification engineers. The scalar CPU module design, standard cell interface design, and associated verification team comprised an additional eight engineers. Additionally, the new vector coprocessor module required three new cus tom parts, a new gate array, and a separate module design effort.
Since the direct shrink of die from the CMOS-1 to
CMOS-2 process would account for only a 30 per cent performance increase, the processor architec ture had to be substantially changed to achieve more aggressive performance. Early in the project the chip design team established a clear goal to meet or exceed the performance of the VAX 8700 proces sor, which has a performance of five times that of the VAX-11/780 system. Some of the architectural changes included the following:
•
A more pipelined architecture, specifically, a six level pipelined engine built around three auto nomous pipes
•
A 64-bit wide data bus with 27 separate address lines versus a 32-bit multiplexed data/address bus used for the CMOS-I chip
9
I
•
Support for decode of the new VAX vector instructions and transfer of instruction operand information to the vector interface bus and onto the vector coprocessor module
•
A 2 kilobyte (K B) primary on-chip cache with single-cycle access supported by a 128KB off chip secondary cache
•
A 16-byte instruction prefetch queue
•
Two quadword write buffers in the bus interface unit
Experience with the CMOS- I chip showed that the fabrication line was capable of producing a dis tribution of die across a performance range of 80 to
100 nanoseconds (ns). As a result, whereas all
CMOS-2 new designs supported the target of a 40 ns
CPU cycle time, it was an explicit goal to support devices as fast as 28 ns, should chip yields produce sufficient quantities of faster parts. The yields at
28 ns actually exceeded predictions and permitted faster parts to be used in all products produced.
First passes of all scalar CPU chips were available in April 1988. These chips could successfully boot both the VMS and ULTRIX operating systems. This success was due in large part to the aggressive use of computer-aided design (CAD) techniques. Func tional design verification efforts alone represented
25 person-years of work on the scalar chip set. An additional 39 person-years were necessary to com plete the scalar CPU module and the vector copro cessor verification efforts.
The power-on of first-pass parts represented a significant accomplishment to those who worked on the chips and the module. However, much
"behind the scenes" work was necessary to achieve this milestone. The success in this area represents the culmination of work across a number of disci plines. Besides the semiconductor devices, a new
224-lead multilayer ceramic package was devel oped. New techniques, including tape-automated bonding (TAB), were explored to attach the die to the package, and new specifications were necessary for the actual printed wire board material and board layup. Finally, new manufacturing processes were necessary to permit surface-mount assembly and test of these devices on both sides of the module.
Although initializing the operating system on the ftrst CPU modules marks a key deliverable for the chip and board designers, it is only a starting point for the systems activity that is necessary to fully test and qualify a new product prior to the start of highvolume manufacturing. It is often difficult to under stand the time lag between the availability of the first prototype unit running the operating system and a product that can be shipped to the customer.
However, a substantial amount of work must be done between these two events. As the formal qual ification process for new semiconductor devices begins, a parallel effort is undertaken to build a large number of early systems. These systems are used to identify any problems that may occur when the pieces of the system are assembled into configura tions typical of those used by actual customers.
Testing is divided across several aspects, including actual beta test of prototype un.its at customer sites, formal testing by any required government agencies
(e.g., FCC , Ul, V DE), systems design verification tests, and architectural testing that ensures that the new system complies with the formal VAX architec ture standards. Once the majority of testing is in process and the required interim milestones have been met, manufacturing begins turning the assem bled inventory into fin.ished products to support volume availability of the system.
When a series of systems such as the VAX 6000 family has established a history in the market, it becomes increasingly important to ensure that the announcement of the latest family member coin cides with manufacturing's ability to quickly deliver a high volume of product on a worldwide basis. If manufacturing cannot do so, a demand will have been created that cannot be filled. Revenue and sales are lost. The full payback from the many per son-years of design, simulation, design verification, and systems test is only finally realized when vol ume manufacturing has begun.
The papers in this issue of the journal will provide insight not only into how microprocessors and systems are designed and architected, but also into the multidisciplinary efforts necessary to bring a successful product to market. The design of one of the first VAX vector coprocessors is also reviewed.
This review offers a summary of how new architec tural issues are resolved and how design trade-offs are made.
Moving a product from advanced development to engineering, through manufacturing, and into the customer site, over a five-year period, requires the efforts of many people around the world.
Although only the direct work of a small percentage of those people are represented in these papers, the credit for the success of the products goes equally to all members of the team.
10
Debra L. Slater
David M. Fenwick
D. john Shakshober
Douglas D. Williams
Vector Processing on the
VAXvector 6000 Model400
The
VAX VAX
6000 family of midrange
CMOS-based multiprocessors to address the computing needs of numerically inten sive applications. The three function units of the vector processor combine to form an overall vector pipeline that operates at speeds of up to 90 MFLOPs for single-precision calculations and 45 MFLOPs for double-precision calculations. The processor's per f
orma
nce can also be enhanced by taking advantage of overlapping and out-of order instruction execution, as well as chaining. Further, applications can be tuned to the
VAX vector 6000 hardware through algorithm optimizations in areas
such
as equation solvers and signal processing routines to achieve optimal performance.
Using the
VAX vector 6000 Model 400 system, perf
orma
nce increases ranging from
3 to 35 times that of the
VAX
6000 Model400 scalar system have been realized.
Vector processing has significantly evolved over t he past t wo decades. In the late 1960s and early 1970s, it was pioneered as a way to increase scient ific applicat io n computer performance over t hat achieved by more traditional scalar computers.
However, t he technology was limited to an elite few who could afford mult im ill ion-dollar super computer systems and who were w illing to sig nificantly re-engineer software applicat ions.
I n the early 1 980s, more sophisticated vectorizing compil er technology was developed.
This technology allowed users to effectively pro gram in high-level languages , such as FORTRAN, rather than to manually vectorize using low-level assembly language. During t his period , t here were also significant developments in computer algorithms t hat were better matched to t he paral lel ism available in vector hardware.
Over t he past few years, a new breed of vector processor, t he mini-supercomputer, has emerged.
This class of machine includes many of t he perfor mance features of traditional supercomputers, but at costs more commonJy associated wit h super minicomputers. Because vector processing is now a mainstream sty le of comput ing t hat is applicable to a wide range of uses, the
VAX
architecture was recent ly extended to include vector operations.
Furt her, the
VAX
product line has been expanded t o include vector processing in both t he
VAX
6000 midrange systems fami ly and t he
VAX
9000 main frame family of systems.l
VAX Vector Processing Overview
The extension of t he VAX architecture to include vector processing features was done in a manner that permitted a wide range of possible implemen tations2 The extension also allowed exist ing VAX processors to execute code utilizing t he new vector instructions under software emulation.
The vector extensions to the VAX architecture include:
•
The addition of 1 6 vector registers, each contain ing 64 64-bit elements
•
A set of load/store instructions used to move up to 64 elements of a vector register to and from memory
•
A set of vector register-to-register arithmetic and logical instruct ions, operating on up to 64 ele ments at a t ime
•
A set of instructions for sy nch ronization bet ween scalar and vector processing subsystems
C on ceptually , the implementat ion of vector instructions w it hi n the VAX family of processors is similar to t hat of float i ng point inst ructions. To implement floating point arit hmet ic, some systems use dedicated floating point hardware, some sys tems use microcode, and others emulate fl oat ing point in macrocode. In vector processing, vector instructions differ from float ing point instructions
Digital Tecbnicaljournal
Vol. 2 No. 2, Spring 1990 11
VAX 6000 Model 400 System in that they are designed to be executed in a semi autonomous manner with scalar instructions. Thus, vecror instmctions can be executed in parallel with scalar instructions or in parallel with other vector instructions. Although the scalar and vector units operate somewhat independently, tht: units art: closely coupled to ensure that memory manage ment exceptions are precisely reported. Special operations ensure floating point exceptions and coherence between vector and scalar memory ref erences are synchronized.
From a vector perspt:ctive, a typical VAX vector implementation can bt: reduced to five hasic units.
The latter four units arc collectively rderred to as a vector processor or vector unit. The hasic units are:
•
A scalar processor that executes scalar instruc tions, decodes vector ·instructions, which may contain multiple internal function units
•
A vector instruction-sequencing control and reg ister scoreboard
•
A vector register file
•
An arithmetic pipeline or pipelines that consist of one or more arithmetic/logic units
•
A load/store unit for memory references
The vector control and scoreboard logic accepts instructions and operands from the scalar processor ami dispatches them to the individual function units within the processor. It also reports exceptions and interrupts to the scalar processor. Since multiple vector instructions can be executt:d in parallel , the unit may contain scoreboard logic ro identify and manage resource conflicts between instructions.
The vector register file contains the 16 vector reg isters, each of which consists of 64 64-bit elements.
The register file has multiple ports that permit loads or stores to operate while operands are sent to the arithmetic pipes and results are received .
The vector arithmetic/logic pipelines implement all the integer, logical, and floating point instruc tions. These pipelines may be composed of separate pipelined add, multiply, and logical units. Or, they may be composed of multiple pipes that operate in parallel, with each pipe consisting of a pipelined add/multiply/logic unit.
The load/store unit is responsibk for memory references. It generates the required virtual addresses (VA), performs translation from virtual to physical addresses, and loads or stores the data to or from the register files to memory . The load/ store unit controls a cache memory and contains a virtual-to-physical address translation buffer
(TB).
Depending on the design of the scalar and vector units, then: are two ways to implement the design memory interface for the scalar and vector processors:
•
A combined scal .
ar and vector processor that shares a common cache, address translation logic, and path to memory subsystem
•
Separate scalar and vector units with separate caches and address translation buffers
Both of these approaches have their relative merits and disadvantages. When significant data sharing between scalar and vector units exists, the combined approach provides more favorable cache performance because the common cache is updated on both scalar and vector references. Separate caches may result in additional cache misses as data is "sloshed" between scalar and vector caches.
For limited data-sharing instances, the separate approach may offer more favorable cache perfor mance. In a combined cache, vector references can displace needed scalar data and vice versa. This problem does not arise with separate caches because the scalar and vector data each has a dedicated cache. The separate cache approach also allows scalar and vector cache operations to occur in parallel.
In implementing a vector processor, the selection between the above alternatives is often driven more by technology constraints than issues of archi tectural elegance. The VAX 9000 system, which is implemented in emitter-coupled logic (EC L), chose the combined approach. This approach supported sharing costly cache RAMS and a common path to memory. T he VAXvector 6000, which is imple mented in complementary metal oxide semi conductor (CMOS) technology, chose the separate approach for two reasons. First, module space and package pin count constraints made it difficult to implement both scalar and vector functions on a single module. Second, the cost penalties for sepa rate scalar and vector cache RAMS and separate paths to memory were not prohibitive.
VAX 6000 Vector Processor
Description
System Block Diagram
The system block diagram for a vector-capable
VAX 6000 Model 400 machine is shown in Figure 1 .
1 2
Vi>/. 2
Nu.
2, Spring 1990
Digital Technicaljournal
Vector Processing on the VAX vector 6000 Mode/ 400
SCALAR
PROCESSOR
I-
VECTOR
PROCESSOR
SCALAR
PROCESSOR
I--
VECTOR
PROCESSOR
I
I
32MB
M E MORY
( U P TO 256MB OR
I
8
MODULES)
I
I
l···
1
32MB
M E MORY
XMI SYSTEM B U S
Bl
PORT
I
CABLE
. . .
ETH E R N E T VAXBI
D E B N I
BI PORT
D I S K
CONTROL
I f--
RA 7 0
STRIPESET
ESE20
L..
SOLID- STATE
DISK
...
RAXX
DISKS
KDB
1--
Figure 1 VAXvector 6000 Mode/ 400 System Block Diagram with Dual Scalar and Vector Processors
The vector processor occupies a slot adjacent to the scalar processor, and both are interconnected by a short interface cable. The vectOr processor receives all instructions from and returns status tO the scalar processor across this cable. For memory references, the vectOr processor has its own independent path to main memory . The VAX 6000 Model 400 system supports configurations of up to six scalar proces sors. However, vectOr systems have additional configuration constraints because of the increased memory bandwidth and XMI slot requirements of the scalar/vectOr processor. The VAXvector 6000
Model 4 00 system supports configurations of single or dual scalar/vector processors. or configurations of one scalar/vectOr processor and up tO three addi tional scalar processors. To satisfy memory band width requirements, VAXvectar 6000 systems with a single scalar/vector processor require at least two memory controllers. Dual scalar/vector systems or single scalar/vector systems with additional scalar processors require at least four memory controllers.
VAX 6000 Mode/ 400 Vector Processor
The block diagram for the vector processor is shown in Figure 2. The machine is divided into three separate function u nits that can operate in combination or independently:
•
A vector controller, implemented as a single chip
•
Arithmetic pipelines implemented by four pairs of chips , i . e . , register file and vector data path
•
A load/store unit, implemented by one chip, which also controls a 1 megabyte (MB) cache
Vector Control Chip
When the scalar processor encounters a vector opcode, it parses and fetches the operands.
The opcode and all its operands are dispatched through the instruction bus to the vector processor.
For arithmetic instructions, the scalar processor will proceed to decode the next opcode in the instruction stream. However, for load and store instructions, the scalar processor is stalled until all address translations are completed. Stalling guaran tees that any memory management violations are synchronous and that the scalar processor can restart the faulting instruction correctly. Within the vector unit the vector control chip is responsible for all scalar vector communication. When instruc tions are received by the vector controller chip, the vector controller buffers the instructions and controls instructions issuing to the other function units within the vector processor.
An important aspect of the vector control chip is the register scoreboard logic, which identifies potential register conflicts when vector instructions are executed in parallel. By maintaining accurate register usage data, the vector control chip can optimize parallelism with the vector processor.
Optimal performance is achieved by executing arithmetic operations in parallel with load and store operations, chaining the results of arithmetic opera-
Digital Tecbnicaljournal
Vol. 2 No.
2,
!>/JI"ing 1990
1 3
VAX 6000 Model 400 System
SCALAR PROCESSOR
FPU C H I P
VECTOR PROCESSOR
VECTOR
CONTROL
C H I P
64
INTERNAL CACHE B U S
VECTOR
REGISTER
FILES
A R I T H METIC
P I P E L I N ES
SCALAR XMI
INTERFACE
DUPLICATE
TAG STORE
LOAD/STOR E
A N D
X M I INTERFACE
X M I SYSTEM BUS
Figure 2 VAX vector 6000 Mode/ 400 Scalar/Vector Processor Block Diagram tions into store operations, and even dynamically re-ordering the execution of arithmetic instructions relative to load and store instructions to improve parallelism.
The vector control chip sends aU error status and machine checks to the scalar processor. When an error is encountered, the control chip attempts to retry the failing transaction. If the retry is suc cessful , a soft error interrupt is sent to the scalar processor. If the retry fails, either a hard error interrupt or a machine check is sent to the scalar processor. Read operations that fail result in machine checks. Write operations that fail result in hard error i nterrupts.
Although not part of the overall control function, the vector control chip also contains logic to imple ment the IOTA instruction. The IOTA instruction builds a set of offsets in a vector register. This func tion did not fit conveniently i n any other vector function. The control chip was selected because it had the space available to contain the function.
Vector Register File and Arithmetic
Pipeline
The VA.Xvector 6000 processor's arithmetic pipe l ine is organized as four pipes. Each pipe consists of a quarter of the register file (every fourth element of the vector registers), and an associated arithmetic/ logic unit. Each i nd iv idual pipeline can retire one single-precision calculation every cycle or one double-precision calculation every two cycles. The four pipes collectively retire four single-precision calculations every cycle, or two double-precision calculations every cycle. Thus, a much h igher float ing point performance is achieved than w ith only one individual pipeline.
The register file chips receive instructions from the vector controLler and data from the cache or load/store unit. The register file chip provides read operands to the arithmetic pipeline and stores write results and mask information. To maximize the use of cache bus bandwidth, two 32-bit operands can be combined i nto a single 64-bit transfer that is simultaneously read or written to two separate reg ister file chips. The register file internally has four
64-bit ports. (One is a read/write port for memory data; two are read ports for operands; and one is a w rite port for results. While one i ns truction is writing its results, a second can start reading its operands. Thus, the instruction pipeline delay is hidden. Variations in pipeline length between instructions are smoothly handled to ensure that no gaps exist in the flow of write data.
The register file can hold two outstanding arith metic instructions in its internal queue. Therefore, the vector controller can preload the arithmetic instruction queue w ith a second instruction, i .e. , deferred instruction. Preloading allows the vector controller to free the cache data bus, which is also used for instruction issuance, for use by a sub-
1 4
Vol. 2 No. 2,
Spring 1990 Digital Tecbnicaljournal
Vector Processing on the VAX vector 6000 Mode/ 400
sequent load or store i nstruction. This feature improves performance because the arithmetic pipeline can execute two arithmetic instructions in the time it takes to execute one load or store instruction.
The register file's operand and result ports are used by the vector arithmetic pipeline chip.
Operand data is sent over a 32-bit bus that is driven twice per cycle. Results are returned on a separate
32-bit bus that is driven once per cycle. The two operands for single-precision instructions can be passed in one cycle, w h ile doub le-precision operands take two cycles to transfer. Each arith metic pipeline chip has a throughput of one single precision operation per cycle, one double-precision operation per 2 cycles, and one single-precision or double-precision divide per 10 or 22 cycles. The arithmetic chip has a pipeline delay of six cycles for double-precision multiplications, and five for all others (except divides), including the data transfer cycles. Integer operations are recoded internally as double-precision floating point data types. The vector arithmetic pipeline chip is a full custom implementation largely based on the design of the scalar processor's floating point unit:'
Load/Store Unit
The control chip uses the vector processor's i nter nal bus to issue instructions to the function units.
However, once a load or store instruction is issued, the load/store chip becomes bus master and con trols the internal bus. Either the load/store chip, vector register files, or the cache can drive the bus.
Once a load or store instruction starts execution, no further instructions can be issued until it com pletes. This rule simplifies the control chip score boarding because once a load or store instruction is started, no further instructions can start. Therefore, scoreboarding of these instructions against the out standing load or store instruction is not necessary.
Because scoreboarding of outstanding instructions requires considerable logic complexity in the vector control chip, it was important to keep the complexity of this operation minimal. An addi tional benefit was the simplification of the internal bus protocol. It was excessively complex to imple ment the capability to stop load or store instruc tions in progress. This alternative was not pursued because the resulting performance benefit was minor in comparison to the amount of work involved.
The load/store chip executes the vector load, store, scatter and gather memory reference instructions. These instructions involve virtual-to-physical address translation, cache management, and inter action with the memory bus. If a load or store i nstruction requires an offset register, such as scatter or gather, the offset register is first read into a buffer and then added to the instruction's base address. This process eliminates turning around the internal bus for each offset read, which would add more overhead. For strided load or store instruc tions, the address is generated by adding the stride to the instruction's base address.
Load or store instructions can operate on either
32-bit (i.e. , long word, single-precision) or 64-bit
(i.e. , quad word , double-precision) data types.
When executing unity-stride 32-bit load or store instructions, the load/store chip operates on two elements at a time. Two 32-bit elements are com bined into a single 64-bit cache reference. This combination significantly enhances performance enhancement in unity stride single-precision data operations.
Virtual-to-physical address translation is per formed using an on-chip, 136-entry, 68-way-asso ciative translation buffer (TB). This configuration maximizes address translation efficiency, which is very important because only limited chip space was available. To optimally service TB m iss conditions, the load/store chip contains dedicated logic that directly references page table entries upon a TB miss. A simpler alternative would have been to use m icrocode in the scalar processor to fetch new page table entries upon a TB miss. However, the dedicated logic approach was chosen to enhance performance for applications that exceed the size of the TB. Under certain TB miss conditions, the vector processor may be unable to compute a new virtual-to-physical address translation. This situa tion can occur when the addressed page is invalid or has been paged-out to disk. When such a miss occurs, the vector unit reports an exception back to the scalar processor. Once the scalar processor corrects the situation, the instruction is retried from the beginning.
Since the scalar processor must be able to restart the faulting vector instruction, it is important to precisely identify any vector memory management exception with the associated vector load or store instruction. This identification is achieved by block ing issuance of further instructions until the vector unit notifies the scalar processor that the vector instruction is free of memory management faults.
The vector unit contains memory management prediction logic, called MMOK logic. MMOK logic
Digital Tecbnicaljournal
Vol. 2 No. 2, Spring 1990
1 5
VAX 6000 Model 400 System allows the scalar processor to issue additional instructions in parallel with the execution of the vector load or store instruction. During execution of a strided vector load or store instruction, once it is established that the current vector element references the same TB entry as the last element of the load or store instruction, and that the associ ated TB entry is free of memory-management exception conditions, the load/store unit can safely report " address translation successful," i . e . , MMOK , to the scalar processor. Early prediction of success ful address translation permits the scalar processor to be released and allows it to operate asyn chronously with the remainder of the vector load or store instruction.
Once a physical address is obtained, the load/ store chip references its 32K entry tag store. The address is delayed and passed to the 1 MB cache data store. This delay permits cache tag lookup and com pare to complete before data is written to the cache on store operations. In parallel, the corresponding register file address is presented to the four register file chips. The data and addresses are automatically aligned for load and store operations to permit correct reading and writing of the register file and cache data RAMs. Upon cache miss, the load/store unit queues the associated 32-byte block read oper ation with the memory interface logic and contin ues processing other elements. Up to four cache misses can be outstanding before the read data for the first miss returns. H its continue to be processed while the misses are outstanding. On vector proces sors, the most important factor is the time required to complete the entire load or store operation, rather than the time needed to fetch an individual element. The cache miss handling feature permits the vector processor to m:vdmize its use of available
XMI bandwidth.
The vector cache tag and data are parity-pro tected. Should a cache parity error occur, the cache is disabled and the instruction retried from the beginning. This method was the simplest option for soft recovery of cache parity errors. The operating system receives a soft error i nterrupt and can, at its option, re-enable the cache.
The load/store chip contains a 32-element write buffer to enhance performance of store operations.
Since the vector cache operates at higher band widths than the system bus, the buffer isolates the store performance from the slower X M I memory bus. Furthermore, a subsequent load instruction that hits in cache can execute while the write buffer i s being written t o memory. I f the l oad instruction takes a cache miss, the load stalls until the store completes. This simple scheme improves instruc tion overlap when load instructions follow store instructions without adding undue complexity to the load/store unit design.
Performance Characteristics
The interaction between the different functional units of the vector processor creates a number of sit uations that affect the performance and execution of vector instructions. These include:
•
•
Overlapping instructions
Out-of-order instructions
•
Chaining
Overlap of Instructions
Arithmetic and load/store instruction execution may overlap because the functional uni ts are inde pendent. In order to achieve this overlap, the following conditions must be met.
•
The arithmetic instruction must be issued before the load or store instruction.
•
There must be no register conflict between the arithmetic and load/store instructions.
In the following examples of arithmetic and load/ store instruction interactions, an
" 1 " represents instruction issue time, and a n " E " represents instruction execution time. The "VR" represent vector registers. The expression "std" is used to represent the stride. A series of periods " . . . " repre sents wait time in the arithmetic u nit for deferred i nstructions. (Note: These examples are not intended as timing diagrams.)
VVADDx V R 1 , VR2 , VR 3
V L D x A , s t d , VR 1
I E E E E E E E E
I E E E E E E E E E E E E E E
A s can be seen in the example above, the execu tion of the vector load instruction (VLDx) can overlap the vector add instruction (VVADDx) because there are no register conflicts between the two instructions. In the next example, instruction overlap is inhibited because the VVADDx instruc tion is writing to the register to be loaded, VR3.
VVADDx VR 1 , V R2 , V R 3
V L D x A , s t d , V R3
I EE E E E E E E
I E E E E E E E E E E E E E E
In comparing these two examples, it i s clear that the overlap of the execution of the VVADDx and the
1 6
Vul
2
No.
2, Spring
1990 Digital Technical]ournal
Vector Processing on the VAX vector 6000 Model 400
VLDx greatly reduces the total execution time of the instruction sequence. By raking advantage of this hardware feature, application codes can show greatly improved performance.
Out-oforder Instruction Execution
The arithmetic unit includes a deferred instruction queue of length 1 . This queue allows the vector control and scoreboard logic to queue one instruc tion to the arithmetic unit while that unit is still executing a previous i nstruction. The vector con troller checks the queue's status for an instruction when it checks the function unit's availability. Both the deferred and currently executing instructions are checked for register availability. This queue frees the issue unit to process another instruction rather than waiting for the arithmetic unit to com plete its current instruction.
For the fol lowing instruction sequence,
VVADDx V R 1 , VR 2 , VR 3
VVMUL x VR3 , VR 1 , VR 4
V L D x A , s t d , VR2 execution w ithout a deferred instruction queue would resemble this example:
I s s u e VVADD x I E E E E E E E E
I s s u e VVMU Lx
I E E E E E E E E
I s sue VLDx
I E E E E E E E E E E E E E E
Execution with a deferred instruction queue would look like the follow ing:
I s s u e VVADD x
I s s ue d e f e r r e d VVMU L x
I s s ue VLDx
I E E E E E E E E
I . . . . . . . E E E E E E E E
I E E E E E E E E E E E E E E
These examples illustrate the use o f a deferred arithmetic instruction. If a deferred instruction queue was not implemented, the VVMULx instruc tion could not be issued until the VVADDx was either completed or nearly completed. The VLDx instruction would not issue until after the VVM U Lx was issued and would complete much later than in the deferred instruction case. Once the VLDx instruction is issued, no other instructions may be issued. The instruction overlap execution made possible by the deferred instruction queue signi ficantly reduces total execution time.
Comparing the last two examples, in the case where a deferred instruction queue wa s used, the
VLDx instruction can begin executing before the
VVMULx. It a lso could complete before the VVMULx instruction completes simply because the VVMULx instruction is sent to the deferred arithmetic instruction queue. This out-of-order execution of instructions allows increased overlap of instruc tions, which again reduces the total execution t ime of the instruction sequence.
Chaining
Vector operands are generally read from and writ ten to the vector register file. An exception to this process occurs when a store instruction is waiting for the results of a currently executing arithmetic instruction. (Divide instructions are nor included in this exception because they do not have the same degree of pipelining as the other instructions.) As resul ts are generated by the arithmetic instruction and are ready to be written to the register file, they are also inunediately available for input to the wait ing store instruction. Therefore, the store instruc tion can begin processing the data before the arithmetic instruction has completed. This process is called "chain into store." The srore instruction will not overrun the arithmetic instruction because the srore instruction can process data faster than the arithmetic unit can generate results.
The following instruction sequence
VVADDx VR 1 , VR 2 , VR3
VVMU L x VR 1 , VR 2 , VR 4
VSTx
VR3 , A , s t d would resemble the example i n Figure 3 i f executed without the chain into store process.
I s s u e VVADDx
I s s ue d e f e r r e d VVM U L x
I s s u e VSTx
I E E E E E E E E
I .
. E E E E E E E E
I E E E E E E E E E E E E E E
Figure
3
Sample Instruction Sequence without Chain into Store
Digital Teclmical]ournal
Vol. 2 No. 2. Spring 1990
1 7
VAX 6000 Model 400 System
If t he instruction sequence were executed with t he chain i nto store process, however, it would follow t his example:
I s s u e VVADDx
I s s ue de f e r r e d VVM U L x
I s s u e V S T x
I E E E E E E E E
I . . . . . . . E E E E E E E E
I E E E E E E E E E E E E E E
In these examples, t he VSTx instruction requires t he result of t he VVADDx instruct ion. Without t he chain into store operation, the VSTx instruction must wait for t he VYADDx to complete before beginning. The chain into store operation allows the VSTx instruction to begin while the VYADDx is still execut ing, increasing t he amount of instruc t ion execution overlap. As a result , the inst ruction sequence requires a shorter period of t ime to com plete execut ion.
Load and Store Unit Performance
Memory access instructions are typically slower t han arit hmetic instructions and will frequent ly dominate the performance of vectorized appli cations. Certain coding techniques help minimize t he t ime spent waiting for load and store instruc t ions to complete and reduce t he resulting impact on performance.
Maximize Instruction Execution Overlap Three important hardware features help maximize instruction execution overlap in t he load/store unit .
First , a load or store instruction can execute in parallel with up to t wo arithmetic instruct ions, pro vided the arit hmetic instruct ions are issued first.
Second, t he chain into store sequence can reduce t he perceived execution t ime of a store instruction.
Finally, early detect ion of no memory faults allows scalar-to-vector unit communications to overlap w ith load or store instruction execut ion .
In t he inst ruct ion sequence shown in Figure 4, t he main loop of a SAXPY or DAXPY B LAS 1 routine, t here is very little instruction overlap. 1
In t he reordered instruction sequence shown in
Figure 5 , t he VVMULx and second VLDx instruc t ions overlap, and less t otal execut ion t im e is required t han in the first example.
The only real difference bet ween the instruction sequences in Figures 4 and 5 is t he order in which t hey are issued. By recognizing t hat t he VVMULx does not require the result of t he second VLDx and can precede t hat instruct ion , a significant reduct ion in execut ion time is achieved.
The overlap of load and store inst ructions can also be effect ively maximized by preceding, wherever possible, all load and store instructions by at least two arithmet ic instructions. In t his way, both t he load and store pipeline and t he arithmetic pipeline are in use.
Minimize Register Conflict Waits A load i nstruc t ion cannot begin execution u nt i l t he regist er t o which it w ill write i s free. A register conflict may occur if the dest inat ion register of a load i nstruction is t he same as the register for a preceding arithmetic instruction . If using a different register for t he load instruction would permit instruct ion execution overlap to occur, t he dest inat ion register should, if possible, be changed.
Locality of Reference of Data The locality of ref erence of data is important i n determining t he per formance of load and store operations. Because unity stride load and store instruct ions are the most efficient memory access instruct ions . whenever possible, data should be stored in t he sequent ial order in which it is usually referenced.
Non-unity st ride load and store operat ions can have a significant ly higher impact t han unit y stride
VLDx X , s t d , VR 1
VLDx Y , s t d , VR2
VVM U L x VR3 , VR 1 , VR 1
VVADDx V R 1 , VR 2 , VR 2
VSTx VR 2 , Y , s t d
I E E E E E E E E E
I E E E E E E E E E
I E E E E E
I . . . . E E E E E
I E E E E E E E E E
1 8
Figure
4
Sample Instruction Sequence of Main Loop of a SAXPY or DAXPY BIAS
1
Routine
Vol. 2 No. 2,
Spring
/')')()
Digital Technical journal
Vect01· Pmcessing on the VAX vector 6000 Model 400
V L D x
X , s t d , V R 1
VVMU L x V R 3 , V R 1 , VR 1
V L D x Y , VR 2
VVADDx VR 1 , V R 2 , VR 2
V S T x V R 2 , Y , s t d
I E E E E E E E E E
I E E E E E
I E E E E E E E E E
I E E E E E
I E E E E E E E E E
Figure 5 Sample Instruction Overlap Sequence of a Main Loop of a SAXPY or DAXPY BLAS 1 Routine operations on the performance level of the
X M I memory bus. More memory references are required for non-unity stride operations of the same vector length. If the ratio of cache miss load and store instructions to arithmetic instructions is sufficiently high and non-unity stride is used, bus speed and bandwidth can limit performance.
Load and store operations that hit in cache are less costly than those that miss cache. Any piece of data must be loaded from memory to cache the first time it is referenced. If data that is referenced more than once remains in the cache, i.e. , is not displaced by subsequent data accesses, other references to that data will not incur memory access costs, and better performance results.
On any cache fill, a 32-byte data block is read i nto the cache. (Note: This is equivalent to 8 long words or 4 quadwords. Single-precision data is a long word; double-precision data requ ires a quadword .)
When non-unity stride loads are used, performance can be improved by using the additional data read in to cache. This improvement can be achieved by fol lowing cache miss non-unity stride loads w ith non-unity stride loads that reference the additional data and will, therefore, hit in cache.
Large data arrays and strides can also have an impact on the efficiency of the translation buffer.
For large strides, i .e., greater than
256
single-preci sion elements or
1 28
double-precision elements, a translation buffer m iss can occur for each vector element. Even with unity stride, the translation buffer miss rate will be higher for large data arrays.
Algorithms It is sometimes necessary to consider the algorithm that is represented by the code to be optimized because some algorithms are not as well suited to vector processing as others. It may be more effective to change the algorithm used or the way it is implemented than to optimize the existing code. Several examples that illustrate effective opti mization methods are discussed in the Algorithm
Optimization section of this paper.
Arithmetic Unit Performance
Once an instruction begins execution in the vector arithmetic uni t , it continues executing until all results are completed . A deferred arithmetic instruction may only begin execution after the instruction in the pipeline completes, or if the first results of the deferred instruction will not complete before the last results from the current instruction are completed. The instruction overlap will be particularly significant for shorter vectors because the startup t ime, i . e. , the time that can be overlapped , for an arithmetic instruction is fixed overhead that represents an increasing portion of the execution time as vector length decreases.
Peak Performance
The Model 400 vector processor has a cycle time of
44.44
nanoseconds. For single-precision opera tions, this cycle time translates to a theoretical peak performance of 90
M F LOPs.
For double-precision operations, the theoretical peak performance is
45
MFLOPs.
Theoretical peak performance is calculated from the number of results per cycle and the cycle time as follows:
Single-precisionpeak =4 / (44 .44 x
101)
Double-precision peak =2
I
(44.44
x
103)
Crossover Point
For any given instruction or sequence of instruc tions, there is a particular vector length where both the scalar and vector processing of equivalent oper ations yield the same performance. This vector length is the crossover point between scalar and
Digital Tecbnical]ournal Vul.
2 No. 2, Spring 1990 1 9
VAX 6000 Model 400 System vecror processing and is unique to the particular instruction or sequence. Scalar operations are faster for vector lengths below the crossover point. Vector operations are more efficient for vector lengths above the crossover point. A low crossover point is considered a benefit because it indicates that it is easier to take advantage of the power of the vector processor. A low crossover point means that more code can benefit from the vector processor.
For any single, isolated vector instruction, the crossover point on the Model 400 system is quite low, generally about 3 or 4. However, an instruction is not performed in isolation . Jn a routine or application, other factors affect the performance of the operations on short vectors. These effects are particularly seen when the short vector's data is used in several vector operations.
On the Model 400 system, performing as much code as possible on the vector processor, including short vector length sections, can mean higher system performance. Performance is improved because the cache is used more optimally. Speci fically, once vector instructions have referenced a piece of data, that data is included in the vector unit's cache. Subsequent scalar operations on that data will require moving the data from the vector cache into the scalar cache. Continued code sec tions of vector references fol lowed by scalar references tend to invalidate the two caches too frequently. Therefore, a vector operation is usually more efficient than a scalar operation. The cross over point on the Model 400 system is low enough that scalar processing is the faster alternative only for isolated operations on short vectors.
Algorithm optimization Examples
The previous section of this paper discussed how the characteristics of the VAX vector 6000 Model 400 system's vector processor can affect performance.
The following examples illustrate how that perfor mance information can be used to build optimized routines. The examples also show how an algorithm and its implemenration can change the performance of an application on the VAX vector 6000 processor.
Algorithm changes can alter the data access pat terns to use the memory subsystem more efficiently, can increase the average vector length, and can min imize the number of vector operations required.
By applying Amdahl's Law of vectorization, we can improve performance by increasing rhe percentage of code that is vectorized.
To rake advantage of the processing power of the
VAX vector 6000 Model 4 00 system, we concen trated on four basic optimization mer hods:
•
Rearrange code for maximum vectorization of the inner loop and remove data dependencies within the loop
•
Vecrorize across contiguous memory locations to produce unity stride vectors for increased cache hit rates and optimized cache miss handling
•
Reuse the data already loaded inro the vector reg isters as frequently as possible to reduce rhe number of vector load and store operations
•
Maximize instruction execution overlap by pair ing arithmetic instructions between load and store instructions wherever possible
(Note: Further i nformation on optimization techniques in FORTRAN can be found in the VA X
FORTRA N Performance Guide available with the
FORTRAN-High Performance Option." Additional information on macrocoding for the VAXvector
6000 Model 400 vector processor can be found in the VAX 6000 Vector Processor Programmer's
Guide.
6)
By analyzing the groups of applications that have high vector processing potential, we identified two basic areas where optimization techniques can be most useful: equation solvers and signal process ing routines. For example, computational fluid dynamics, finite element analysis, molecular dynamics, circuit simulation, quantum chromody namics, and economic modeling applications use various types of simultaneous or differential equa tion solvers. Applications such as air pol lution modeling, seismic analysis, weather forecasting, radar imaging, speech and image processing, and many other scientific and engineering applications use signal processing routines, such as fast Fourier transforms, to obtain solutions.
Equation Solvers
Equation solvers generally fall into four categories: general rectangle, symmetric, hermitian, and tri diagonal . The most common benchmark used to measure a computer system 's abil ity to solve a general rectangular system of linear equations is
Linpack.� The Linpack benchmarks, developed at
Argonne National Laboratory, measure the perfor mance across different computer systems while solving dense systems of 100, 300, and 1000 linear equations.
20
Vol. 2 No. 2. Sprin!!,
1990 Digital Technical journal
Vector Processing on the VAXuector 6000 Mode/ 400
These benchmarks are currently written to call subroutines from the Linpack library . The subrou tines , in turn , call the basic linear algebra subrou tines (BLAS) at the lowest level . For each benchmark size, there are different optimization rules which govern the type of changes permitted in the Linpack report. Optimizations to the BLAS routines are a lways allowed . Modi fications can be made to the
FORTRAN source or by supplying the routine in macrocode. A lgorithm changes are only allowed for the largest problem size, the solution to a system of
1000 linear equations.
The smallest problem size uses a two-dimensional array that is
100 by
100.
The benchmarks are writ ten to use Gaussian elimination for solving 100 simultaneous equations. This two-step method fea tures a factorization routine, xGEFA , and a solver, xGESL .
Both are column-oriented algorithms and use vector-vector level 1 BLAS routines. Column orientation increases program efficiency because it improves locality of data based on the way
FORTRAN stores arrays.
As shown in Figure
6, the
BLAS level
I routines allow the user to schedule the instructions opti mally in vector macrococle. Deficiencies in BLAS I routines include frequent synchronization, a large cal l ing overhead , and more vector load and store operations in comparison to other vector arithmetic operations.
The performance of t he Linpack
IOO by
IOO benchmark, which calls the Figure
3 routine, shows how an algorithm with approximately 80 percent vectorization can be lim ited by the scalar portion .
One form of Amdahl's Law relates the percentage of vectorizecl code compared to the percentage of scalar code to define an overall vector speedup. This ratio between scalar runtime and vector runtime is described by the following formula:
Vector speedup
= time scalar I ( % [scalar
I x time scalar) + ( I % vector
I x time vector )
Under Amdahl's Law, the ma.:ximum vector speedup possible, assuming an infinitely fast vector processor, is:
Vector speedup
=
1 .0 I (0. 2) X 1 .0 + (0.8) X 0
=
1.01 0.2
=
5.0
As shown in Figure 7, the Model 400 processor achieves a vector speedup of approximately 3 for the
100 by
100
Linpack benchmark when using the
BLAS 1 subroutines. It follows Amdahl's Law closely because it is small enough to fit the vector proces sor's
1
Mbyte cache and, therefore, incurs very little overhead clue to memory hierarchy.
For the Linpack 300 by 300 benchmark, opti mizations include the use of routines that are equivalent to matrix-vector level 2 BLAS routines. x A X P Y - comp u t e s Y ( I )
•
Y < I l
• aX ( I ) w h e r e x . p r e c i s i o n F , D , G
L O O P :
M S Y N C
V L D x X ( I l , s t d , VR O
V S M U L x a , VR O , VR O
VLDx Y < I l
, s t d , VR 1
VVADDx VRO , V R 1 , V R 1
V S T x VR 1 , Y < I ) , s t d
I NC
I F < I
<
S I Z l GOTO LOOP
M S Y N C
; s y n c h r o n i z e w i t h s c a l a r
; X ( I ) i s l o aded i n t o VRO
; VR O ge t s the p r o d u c t of V R O
; an d t h e s c a l a r va l ue " a "
; Y < I ) g e t l oaded i n t o VR 1
; VR 1 g e t s V R O s ummed w i t h VR 1
; VR 1 i s s t o r ed b a c k i n t o Y < I )
; i n c remen t I b y vec t o r l e n g t h
; L o o p f o r a l l va l u e s o f I
; s y n c h r o n i z e w i t h s c a l a r
Figure 6 Core Loop of a BLAS 1 Routine Using Vector- Vector Operations
Digital Technicaljournal
Vol. 2 No. 2, .vJrin� /'J'JO 2 I
VAX 6000 Model 400 System
35
0..
::J
30 w w
0..
(f) a:
12 w
>
25
20
1 5
1 0
5
0 200 400 600 800
DIMENSION OF PROBLEM
S I Z E
1 000 1 200
Figure
7
Linpack Performance Graph,
Double-precision BLAS Algorithms
Figure 8 details the core loop of a BLAS 2 routine.
BLAS 2 routines make better use of cache and trans lation buffers than the Bl.AS
I routines do. Also,
BLAS 2 routines have a better ratio between vector arithmetics and vector load and stores. The larger matrix size increases the average vector lengt h . Per formance is improved by amortizing the time to decode instructions across :t larger work load.
By removing one vector load and one vector store from the innermost loop, the
BLAS
2 routine has a better ratio of arithmetic operations to load and store operat ions than B LAS I routines. Although the
300 by .300 array fits into the vector processor's
I
MB cache, not all the cache can be mapped by i ts translation buffer. By changing the sequence in which this routine is called in the program , the data access patterns can be altered to better use the vector unit's translation buffer. Thus, higher per formance is obtained.
The percent of vectorization increases primarily because of the increase in the matrix size from 100 by 100 to 300 by 300. With a vector fraction of approximately 95 percent, Figure 7 shows the speedup impron.:ment in the
300 by mark when using methods based on
300
BLAS bench
2 rou tines. With a matrix vector algorithm, the 300 by
300 benchmark yields speedups of between 10 and
12 over its scalar counterpart.
There are no set rules to follow when solving the largest problem size, a set of 1000 simultaneous equations. One potential tool for optimizing this benchmark is the LAPACK library developed by
Argonne National Laboratory, in conjunction with the University of Ill inois Center for Supercomputing
Research and Development (CSRD). The LAPACK library features equation solving algorithms that will block the data array into sections that fit into x G EMV - c omp u t e s Y C I ) = Y C I )
+
X ( J ) • M ( I , J ) w h e r e x = p r e c i s i o n = F , D , G
; s y n c h r o n i z e w i t h s c a l a r
! LO O P :
MSYNC
VLDx
J L O O P .
Y (
I )
, s t d , VRO
VLDx M ( I , J) , s t d , VR 1
VSMU L x X C J ) , VR 1 , VR2
; Y ( I ) i s l o aded a s VRO
; VR 1 g e t s c o l umn s of M ( I
,
J
)
; VR2 g e t s t h e p r o d u c t o f VR 1
; an d X ( J ) as a s c a l a r
; VR O g e t s VRO s ummed w i t h VR2 VVADDx V R O , VR2 , VR O
I NC j
I F ( J
<
S ! Z ) GOTO J LOOP ; L o o p for a l l va l u e s of J
V S T x VR O , Y ( ! ) , s t d
I NC
I F ( I
<
S I
Z )
GOTO I LOOP
M S Y N C
; VR O g e t s s t o r e d i n t o Y C I )
; L o o p f o r a l l va l ue s o f I
; s y n c h r o n i z e w i t h s c a l a r
2 2
Figure 8 Core Loop of a BLAS 2 Routine Using Matrix-Vector Operations
Vol. 2 No. 2, Spring 1?90
Digital Tecbnicaljounlal
Vector Processing on the VAX vector 6000 Mode/ 400 a given cache size. The LAPACK library calls not only the BLAS 1 and BLAS 2 routines but also a third level of BLAS , called matrix-matrix BLAS or the BLAS level 3."
Figure 9 shows that a matrix-matrix multiply is at the heart of one BLAS 3 routine. The matrix multi plication computation can be blocked for modern architectures with cache memories. H ighly efficient vectorized matrix m u ltipl ication routines have been written for the VAX vector architecture. For example, a double precision 64 by 64 matrix multipl ication achieves over 85 percent of the peak
M F LOPs on the Model 400 system.
Performance can be further improved with other methods that increase the reuse of data while it is contained in the vector registers. For example, loop unrolling can be done until all the vector registers have been fully utilized. Partial results can be formed within the innermost loop to minimize the loads and stores required. Because both rows and columns are traversed, the algorithm can be blocked for cache size. The VAXvector Model 400 system exhibits vector speedups greater than 35 for t he 64 by 64 matrix multiplication described above.
Although the overall performance of the 1000 by
1000 size benchmark is less than a single 64 by 64 matrix multiplication, it does indicate the potential performance when blocking is used. Improving the performance of this benchmark is most challenging because the 1000 by 1000 matrix requires about eight times the vector cache size of l l\-18 . Further analysis is being conducted to determine the most efficient block size that would maximize the use of
BLAS 3 and remain within the size of the cache for a given block of code.
The vectorized fraction i ncreases to approxi mately 98 percent for the 1000 by 1000 benchmark.
The proportion of vector arithmetics relative to vector loads and stores is much improved for the
BLAS 3s. Although the cache is exceeded , perfor mance more than doubles when using a method that can block data based on the BLAS 3 algorithms.
Therefore, the VAXvector 6000 Model 400 pro cessor's performance for Linpack 1000 by 1000 x G EMM - c omp u t e s Y < I , J ) = Y < I , J )
•
X ( w h e r e x = p r e c i s i o n = F , D , G
I
, K l * M ( K , J )
I J L O O P :
M S Y NC
VLDx
K L O O P :
VLDx
VSMU L x
Y ( I , J ) , s t d , VR O
M ( K , J l , s t d , VR 1
X
( I , K ) , V R 1 , VR 1
VVADD x V R O , VR2 , VR O
I N C K
I F ( K
<
S I Z l GOTO K L D D P
R E S E T
V S T x
I NC
K
V R O , Y < I , J ) , s t d
I
I F ( I
<
S I Z l GOTO I J LOOP
I NC J
R E S ET
I F ( J
<
S I Z l GOTO I J LOOP
M S Y N C
; s y n c h r o n i z e w i t h s c a l a r
; Y ( 1 : N , J ) g e t s l o aded i n t o V R O
; K ( 1 : N , K ) g e t l oaded i n t o VR 1
; VR 1 ge t s V R 1 s ummed w i t h
; X (
I , K ) a s a s c a l a r
; VR O g e t s V R O s ummed w i t h VR2
; i n c r eme n t K b y ve c t o r l e n g t h
; r e s e t I t o S I Z
; VR O g e t s s t o r ed i n t o Y ( I , J )
; i n c reme n t I b y ve c t o r l en g t h
; i n c remen t J b y ve c t o r l en g t h
; r e s e t I t o S I Z
; s y n c h r o n i z e w i t h s c a l a r
Figure
9
Core Loop of a BLAS 3 Routine Using Matrix-Matrix Operations
Digital Technical journal Vol.
2 No. 2, Spring 1990
23
VAX 6000 Model 400 System obtained a vector speedup of approximately 2 5 , as shown in Figure 7.
Signal Processing- Fast Fourier
Transforms
The Fourier transform decomposes a \vaveform, or more general l y. a collection of data, into com ponent sine and cosine representation . The discrete
Fourier transform
(DFr) of a data set of length .v performs the transformation following the strict mathematical definition which requires U (N2) floating point operations. Jn
1965 , the fast Fourier transform
(FFT) was developed by Cooley and
Tukey. FFT reduced the number of operations to O (N x LOG [N l), which is a significant improve ment for computational speed
.H
As shown in Figure 10, the complex data in the bottom butterfly is multiplied in each stage by the appropriate weight. The result is then added to the top burterfly and subtracted from the bottom butterfly. If the algorithm is left in this configura tion, it must use non-unity stride vectors, very short vectors, or masked arithmetic operJtions to per form the very small buttertlies.
Optimized One-dimensional Fast Fourier
Transforms
The bit-reversal process that permutes the data to a form that enables the Coolcy-Tukey algorithm to work i s also show n in Figu re
10.
When using vectors, a common approach to performing the bit reversal reordering is to use vector gather or vector scatter instructions.<) These instructions allow vector loads and stores to he performed using an index register. Vector loads and srorcs require a con stant stride. However, vector gather and scarter operations a l l ow the user to build a vector of offsets to support indirect addressing in vector mode. Both gather and scatter instruct ions are available with
VAX vectors.
A vector implemenration of the FIT algorithm has been developed that is wel l suited for the
VAX vector architecture. One optimization made to the algo rithm involves moving the bit-reversal section of the code to a place where the data permutation will benefit vector processing. By doing so, two goals are accomplished . First, the slower vector gather operations are moved to the center of the algorithm such that the data w i l l a lready be in the vector cache. In Figure 10, the first
FFT stage starts out with large butterfly distances. After each stage the butter fly distance is halved. For the optimized ,·ersion
4
5
6
7
8
9
1 0
1 1
BIT
R E V E RSAL
R EORDER
0
1
2
3
0
8
4
1 2
2
1 0
6
1 4
1
9
5
1 3
1 2
1 3
1 4
1 5
3
1 1
7
1 5
LOG ( N )
STAGE 1
STAGE 2 STAGE 3 STAGE 4
STAG ES
1 2
1 3
1 4
1 5
4
5
6
7
8
9
1 0
1 1
COR RECT
R ES U LT
0
1
2
3
Figure
10
The Cooley-Tukey Butterfly Graph,
One-dimensional Fast Foun·er
Transform for N
=
Hi) shown in Figure I
I , the hit-reversal permutation is performed as dose to the cenrer as rossible, when the stage number
=
LOG (N)/2. To complete the algorithm, the butterfly distances now increase again . Second , this process ent ire l y eliminates the need for short butterflies.
Another optimization made to t he FFT algorithm is the use of a table lookup method to access the sine and cosine factors, which reduces repetitive calls to the computatiorully inrensive trigonometric func tions. The initialization of this trigonometric table has been fully vectorized hut shows only a modest factor of 2 performance gai n. ·ro build the table, a first-order linear recurrence loop is formed that severely limits vector speedup. Because this cal cula tion is only done once, it becomes negligible for multiple cal ls to the one-dimensional FI·Ts and for all h igher dimensional
FFTs.
The benchmark shown in Figure 1 2 was looped and includes the calcu lation of the trigonometric table performed once for each
FFT data length .
Reusing data in the vector registers also saves vector processing time. The provides
VAX vector architecture
16 vector registers. If a l l 16 registers are used careful ly, data can be reused by two successive butterfly stages without storing and reloading the data. With half the number of loads and stores, the vector performance almost doubles.
24
Vol.
2 No. l. .\jJrinJ< I'J'JO
Digital Techn icaljournal
Vector Processing on the VAX vector 6000 Model 400
0
1
2
3
BIT
R EV E R SA L
R E O R D E R
0
1
2
3
0
8
4
1 2
COR RECT
R E S U LT
0
1
2
3
4
5
6
7
1
8
9
0
1 1
1 2
1 3
1 4
1 5
8
9
1 0
1 1
4
2
5
1 0
6
6
7
1 4
1
9
5
1
3
1 2
1
3
1 4
1 5
3
1 1
7
1 5
STAGE 1 STAGE 2 STAGE 3 STAGE 4
( H A S
R EG - R E U SE)
1 2
1 3
1 4
1 5
4
5
6
7
8
9
1 0
1 1
Figure I I
Optimized Cooley-Tukey Butterfly
Crapb, One-dimensional Fast
Fourier Transform for N
=
16
Optimized Two-di
mens
ional Fast Fourier
Transforms
The optimized one-dimensional
F FT can be used to compute multidimensional FFTs. Figure 13 shows how an N by N two-dimensional
FFT can be com puted by performing N one-dimensional column
FfTs and then N one-dimensiona l row F FTs. The same routine can be called for column or row access
FFTs by simply varying the stride parameter that is passed to the routine. (Note: In FORTRAN, the column access is unity stride and the row access has a stride of the dimension of the array.)
For improved performance on
VAX vector systems, the use of a matrix transpose can dra matica l ly increase the vector processing perfor mance of two-dimensional FFTs for large values of
N, i .e. , N
>
256. The difference between unity stride and non-unity stride is the key performance issue.
Figure 14 shows that a vecrorized matrix transpose can be performed after each set of N one-dimen sional FFTs. The computation will be equivalent to
Figure 10 but with a matri.,x transpose: each one dimensiona l FFT will be column access which is unity stride. The overhead of transposing the matrix becomes negligible for large values of N
When the value of N is relatively small , i . e . ,
N
<
256, the two-dimensional F F T can b e com puted by calling a one-dimensional FFT of length N2
1 6
1 4
(l.
5
1 2 tlj
1 0
(l.
Cf) 8
�
&l
4
>
0
1 000 2000 3000
LENGTH OF DATA S I Z E
KEY:
& WITH R EG - R E U S E
4000
0 NO R E G - R E U S E
5000
Figure 12 One-dimensional Fast Fourier
Transform Performance Graph,
Optimized Single-precision
Complex Transfonns
The small two-dimensional FFT can achieve per formance equal to that of the aggregate size one dimensional F FT by linearizing the data array.
Figure 15 shows the trade-off between using the lin earized two-dimensional routine (for smaiJ N) and the transposed method (for large N ) to maintain high performance across all data sizes.
The optimization of an algorithm that vectorizes poorly in its original form has been shown. The resulting algorithm yields much higher perfor mance on the
VAX vector 6000 Model 400 processor.
H igh performance is due to the unique way the algorithm touches contiguous memory locations and its effort to maximize the vector length . The implementation described above always uses unity stride vecrors and always results in a vector length of
64 for F FT lengths greater than 2 K (2 x 1024).
N 1 -0 FFTS
N 1 - D FFTS
COL U M N
ROW
Figure 13 Two-dimensional Fast Fourier
Transfonns Using N Column and
N Row One-dimensional Fast
Fourier Transfonns
Digital Technical journal Vol. 2 No.
2,
Spring
/'J<JO
25
VAX 6000 Model 400 System
N 1 - D FFTS
COL U M N
N 1 - D FFTS
COL U M N
TRANSPOSE
TRANSPOSE
Figure 14 Two-dimensional Fast Fourier
Transforms Using a Matrix
Transpose between Each Set of
Column One-dimensional Fast
N
Fourier Transforms
1 4
(l_
1 2
::::l
5J 1 0 w
BJ 8
� 6 f-
�
4
>
2
0 200 400 600 800
D I M E N S I ON OF PROBLEM S I Z E
1
000 1 200
KEY:
AI.
TRANSPOSED
0
L I N E A R I Z E D
Figure 15 Two-dimensional Fast Fourier
Transform Performance Graph,
Optimized Single-precision
Complex Transforms
Summary
The VAXvector 6000 Model 400 processor delivers high performance for computationally intensive applications. The CMOS-based VAX vector 6000 is capable of operating at peak speeds of 90 :\!FLOPs single precision and 45 MFLOPs double precision.
Linear algebra and signal processing applications that utilize the various hardware features have demonstrated vector speedups between 3 and 35 over the scalar VAX 6000 Model 400 CPU times.
With the integrated vector processing available on the VAX vector 6000 Model 400, the performance of computationally intensive applications may now approach that of supercomputers.
Acknowledgments
The authors would like to acknowledge the tech nical contribution of the following people: Paul
Brodeur, Doug Burns, Giao Dau, Bob Dickson,
Darrel Donaldson, Hugh Durdan, Bill Gist, Ani!
Jain, Chandrika Kamath, Dwight Manley, Mike
Pline, John Redford, Sean Reilly, Tim Stanley, and
M ike U h ler.
References
1 . ]. Croll,
"
VAX 6000 Model 400 Multiprocessor
System Overview," Proceedings of COMPCON
'90 (IEEE, forthcoming 1990).
2. D. Bhandarkar and R. Brunner, " Vector Exten sions to the VAX Architecture," Proceedings of
COMPCON '90 (IEEE , forthcoming 1990).
3. M. Gavrielov et al . "A 50 MHZ Uniformly Pipe l ined Floating-Point Arithmetic Processor,"
Proceedings of ISSCC '89 (IEEE, February 1989):
50-5 1 .
4 . ]. Dongarra,
Performance of Various Computers
Using Standard Linear Equations Software in a
FORTRAN Environment,
(Argonne, IL: Argonne
National Laboratory, June 1989).
5.
VAX FORTRAN Perfonnance Guide
(Maynard:
Digital Equipment Corporation, Order No.
AA-P B75A-TE, forthcoming 1990).
6. VAX 6000 Vector Processor Programmer's Guide
(Maynard: Digital Equipment Corporation,
Order No. EK60VAA-PG, forthcoming 1990).
7. C. Bischof et al . , LAPA CK, (Argonne, IL: Argonne
National Laboratory, April 1989).
8. ]. Cooley and J. Tukey, "An A lgorithm for
Machine Calculation of Complex Fourier Series,"
Mathematical Computing,
297-301 . no. 1 9 ( 1965):
9. P. Swartztrauber, "Vectorizing the FFT's,"
Parallel Computing (Academic Press, 1982):
5 1 -85.
26
Vol. 2 No. 2, Spring 1990
Digital Tecbnicaljournal
Patrick Sullivan
Michael A. Callander, Sr. james R. Lundberg
Rebecca L. Stamm lJ!illiam� ll�bill
The VAX 6000 Model400
Scalar Processor Module
The VAX 6000 klodel 400 CPU module is the latest generation of the compatible
VAX 6000 family of computers. The Model 400
is
a single-board, OvtOS-based CPU that significantly extends the performance of the VAX 6000 series. The system provides nearly
7
VAX units of performance {VUPs) in single-processor applications and up to
36 VUPs in six-processor systems. The Model 400 module
is
a plug-in replac emen t for the Model 200 a1ul Model 300 processors. Chip set and module designers of this new system cooperated closely to meet aggressive timing and performance goals. Several enhancements were made to the cache and bus interface units to improve multipro cessor petf orma nce.
A vector interface was included for connection to a companion vector processor module. Signal integrity was an important consideration for both chip and module design.
When the Midrange Systems Business (MSB) Group began to develop the
VAX
6000 series, the Semi conductor Engineering Group (SEG) had started development of a new CMOS-based CPU chip set.
1
The project's goals were the following:
•
Achieve CPU performance at least equal to the
'5 . '5 YUPs of the VAX 8700 system
•
Support a 28-nanosecond (ns) module cycle time
At that time, the
VAX
8700 system was the fastest
VAX available and used emiuer-coupled logic (ECL) to achieve a 45-ns cycle t ime. SEG designers believed a system implemented in CMOS technology could meet or exceed that performance level and at a much lower manufacwring cost . With reference to the second goa l . a 28-ns cycle time would take advantage of chip sets that could run faster t han the projected 40 ns cycle time.
In discussions with MSB , we realized our module project could be modified to include an XMI inter face and, therefore, become another member of the
VAX 6000 series 2 We then agreed to undertake a joint development effort between MSB and SEG , which resulted in the VA){ 6000 Model 400 scalar
C PU . Development of this scalar processor module is the focus of this paper.
Halfway through the project, support for a vector processor module was included because the VAX architecture was extended to include vector instructions:1 The vector processor module, which was developed by MSB, can provide a significant performance boost for certain classes of vector problems.
The first Model 400 systems were shipped in
July 1 989. This delivery was made just 15 months after the introduction of the initial 6000 Model 200 series.
Design Challenges
The aggressive 28-ns cycle time goal for the module design required a tight coupling of the chip set and module design efforts. With this short 28-ns cycle time goal, the module interconnect had to be treated as transmission lines. Consequently, signal integrity considerations were critical to design suc cess and would impact all areas of the design chip, package, and etch board . The approach taken to address signal integrity is described in more detai l in the Signal Integrity section .
The performance goals also dictated a change i n the
VAX
6000 Model 400 data and address line
(DAL)
pin bus. The older, less complex multiplexed 32-bit bus would have to be separated into a separate
27-bit address bus
(A-bus)
and a 64 -bit data bus
(D-bus).
That decision in turn resulted in the need for h igh pin count packages (224 pins) and the asso ciated signal integrity challenges of dealing with as many as 90 output drivers switching simulta-
Digital Technical journal
Vol. 2 No. 2, Spring 1990
27
2H
VAX 6000 Model 400 System neously. while driving transmission lines with a low effective impedance of 60 ohms.
Two new technologies were needed to meet these challenges. First, new chip packages had to be developed to supply the increased number of signal pins. The packages use multilayer ceramic sub strates to provide signal planes, as well as separate power and ground planes for both internal logic and pad ring power. The packages have 2 24 pins on a 25-mil pitch and are surface mounted for better module routability. Second. the 25-mil package pin pitch required the use of a finer geometry etch board : 1 3-mil module vias and 10-mil routing pitch .
This specification required the physical design team to initiate a very close development and qual ification effort with the etch board vendors. '
Major Module Subsections
The Model 400 module, or XR P , is a single-board
VAX CPU implemented with the Model 400 chip set and a REXMI interface to the XMI bus.2 The chip set includes five chips: a CPU chip (RFX 520), a floating point accelerator chip (F-chip), a backup cache controller chip (VC), a clock d istribution chip
(CLK), and a system support chip (RSSC). The R EXMI interface consists of three chips: two copies of the data path chip (XDP), and a controller/interface chip (XCA).
A block diagram of the XRP module is shown in Figure 1 . The module consists of four m ajor sections:
•
CPU and F-chip floating point accelerator
•
VC chip and backup cache RAM array
•
The RSSC system support chip
•
The XMI interface, including REXMI
The REX520 is the first VAX microprocessor chip to implement a fully pipelined microarchitecture.
The F-chip has (-,4-bit-wide data paths and a pipe lined execution unit. These chips cooperate to implement the base instruction group of the VAX architecture. The two chips represent the CPU sec tion of the XRP module, and both chips connect directly to the DAL . Both chips also have a private
8-bit bus for control and status information .
R EX520
CPU W I T H
CAC H E
CONTROL
DATA
�
D A C S S
BAC K U P
CONTROL
CAC H E
RAMS
VECTOR
I NT E R FACE
BUS (CABLE)
F-C H I P
DAL
I
CLOCK
I
RSSC
I
SYSTEM A N D A U X I L I A R Y CONSOLE
ROM
DATA
A N D
A D O R E
EEPROM
�
VC C H I P
l
I - B U S
I
R EXMI
I
XCI
I
I
I
, - -+- -
- - - -
-t
- - -
,
L - -
X CLOCK XLATCHES (7)
�
- - - - - -
�
- - - _j
I
:
I
X M I
COR N E R
X M I
Figure 1 XRP Module Block Diagram
Vol. 2 No. 2, .\j!rinp, /1.)')0
Digital Technicaljournal
The VAX 6000 Model 400 Scalar Processor Module
The REX520 provides the hardware and microcode necessary to parse specifiers, execute instructions, handle exceptions, and otherwise implement the VAX architecture. The REX520 also provides full VAX memory management, a
4 -gigabyte (GB) virtual address space, and support for 5 1 2 megabytes (MB) of physical memory . The chip contains a (">4-eorry, fully associative transla tion buffer. Both process and system-space map pings are stored in the buffer. The chip also includes a 2-kilobyte (KB), direct-mapped instruction and data cache (primary cache) with an 8-byte block and fill size.
The F-chip enhances the computation phase of floating point and certain integer instructions in conjunction with the R EX 520 chip. The F-chip exe cutes the F- , D-, and <�-format floating point instruc tions, as well as the long word variants of integer multiply . The chip receives operands from the
REX520, computes the result , and passes the result and status back to the HEX520. The REX520 chip completes the instruction.
The V\. chip implements a 2 K tag store and neces sary control for a 1 28KB backup cache. The cache uses 1 5 -ns, 16K-by-4 static random-access mem ories (SIV\Ms) on the module. The vc chip also includes a copy of the primary cache tag store, the invalidate bus (1-bus), and an interface to the vector interface bus (VI B). The 1-bus connects to the REXMI and boosts performance by doing invalidate filter ing. The cache design is discussed in more detail in the Model 400 Caches section.
The RSSC chip is a modification of the sse chip used in a number of previous \.PUs. It incorporates the common core of functions that support the chip set in the XMI system environment. The RSSC chip is discussed in detail in the Model 400 System Support
Chip section.
The XMl interface consists of the standard X M L corner components and the REXMI chip set. The
HEXMI chip set interfaces the DAL to the XCI , which is the user side of the XMI corner. The XMI interface is discussed further in the XM
I
interface and R EXM I section.
DAL operations are synchronous on the XRP module. Therefore, a very low skew clock distri bution system was required . The clock chip and controlled, equal-length clock lines to each of the chips provide this distribution. The clock chip receives a 1 43-megahertz (,\1Hz) oscillator input and provides two sets of synchronous four-phase clocks. Each clock phase driver output can drive four 50-ohm lines in paral lel . In the XRP design, there are eight chips that require all four phases, and each set of clock outputs drives four chip loads.
Discrete Schottky diodes are used at the receiving ends of a l l the clock lines for termination. This design achieved a clock skew of less than 0.5 ns at the receiving chips.
The DAL pin bus is a fully synchronous, hand shake protocol bus. The DAL is nonpended in that it has only one transaction outstanding at a time, and it is also nonmultiplexed w ith separate address and data lines. The design consists of a 27-bit address bus (A-bus), a 64-bit data bus (D-bus), and associated control signals. The OAL runs at a 28-ns cycle time, synchronous with the Model 400 chip set. The tim ing is controlled by 4 overlapping 1 4 -ns-wide clock phases, which are separated from each other by
7 ns. The P-chip, F-chip, vc chip, backup cache
RAMs, and the REXMI communicate through the
DAL . The P-chip is the default bus master and contains the arbiter for the bus. The P-chip uses the
OAL to initiate reads and writes to the backup cache and main memory through the REXM I . The backup cache and the R EXMJ send read data to the P-chip.
The bus master notifies other bus nodes of the start of a transaction. The receiving node can termi nate a transaction in one of three ways. It can indicate a successful completion, indicate an error, or request that the transaction be retried. Once the receiver indicates one of the transaction termina tion signals, the bus master deactivates.
One DAL transaction takes a minimum of three cycles. The maximum transaction time depends upon the system response to read and wri te requests. The RSSC includes a bus timeout mecha nism that prevents the system from hanging.
Most DAL signals are transferred in three phases, a lthough some control signals are t ransferred in two.
As noted earlier, vector interface capability was added after the VAX architecture was extended to support vector operations. The vector functionality was added to the Y\. chip because that chip could accommodate the extra pins required for the inter face, i.e. , 224 pins versus 164 .
The major units of the XRP module are described in the following sections.
Mode/ 400 Caches
The XRP module incorporates a two-level cache hierarchy that maximizes CPU performance. The first-level cache is the primary cache, called the
P-cache, which is contained entirely in the P-chip.
The second-level is the backup cache or B-cache,
Digital Tecbnicaljournal .!,
Sprin!( 1<)<)0
29
30
VAX 6000 Model 400 System and it consists of rhe vc chip and 24 1 5-ns, 16K-by-4 static HA:-.-ts. The vc chip contains the rag store and the control logic for the B-cache. The SRAMs store the cache data. The P-cache and the B-cachc contain both instructions and data.
The P-cache is a 2 KB cache and is direct-mapped, with an 8-byre block size. The cache is read allocate, no-write-allocate, and write-through . In write-through, writes that hit in the cache are simultaneously written to the cache and to main memory. The P-cache can perform a new access once every cycle.
Each P-cache rag entry includes an 18-bit tag, one valid bit, and one parity bit. There arc 256 tags cor responding to 256 data blocks. Each data block con tains 8 data bytes and 8 parity bits, w ith parity was implemented on each byte. Parity permits a byte write to be done without the need to recalculate parity across the other bytes. This process avoids the performance penalty that occurs when all byres are not written at once, as in the read-modify-write process.
The B-cacl1e is a 128KB cache and is direct mapped, with an 8-byte access size. It has a 16-byte sub-block size and a 64-byre block size. The cache is also read-allocate, no-write-allocate, and write through. There arc 2048 entries in the B-cache tag store. Each entry comains a 12-bit tag, 4 valid bits, and a parity bit.
Backup Cache Hit
Designers optimized access t ime to the backup cache by connecting the cache RAMs directly ro the
DAL . The chips, the bus, and the specialized hit sig nals are shown in Figure 2 .
When the P-chip issues a read o n the DAL , it drives the address on the A-bus to the vc chip and cache RAMs. The P-chip asserts memory read
(ME.\LR D_L). The VC chip uses MEivLRO_L to enable the assertion of the cache RAM chip select lines (BC_cs_ L< 7 : 0 > ). This process accomplishes two things. The chip select lines are valid by the time the P-chip has driven the A-bus to a val id state.
Further, the total time to perform the read from the
RAMs is minimized .
The RAM access begins in parallel with the tag store access. If the vc chip finds a match in the tag store, it asserts B C
_
H I T _L to notify the P-chip. By the time BC_ H !T_L is asserted, the data from the cache RAMs is valid on the 0-bus , and the P-chip then accepts the data.
If the vc chip does not find a match in the tag store, it asserts BC-MISS_L . This signal notifies the
F -C H I P
P-C HIP
24
1 6K-BY-4 SRAMs r{]·
. .
A-BUS
D-B US y
BC_CS_L<7:0>
BC_WE_L
I I
I
BC_HIT_L
MEM_RO_L
VC C H I P
�
BC_MISS_L
R E X M I
1--
1-----
X M I
Figure 2 Backup Cache Access Diaf!,ram
REXM! to send the read request to memory . At the same time, the VC chip asserts the cache RAM wrire enable (BC_ WLL) and waits for the data to return from memory . When the REX1vl ! returns the data, it asserts a transaction termination signal. This signal informs the P-chip, the
VC chip, and the F-chip that data is ready on the bus. All three chips receive the data simultaneously, and the data is written into the cache RAMs. The transaction ends when the P-chip accepts the data.
XMI Interface and REXMI
The XRP module accesses all memory and 1/0 devices over the XMI bus. The REXM! interfaces the chip set to the X M ! bus by means of two data path chips (XDPs) and one control/address chip
(XCA).
Each X DP is responsible for 32 bits of the H EX M I 's
64-bit data path. The XCA is responsible for the address data path, DAL control logic, X l\1 ! control logic, and control of the two XDPs. Both chips are implemented in CMOS-2 standard cells.
The primary tasks of the XMI interface are to
•
Forward REX520 references to the X l\1 1
•
Implement a write buffer that reduces traffic to main memory
•
Support control of cache fills and cache invalidates
•
Support Xl\1! interrupt logic
Vul. 2 No. 2. 5jJring 1<)<)1! Digital Technical journal
The VAX 6000 Model 400 Scalar Processor Module
CMOS-2 Standard Cell Technology
Custom chips were not used to implement the XCA and XDP chips. Instead, two alternatives were investigated . These alternatives were to use either gate arrays or standard cells from external vendors, or internally built CMOS-2 standard cells.
A number of external vendors offered products with the density and internal gate speeds needed to implement the REXMI chips. However, none offered the performance and flexibility needed to interface to the
DAL.
Therefore, we chose the C MOS-2 stan dard cells, which could interface to the
DA L .
We added a number of new cells to the standard cell library : several versions of a high-performance
3. 3-volt output driver, input latch with the required
0-ns set-up time, and a low skew internal clock buffer capable of driving over 30 picofarads of capacitance. These new cells were largely based on circuits designed for the custom CMOS-2 CPU chips, which met the high performance goals set by the
Model 400 program.
Perfo
rma
nce Considerations
Performance bottlenecks in high-speed computer systems most often occur in the path ro and from main memory . The X R P 's two cache levels greatly reduce the number of reads required. However, since both caches are write-through, all writes must be forwarded to main memory by the REX1'vll. To improve performance. the REXMI implements a write buffer with four octaword ( 1 6-byte) entries.
The write buffer improves performance in two ways. First, it decouples the REX520 write rate from the slower XMI write rate. The REX520 can transfer
8 bytes of write data to the
REXMI every three cycles. This data is loaded into the write buffer and later transferred to main memory at XMI speeds.
Second, the write buffer combines multiple
REX520 writes into a single X MI write. The most efficient write transaction is a full octaword write.
The R EXMI always tries to combine multiple
R EX520 writes into full octaword writes. The
REX.MI loads write data into a write buffer entry.
The write data is held umil either a new octaword address is received or a purge write buffer condi tion occurs. When the REXMI receives a write to a new octaword address, the current write buffer entry is marked fu ll. A new write buffer entry is then opened with the new octaword address. Write buffer entries are transmitted as they are marked full. To guarantee that the write buffer data is writ ten to main memory in a timely manner, the write buffer is flushed before the following conditions:
•
XMI l/0 space read or write
•
Interlock read or unlock write
• lnterprocessor interlock
•
XMI read to an octaword location that includes data contained in the write buffer
•
In response to a clear write buffer conunand
Combining P-chip writes reduces the number of write transactions needed by over 40 percent. This reduction certainly improves single-processor per formance. However, the greatest improvement is in multiprocessor performance where the XMI band width required by each CPU is reduced.
Cache Coherence
The XRP module allows data to be shared among mu ltiple processors. The XRP design assures that the most recently written copy of any data is pro vided to a running process. This process is called cache coherence.
In multiprocessing systems, coherence can be ensured in two ways. Cached copies of data that have been written can be inva lidated, or each cache can be updated with more recent data. Since it is simpler to invalidate than to update, an invalidation scheme was implemented on the XRP module.
Every X R P processor write is sent to XMI mem ory . When an XRP module broadcasts a write on the
X M I , the command, address, and data are captured in memory . Other XRP modules capture the com mand and address to invalidate any valid B-cache or
P-cache entries that correspond to the address.
We could have opted to broadcast all XMI writes as invalidates on the
DA L.
However, this method would have greatly increased the
DAL traffic and would have reduced the processor's performance.
To increase the performance of multiprocessor systems, the vc chip provides a low-overhead invalidate mechanism through the J-bus. The REXMI can determine through the !-bus if data is currently cached. The REXMI sends an invalidate on the
DAL only if the data is cached.
The vc chip maintains a duplicate copy of the primary cache tag store. The chip accesses the copy in parallel with the backup cache tag store whenever invalidate addresses are placed on the
1-bus. When an !-bus address match is detected in either tag store, the chip notifies the REXMl of the hit. The REXM! broadcasts the invalidate address onto the
DA L .
The invalidate address notification is recognized by both the VC chip and the REX520.
Digital Technical journal Vol. 2 No.
2, Spring 19<)0
3 1
VAX 6000 Model 400 System
The !'-chi p inv:.tlid;.tt<:s th<: !'-cache entry. and the vc chip inval idates the entries in both the B-cache and in the copy of the I'-cache tag store.
If the 1'-ctChe were a subset of the B-cache, we would not han: had to implement the P-cache tag stor<: copy on the VC chip. In this design. any address that hit in th<: B-each<: would ha\'e been sent to the DA L. hut this process wou ld h:t\T caused an in\'al idate request to be sent to the P-each<: as well.
However. \V ith this approach, the vc chip wou ld have ro send i m·alidares ro the P-cache whenever a
R-cache block was d isplaced . The transmission to the !'-cache wou ld have been required because the
1'-cach<.: and B-cachc block sizes ar<: diffcrt:nt. It was simpler to impl<.:mem the copy of the P-cache tags and irs control .
Anorhn alternative wou ld have lx:en ro use a dupl icate copy of the B-cache and I'- cache rag stores impll'mented i n cx tt:rnal logic on the module. The
R I'X:VI I then wou ld havt: interrogated the tag stores directly . This alternative was rejected because it would have used too much space on the module.
The Mode/ 400 System Support Chip
A \'ariety of su pport logic is required to complete the fu nctionality of a VAX C l'l i module. The Model
400 S\ S tt:m support chip ( RSSC) integrates the com mon core of funnions ncn:ssary to support the VAX system em·ironmt:m onto a single chip. It prov idt:s th<: operat ing system with the hardware primitives nt:nlu.J to implement the hoot a nd console rou tines. Tht: chip a lso pro\' idt:s st:n:ral nect:ssary tim ing mechanisms. Th<: RSSC is designed to interface direct ly with the Modcl · iOO chips. 1 It is based on t he systt:m support chip (SSe) t hat v.·as design<:d for use with tht: earl ier C . '<IOS-hased VAX systt:ms.'
The Vector Interface Bus
The vector intt:rbce bus connects the Model 400
C P l l mod uk to :�n opt ional vector modu k 5 The vector mod ule can perform fast ct'lculat ions on vector data. This capabi lity gready increases the execution spt:ed of certain applications.
There art: two levels to the i nterface. Th<: first is the microcode that implements the vector instruc tions defined by the VAX architecture. The second is the hardware imp lementation features required by the microcod<:.
Tht: R E X '; 2 0 's microcode vector support is l im itt:d to decoding vector instruct ions , p3rsing the sp<:c i fiers, and passing operand and control infor mation to the vector unit. The sc;�.la r CPU module dot:s not pass th<: actual vector elements ro the vector mod ule through the vector int t:rface. Ratht:r, the vector module rt:ft:rt:nccs t he ,.t:ctor data directly over the X :\11 bus. To perform this [XOC<:�s. the vector mod ule implemt:nts rh<: fu l l \'A X memory management ;�.rch itectur<:.
The opcodc and operand informat ion is tra ns ferred to the vector module through tht: \'ector i nterface The RF.X')20 uses in ternal processor rt:gis ters ( I I'Rs) for read and write operations to transfer the information ro the vector interface. I P Rs ar<: a lso used to read and w rite register information stored both in t he interface hardware and on the \'ector modu le.
The vector unit <:x<:cutt:s most vector instructions in parallel with the sc:�lar Cl'l l execlltion of subs<: quent i nstructions. For som<: v<:cror i nstructions, particularly memory transfer i nstruct ions , th<:
R EX ') 2 0 m icrocode reads a vector unit register at the end of the i nst ruction . At this point , the
R EX ')20 sta l l s until the vector unit r<:sponds and eff<:ctively forces synchronous <:xecution of instructions.
The hardware implementation of the vector i nterface consists of two pieces. The first is an i nt<:r face to the Model 400 DA L . This interfac<: al lows microcode-generated orcode, orcrand, and regis ter data to be received from ami driven to t he f{ f.X')20. The second p ice<.: is the vector interface bus ( V J B) that connects the vector and scalar mod uks together over
:1 cable. This interface and th<: connection to the V I B arc imp kmented in the VC chip on the scalar side and in the VECTL chip on the vector s ide.
Thc vector module clock sysr<:m is asynchronous to the scalar mod ule. The V!B runs synchronousl y with respect t o the \'ector mod ule clock system for design simplicity. The vc chip im rlements a l l t h<: asynchronous control l ogic that is required to trans fer data between the two clocking sysr<:ms.
Signal Integrity
The X R P design had very aggressi\'c rimi ng goals.
For example, 1 4 -ns and 2 1 -ns data t ransft:rs wt:re required to meet the 2H-ns ta rgt:t cycle time. These requirements were mad<: more difficult by the potential noise problems that could be caused by over 90 package. drivers swi tching simul tant:ously in on<:
Conseq uent ly, signal i ntegrity assu m<:d a major role in the XHP des ign . Signal in tt:grit y rroblems were compounded by fast CMOS-2 edge rates (i . c . ,
1 . '; ns). long interconnects ( u r to 2 5 i nch<:s w i t h as many as 8 loads per signal), and impedance mis matches associated with capacitive CMOS n-:c<:i vers.
Digital Teclm icaljournal
The VAX 6000 Mode/ 400 Scalar Processor Module
The XRP signal integrity methodology was developed early in the design process. All simula tions were made in the SPICE program (' Ideally, the entire module wou ld have been modeled . However, the complexity of the module environment and computer resource constraints precluded that approach . Instead, only one driver was used in the simulations. The effects of intersignal coupling and parallel switching were included by appropriately scaling interconnect and package impedances. Even with this simplified method, an estimated hours on a VAX
1 500
CPU
H700 system were required to per form rhe X R P signal integrity simulations.
Simulations and Models
A typical simu lation included all circuits and any accompanying parasi!ICS from the external enabl ing clock edge through to the receivers. A ll simulations modeled a three-stage clock receiver, on-chip resistance capacitance delay, the output driver, package impedance models for signals and internal/external power, receivers, and module interconnect.
The clock receiver model generated the on-chip clock phases from rhe low-skew double phases received from the clock chip. The model also con tained an internal switching model. It switched several capacitances that were similar to the chip internal loading. Over a cycle, this modeling met the chip's maximum internal power. The switching model, together with the chip package impedance model for internal power, produced worst-case noise on rhe chip internal power rails and substrate.
The module interconnect was modeled as trans mission lines. The impedance of a transmission line accounted for the coupling of adjacent signals switching. For example, if adjacent lines were switching in the same direction, the effective impedance would be increased; whereas switching in the opposite direction would lower the effective impedance.
Finally, the signal and external power impedance models were scaled to reflect worst-case coupling, and size and number of drivers switching in paral lel . Collectively, these models accounted for the effects of an entire hus switching.
Worst-case Conditions
There are two distinct subsets of the simulation models. One set simulates worst-case signal noise or ringing. The other simulates worst-case, i . e . , slow est, settling times. These subsets are, for the most parr, mutually exclusive. In other words, paramerers that rend to suppress ringing generally increase settling t imes and vice versa.
The overall design had ro satisfy the criteria for achieving specified settling times and simulta neously reduce ringing to an acceptable level.
Several modeling parameters were simulated to worst -case starus. These parameters are listed in
Table 1 . The impact of these parameters can be fur ther understood by referring to
Figure 3 . This figure represents the same simulation under worst-case slow and fast conditions.
Table 1 Worst-case Modeling Parameters
Parameter
Simulation
Worst-case Worst-case
Settling Noise
CMOS process corner Slow
CMOS j u nction temperatu re
Max i m u m
I nternal power supply M i n i m u m
Output series resistor Max i m u m
N u mber o f d rivers switch i ng
Maxi m u m
Maxi m u m Module interconnect length
I ntercon nect effective ZO
M i n i m u m
Fast
M i n i m u m
Maxi m u m
M i n i m u m
Max i m u m
Maxi m u m
Max i m u m
6.0
5.0
4.0
(f)
3.0
_J
0
>
2 . 0
VIH
1 .0
WORST-CASE SETT L I N G T I M E
(SLOW)
- 1 . 0
":---:'-::--�-
45 50 55
�
60
-!:--::'::-----'--::':-----:'-__J
65 70 75 80 85 90
NANOSECONDS
KEY
V I H - I N PUT VOLTAGE H IG H
V I L - I N PUT VOLTAGE LOW
Figure 3 Simulation under Worst-case
Slow and Fast Conditions
Digital Tecbnica/journal Vol .! No . .!, .�fJrin[!, f<J')I! 33
VAX 6000 Model 400 System
Signa/ Integrity Constraints
The signal integrity analysis had a direct impact on the design of the scalar CPU at the chip. package, and module levels. The impact at the chip kvel was seen in three areas. The first was the design of the
I/O buffers. The second was the determination of the optimal series output resiswrs (on-chip) for different drivers. The third was the segregation of the external power buses to eliminate noise at quiescent drivers.
At the package level, two design decisions were made. The external power reference planes were split to prevent coupling of asynchronous buses.
Power was delivered to the lower bonding tier to reduce power supply loop inductances. At the mod ule level, several specifications were developed, which included the following:
•
All discrete terminations and their placement
•
The maximum allowable etch length per signal and order of connection
•
The maximum allowable package dispersion etch length
•
The module etch technology tO best reduce coupling
Cross-organizational cooperation was essential to the successful production of these design levels.
For example, a joint review of the chip packaging technology in the design stage ensured that the design met stringent signal integrity requirements.
Working closely with the module PC designer ensured optimal component placement and inter connect routing. Cooperative efforts such as these helped ensure the reliability and performance of the design .
Results
Two correlations of bench versus simulation results were used for verification purposes. The first corre lation was run on a test module early in the design phase. The difference between simulated and bench results averaged 4 .4 percent. The correlation on the final XRP implementation presented an average dis crepancy of 2 percent, which is less than 200 pico seconds (ps). These results strongly validated the modeling methodology. Figures 4 and 5 show the results of the bench and simulation of the REX520 driving the 0-bus. The waveforms exemplify the excellent correlation obtained on the final module.
(Note: The results shown in these figures represent nominal rather than worst-case conditions.)
�
>
7.0
6.0
5.0
4.0
3 01---+----1-
2.0
1 .0
0.0 1--+--+--
KEY:
VIH - INPUT VOLTAGE HIGH
VIL - I N PUT VOLTAG E LOW
43 200
Figure 4 Bench Results of the REX520
Driving the
D-bus
6.0
5.0
4.0
�
3.0 t:::;
� 2 0
1 .0
VIH
0.0
-1.0
L_�'-------'------'-----'------'----'----'----'---'
NANOSECONDS
KEY
VIH - I N P U T VOLTAGE HIGH
VIL - I N P U T VOLTAGE LOW
Figure 5 Simulation Results of the REX520
Driving the D-bus
Signal integrity had a major impact on the design, performance, and reliability of the Model 400 scalar
CPU. All critical signals were carefully simulated and analyzed prior to chip and module implementa tion . The Model 400 is currently the fastest VA X system in production . The Model 400 also has the distinction of needing no revisions, from prototype tO final product, for signal integrity purposes.
Performance
The VAX 6000 Model 400 system , based on the XRP module, represents the highest performance VA X system yet released . Performance ranges from nearly 7 VUPs in a single-processor system to
36 VUPs in a six-processor system. The VAX 6000
34
Vol.
2
No.
2, Spring 1990 Digital Technicaljountal
The
VAX
6000 Model 400 Scalar Processor Module
Model 4 10 system provides roughly twice the performance of the previous generation
CMOS based system, the VAX 6000 Model 3 10. It also represents a 28 percent performance gain over the previous generation
EC L system , the VAX 8700 sys tem. (Note: One V U P is equal to the performance of the VAX- 1 1 /780 system.)
Table 2 compares the performance of the
VAX 6000 Model 4 10 system to other VAX systems.
Table 2 Single-processor Performance
Comparison
System
VUPs
VAX 1 1 /780
VAX 6000 Model 2 1 0
VAX 6000 Model 3 1 0
VAX 8700
VAX 6000 Model 4 1 0
1 .00
2.53
3.38
5.20
6.64
Up to six XRP modules may be configured in a single VAX 6000 Model 400 system. These modules deliver up to 36 V UPs. Ideally, a linear performance increase is expected as more processors are added to a multiprocessor system. However, a number of factors l imit the overall system performance, such as contention for bus bandwidth, increased mem ory latency, and additional software overhead.
A great deal of effort was expended in the
Model 400 design to limit the amount of perfor mance lost in a multiprocessor system.
A number of multistream benc h marks were assembled. These benchmarks were run on the
VAX 6000 Model 400 system to efficiently measure performance by simulating real work environments across a number of areas. The results for each system are shown in Table 3.
Table 3 Mu lti processor Performance
Comparison
Work Area 1 CPU 2 CPUs
4 CPUs
6 CPUs
Engineeri ng
Scientific
Commercial
1 .00
1 . 00
1 .00
1 .92
1 . 92
1 . 98
3.71
3.74
3.80
5.31
4 . 78
5 . 25 ule development effort: Dick Bagley, Jean Basmaji,
Rick Calcagni, Glenn Garvey, John Guildoo, Andy
Ladd, Bill LaPrade, Bob McCarty, Dina McKinney,
Curt Miller, Roland Ouellette, Dave Rouille, John
Sweeney, M ike Uhler, M i ke Warren, and Joh n
Wilson.
References
1 .
H . Durdan et al. , "An Overview of t he VAX
6000 Model 400 Chip Set," Digital Technical journal, vol. 2 , no. 2 (Spring 1990, this issue):
36-51 .
2 . B. Allison, "An Overview of the VAX 6200 Family of Systems," Digital Technical journal, vol. 1 , no. 7 (August 1988): 10- 18.
3. D. Slater et a l . , "Vector P rocessing on the
VAXvector 6000 Model 400,"
Digital Technical journal,
vol. 2 , no. 2 (Spring 1990, this issue):
1 1 -26.
4. ]. Bartoszek et a\., "VAX 6000 Model 400 Physical
Technology," Digital Tecbnicaljournal, vol. 2 , no. 2 (Spring 1990, this issue): 52-63 .
5. ]. Winston, "The System Support Chip, a Multi function Chip for CVAX Systems,"
Digital
Technical journal,
vol. 1 , no. 7 (August 1988):
1 2 1 - 128.
6.
SPICE is a general purpose circuit simulator program developed by Lawrence Nagel and Ellis
Cohen of the Department of Electrical Engi neering and Computer Sciences, University of
California, Berkeley.
Acknowledgments
The authors would like to acknowledge the follow ing people for their contributions to the XRP mod-
Digital Tecbnical]Otlrnal Vol.
2 No. 2, Spring 1990
35
W.
Hugh Durdan
William]. Bowhill john
F.
Brown
William
V.
Herrick
Richard
C.
Marcello
Sridhar Samudrala
G.
Michael Uhler
Nicholas Wade
An Overview of the VAX 6000
Model 400 Chip Set
The VAX 6000 Model 400 processor is a
CMOS implementation of Digital's
V,4X architecture, offering an average of seven times the performance of the VAX-111780 processor at a cycle time of 28 ns. The processor comprises five custom chips imple mented in Digital's proprietary
CMOS-I and
CMOS-2 semiconductor processes. The chip set design incorporates the best features of the previous VAX 8700 and VLSI VAX designs and in addition implements new performance features. Among these are a larger translation buffer and primary cache, a de-multiplexed 27-bit address and
64-bit data bus, and a tight�y coupled 128KB backup cache. The five chips, which are designed for multiprocessing environments, are the REX520
CPU, the floating point accelerator, the vc vector and cache controller chip, the
RSSC systen1 support chtjJ, and the
CLK clock chip.
Introduction
The VAX 6000 Model 400 chip set consists of five custom VLSI (very large scale integration) chips implemented in Digital's CMOS- I and CMOS-2 pro cesses. The five chips are the CPU chip {REX520), the floating point accelerator chip {F-chip), the vec tor and cache controller chip (VC chip), the system support chip (RSSC), and the clock chip (CLK chip).
These chips are designed to be used in multiple system environments, of which the VAX 6000
Model 400 series is one such example. '
T he REX520 chip i s a pipelined VAX CPU that implements the VAX base instruction group and controls the operation of all other chips 2 The design for the REX520 is an evolution of both the previous generation CMOS processor chip and the
VAX 8800 processor:H The REX520 is logically divided into four sections: the 1-box, E-box, M-box, and bus interface unit (BtU). The 1-box fetches and decodes VAX instructions, and provides this infor mation to the E-box. The microcode-controlled
E-box parses instruction specifiers, executes VAX instructions, and processes interrupts and excep tions. The M-box contains a 64 -entry, fully associa tive translation buffer, and a 2-kilobyte (KB) on-chip primary cache. The BIU acts as the interface between the REX520 and the interchip environ ment described below.
The F-chip is a companion chip to the REX520.
It accelerates the computation phase of the VAX F- ,
D-, and G-format floating point instructions, and the longword-length integer multiply instruction.
The F-chip receives control information from the
REX520, operands from either the REX520, the backup cache, or memory, and returns status and results to the REX520.
The vc chip provides the tag store and necessary control for a 128KB backup cache that is imple mented in external random access memory
(RAM) on the CPU module. The chip implements a dupli cate tag store for the REX520 primary cache and an interface through which the system environment can determine if data is cached at a particular address. Through a vector interface bus (V I B), the vc chip also provides the control and status inter face between the CPU module and an optional vector module.
The RSSC chip incorporates the common core of functions required to support the VAX 6000
Model 400 chip set in a system environment. RSSC supports read-only memory (ROM) and electri cally erasable programmable read-only memory
(EEPROM), and contains 1 KB of battery hacked-up
RAM, the console terminal universal asynchronous receiver/transmitters (UARTs), interval and pro grammable timers, and a time-of-year clock.
36
Vol. 2 No 2, Spring 1990 Digital Tecbnicaljournal
An Overview of the VAX 6000 Mode/ 400 Chip Set
The CLK chip receives a 143-megahertz (MHz) osciUator input. The chip provides four low-skew clock phases to the chip set and the module environment.
Interchip and Module Environment
The chips in the Model 400 chip set are connected together and to the rest of the module environment a5 shown in Figure 1 .
Unlike the previous generation CMOS processor design
,"
the Model 400 chip set implements sepa rate 27 -bit address and 64-bit data buses. Backup cache reads, writes, and fills are done with the address on the A-bus. The data is driven to or received from the parity protected D-bus. Control for the backup cache RAMs is provided by the vc chip on dedicated control lines.
Memory ,
110 space, and external processor regis ters are accessed by driving the address to the A-bus and the data to the D-bus. ROM and EEPROM control is provided by the RSSC on dedicated control Jines.
F-chip operands are driven to the D-bus from the
R EX520, backup cache, or memory to the F-chip.
Results are driven back to the R EX520 on the 0-bus.
Control and status information for these transfers is performed on a private bus between the REX520 and F-chip.
Vector instructions are decoded by the REX520.
The opcode and instruction operand information is transferred from the R EX520 to the vc chip. From there, the information is transferred to the V!B cable and then to the optional vector unit. Note that only status and scalar operands (contained within the instruction stream) are transferred on the V I B . Data transfers into and out of the vector register file are performed by the vector processor through a direct port to the memory subsystem .
Interrupt requests are received by the REX520 from the module environment on nine dedicated interrupt request lines. Five of these Jines are for requests for special purpose interrupts, such as interval timer requests.
Performance Goals and Design
Considerations
The goal of the design was to meet or exceed the performance of the v AX 8700 processor. To meet this goal, a 40-nanosecond (ns) cycle time was required under worst-case conditions. As the design progressed, it became clear that the CMOS-2 pro cess, in which most of the chip set is implemented, offered enough performance to allow the target cycle time to be decreased.
As a result, the cycle time was reduced from 40 ns to 28 ns. At this cycle time, the VAX 6000 Model 4 10 system runs at nearly 7 VUPs in most applica tions, or roughly 1 .3 times the performance of the
VAX 8700 processor. (The acronym vurs stands for
VAX units of performance; 1 VUP equals the perfor mance of a VAX- 1 1/780 system.) The performance of the system may be further expanded by adding processors to the system, to a maximum of 35 VUPs in the VAX 6000 Model 460.
To achieve the performance goals, a number of microarchitectural trade-offs were made relative to the VAX 8700 and previous VLSI VAX designs. I n essence, the best features o f each were incorporated into the design of the VAX 6000 Model 400 chip set .
A-BUS
D-B US
INTERRUPT
REQUESTS
CLOCKS
1 43 MHZ
OSCI LLATOR
I-BUS
VIB CABLE
Figure I
VAX 6000 Mode/ 400 lnterchip Environment
Digital Technical journal
Vol.
2
No.
2, Spring
1990 37
VAX 6000 Model 400 System
For example, the REX520 chip and the F-chip are fully pipelined designs that make shorter cycle times possible and improve performance. The
REX520 design also includes read-and-run and write-and-run features. These features decouple subsequent execution in the CPU pipeline from the completion of references to cache and memory. As opposed to previous VLSI designs, a larger transla tion buffer and primary cache and a de-multiplexed
27-bit address and 64-bit data bus in the REX520 also improve performance. The tightly coupled
1 28KB backup cache significantly reduces the read latency seen by the CPU when a read misses in the primary cache. It also reduces the read traffic seen by the memory subsystem .
The chip set was designed from the beginning with multiprocessing in mind. Because caches must remain coherent across all CPUs in a syste m , a method must be provided to invalidate cached loca tions in all other caches when one CPL writes to that location. One option would have been tO mir ror all writes done by any CPU in the system onto the A-bus of all Cl'l·s. However, mirroring is a rela tively expensive operation, especially when most of these addresses are not cached in any other C P U .
Instead of the mirroring method, w e chose to implement a duplicate copy of the primary cache tag store in the vc chip and to implement a low overhead port, the 1-bus. The module environment uses the 1-bus to determine if the address is actually cached by either the primary or the backup cache.
With this method, only those invalid addresses that correspond to cached locations must be mirrored onto the A-bus.
CMOS and Packaging Technologies
The chip set was implemented in Digital's proprie tary second-generation complementa ry metal oxide semiconductor technology, CMOS-2 . We selected CMOS as the chip processing technology because it offers high density, high rel iabi li ty, low power, and low-cost performance.
Table 1 Summary of Chip Statistics
By developing the technology in-house, Digital has gained competitive performance and time-to market advantages for its low-end and midrange products. A summary of the transistOr and pin count statistics for the chip set is shown in Table I .
CMOS-2 is an N-wel l , P-epitaxial, double-metal,
5-volt process with 1 .5 micron minimum feature sizes. With respect to CMOS- I , the ftrst generation process, CMOS-2 offers a 25 percent reduction in lateral and key vertical dimensions, a 78 percen t improvement in circui t density, and nominally a
33 percent improvement in chip performance. The
VAX 6000 Model 400 chip set cycle time and cir cuit density requirements helped drive the develop ment of the process that has been optimized for microprocessor chip performance. Details of the process features and capabilities are shown i n
Table 2 .
Metal oxide semiconductor field effect tran sistors (MOSFETs) are built in a P-epitaxial layer
(30 ohm-em) grown on a low-resistance P+ silicon substrate (0.02 ohm-em). The high resistance of the epitaxial layer keeps parasitic junction capacitance low and allows better transistors to be fabricated.
The low-impedance substrate dramatically reduces latchup, a phenomenon in which parasitic bipolar transistors are triggered into a sustained high cur rent mode. Latchup disrupts normal circuit opera tion and often destroys the chip.
The P-channel MOSFETs are made in the N-doped well regions of the epitaxial layer. The N-channel
MOSFETs are made in the as-grown P-regions of the epitaxial layer. This process optimizes the mobility of the N-channel MOSFETs and overall circuit speed.
An N+ polysilicon (polycrystalline silicon) and tungsten-disilicide sandwich material, polycide, forms the MOSFET gates. Polycide resistance is in order of magnitude lower than that of the poly silicon used in CMOS- 1 . The resulting smaller para sitic delays across the MOSFET gates and local interconnections help improve circuit speeds.
Chip
REX520 CPU
VC cache control ler
F-chip floating point
CLK clock
Signal
Pins
1 57
1 78
1 03
20
Die Size
1 2m m x
1 2m m
1 0 .7mm x
1 0 . 8 m m
1 2 .7mm x
1 1 m m
5mm x
5mm
Power
Dissipation
6W
2 . 5W
4W
2W
Total:
Transistor Cou nt
Memory
Control
&
Data Path
1 80K
1 84 K
364K
1 40K
34K
1 34K
5K
3 1 3 K
38
Vol 2 No.
2, Spring
1990
Digital Tecbnicaljournal
An Overoiew of the VAX 6000 Mode/ 400 Chip Set
Table 2 CMOS-2 Process Features and Capabilities
Effective channel length
Metal 1
Metal
1 contact
Metal 2
M etal 2 contact
Gate oxide thickness
Metal 1 field oxide
M etal 2 field oxide
Polycide resistance
Typical gate delay
Polycide equ ivalent gate delay length
Metal 1 equ ivalent gate delay length
Metal 2 equ ivalent gate delay length
0.9 m icrons
3.0 micron w idt h
,
1 . 5 m icron space
1 . 5 micron x
1 .5 m icron
3.75 micron wid t h , 1 . 5 m icron space
1 .5 mic rons x
1 . 5 microns
225 angstroms
1 . 1 m icrons
1 .9 microns
2-4 ohms/sq uare
300-500 picoseconds
400 mic rons
5000 m icrons
8000 m icrons
O ther interconnections are accomplished through two layers of aluminum. The first layer, metal 1 , can connect to either polycide or the
N+/P+ source/drain regions via metal 1 contacts.
The upper layer, metal 2, can connect to metal I through metal 2 contacts. No other connections are allowed . Based upon its parasitic delay charac teristics, polycide is used only for local interconnec tions. Metal 1 is used for signals communicating across distances less than half of the chip dimen sions. Metal 2 is used for global signals, clock dis tribution, and power and ground distribution.
Because of the speeds and complexities of the
VAX 6000 Model 400 chips, noise is a particularly difficult problem for the chip designer. CMOS-2 contains a deep P+ implant that can be used to provide a very low resistance connection between the tap surface of the chip and the P+ substrate. The chip designer uses this deep P+ implant to reduce substrate noise that can upset the operation of dynamic circuits.
The VAX 6000 Model 400 chips were packaged in custom-designed, rigid perimeter-leaded, single chip ceramic packages. The packages included four power and ground planes to help maintain inter chip signal integrity and allow full-speed operation of the chip set.
'i
The REX520 CPU Chip
The R EX520 CPU chip is a third-generation, single chip VAX m icroprocessor. The R EX520 provides the hardware and microcode sufficient to parse operand specifiers, execute instructions, and han dle interrupts and exceptions. It cooperates with the F-chip to implement the base instruction group of the VAX architecture.
Although the R EX520 hardware organization and placement resemble that of the previous gener ation microprocessor, the REX520 performance deviations from the microarchitecture of the ECL based V�'< 8700 system were made in the CMOS based REX520 to compensate for technology differences and to exploit the beneficial aspects of
VLSI design. A photomicrograph of the R EX520 is shown in Figure 2 .
The chip employs a si..'c-level pipelined engine built around three autonomous pipes. These pipes provide simultaneous instruction prefetch and decode, instruction formatting, operand reference, execution, address translation and result stare, and
110 access.
As shown in Figure
3,
the major hardware functions of the R EX520 are partitioned into the following:
•
An instruction box (!-box) that contains the instruction decoder and a 16-byte prefetch queue (PFQ)
•
A microcode-controlled execution box (E-box) that provides the capability for data manipu lation in a 32-bit data path
•
A memory box (M-box) that implements VAX memory management by utilizing a 64-entry, fu lly associative translation buffer
•
A 2KB write-through, direct-mapped primary cache (P-cache) with a quadword (8 byte) fiU size
•
A bus interface unit (BIU) that controls a fully handshaked, synchronous chip bus
Digital Technical journal
Vol. 2 No. 2,
Spring
1990
39
VAX 6000 Model400 System
•
•
•
•
•
.
.
:
.
: :
Figure 2 Photomicrograph of the REX520 Chip
REX520 Pipeline
The REX 520 pipeline contains si,x functional seg ments that cooperate in the execution of instruc tions. As shown in Figure 4 , there are two segments in the !-box, three microcode-controlled segments in the E-box and M-box, and a single segment for the
BIU and P-cache control.
The !-box decodes the VAX instruction strea m .
A 16-byte prefetch queue (PFQ) is filled with the instruction stream asynchronously to the pipeline control during otherwise unused bus cycles. !-box segment 1 updates the PFQ and parses the next piece of the instruction stream. This segment sends microcode addresses to the E-box microsequencer.
Segment 2 formats immediate data, the opcode, and the instruction data length for the E-box.
Collectively, this information is called the context for the instruction.
The !-box divides each VAX instruction into a microcode subroutine, or microflow, for each specifier, and a microflow for the execution of the instruction. The control programmed logic arrays
( P LAs) in the !-box cause it tO sequence through the specifiers of each instruction, sending a microflow address to the m icrosequencer, and immediate data
(if needed) to the E-box for each specifier. When the last specifier is parsed , the !-box sends a microflow address, the opcode, and the data length of the instruction ro the E-box for the execution of the instruction.
40
Vol. 2 No.
2, Spring 1990 Digital Technical journal
An Overview of the VAX 6000 Model 400 Chip Set
M I CROFLOW
A D D RESS
MICROI NSTRUCTION
I N STRUCTION
1---1-
CONTEXT
I N T E R C H I P BUS
Figure
3
REX520 Block Diagram
The !-box pipeline runs autonomously to the
E-box pipeline. That is, the 1-box segments con tinuously parse instruction stream data, making microt1ow addresses available. The I-box pipeline advances whenever the microsequencer accepts a microtlow address.
The microsequencer in the E-box performs a read of the 1696 word control store each cycle, overlap ping in time with 1-box segment
2 .
It presents the
fetched microinstruction to E-box segment 1 . The microinstruction is pipelined forward to E-box seg ments
2 and 3 in consecutive cycles.
The microsequencer fetches one or more microinstructions of a microflow starting at the initial microflow address supplied by the !-box.
If a microflow contains more than one micro instruction, the microsequencer computes subse quent intrafl.ow microaddresses and fetches the corresponding microinstructions. A field within t he microinstruction indicates the end of each microflow.
The E-box performs all the address and data manipulation required for the R EX520 to adhere to the VAX architecture. The three E-box segments operate under microinstruction control; the operand fetch segment reads operands from gen eral-purpose registers (GPRs) or from the memory data (MD) file and presents them to the functional units; the execution segment performs data manip ulation on the operands; and the result store seg ment writes results to registers or memory. The
E-box pipeline segments are fully folded, i .e., each segment simultaneously operates on a different microinstruction.
The performance of memory read accesses is improved using a read-and-run technique in the
E-box. The destination for data stream memory reads is the E-box MD file. Microinstructions that initiate a memory read simply queue the request to the B I U . The microfl.ow may then continue, without waiting for the request to complete. In this manner,
I-BOX
S E G M E N T
1
I-BOX
S E G M E N T
2
I N STRUCTION
PARSE A N D
DECODE
f.--
I N STRUCTION
DATA A N D
CONTEXT
FORMATT I N G
f-
M IC ROFLOW
A D D R ESS
G E N E RATION
CONTROL STORE
ACCESS
E-BOX/M-BOX
S E G M E NT
1
OPERAND
FETCH
E-BOX/M-BOX
S E G M E N T
2
E-BOX/M-BOX
S E G M E N T
3
EXECUTION
,._
VIRTUAL
A D D R ESS
SELECTION A N D
TRANSLATION
,.
R E S U LT STORE
ACCESS
C H E C K I N G
BI U/P-CACHE
S E G M E NT 1
BUS
ARBITRATION
C AC H E ACCESS
Figure 4 REX520 Pipeline Segments
Digital Tecbnicaljournal
Vol. 2 No. 2,
Spring
1990
4 1
VAX 6000 Model 400 System some or all of the time required for memory accesses is hidden during productive microflows.
The request completes when the BIU and memory subsystem return data to the MD file.
Because subsequent microinstructions may refer ence an MD file location for which a memory read has not yet completed, MD file accesses are syn chronized through a valid bit mechanism. Each lvi D file location has a valid bit that is reset when a mem ory read is started. The bit is then set when data is written to the file. If a microinstruction attempts to reference an MD file location whose valid bit is not set, the pipeline stops advancing (stalls) and waits for the data to be returned .
T he three M-box segments are m icrocode controlled and typically run synchronously with the E-box segments. M-box segment 1 decodes the current microinstruction. Segment 2 then selects an appropriate address source and performs a vir tual to physical address translation if required. The final segment issues the physical memory request to the BIU .
During normal pipeline flow, the BIU and P-cache operate in a single pipeline segment w hich is over lapped with segment 3 of the E-box/M-box pipeline.
The BIU acts as an arbiter for the external and some internal buses, and supplies control to the P-cache.
The BIU receives address and memory request information from the M-box. The BIU then decides whether the M-box or an external reference should receive service from the P-cache and sends the appropriate information to the P-cache to process the request . The BIU sends read and write requests that cannot be serviced by the P-cache to the off chip memory subsystem.
Write buffers in the BIU improve the perfor mance of memory write operations. Microinstruc tions that initiate memory writes queue the request to the BIU, and the microflow continues without waiting for the request to complete. In this manner, the time required for memory writes can be hidden under productive m icroflows. The BIU can store up to two quadword writes (32 bytes) in its buffers while waiting for the memory subsystem to become available.
Adaptations for VLSI
VLSI designs have relatively good signal integrity on the major bus structures within a chip. However, signal integrity diminishes when crossing a chip boundary because off-chip bandwidth is incapable of sustaining the internal data rates. This character istic of VLSI design influences microarchitecture greatly, as demonstrated by the P-cache and control store designs.
The VAX 8700 system uses cache memory to decrease the effective memory access time by employing a 64 KB cache in ECL RAM with a single cycle access. It was not possible to implement a
RANI of this size on the REX520, and signal integrity constraints dictated a minimum of three cycles to access off-chip RAM . The REX520 compensates by using a two-level cache. The REX520 has a 2KB pri mary cache on the chip with a single-cycle access, and there is a 128KB off-chip secondary cache. The cache hierarchy reduces the effective memory access time sufficiently to meet the performance goals, and it keeps the off-chip bandwidth within reasonable rates.
Similar signal integrity constraints are imposed on the REX520 control store design. The VAX: 8700 system uses a 1 6K entry control store implemented in ECL RAM . The VAX 8700 system designers use the large control store address space to improve per formance and simplify hardware. The control store access of the VLSI REX520 canno t cross a chip boundary and keep pace w ith the fast internal cycle time. Because of these limitations, an on-chip con trol store ROM was used. Chip area limitations con strained the size to 1696 entries. The designers of the REX520 compensated for the lack of control store address space by altering the coding style to make microflows more serial than the v AX 8700 system microflows. Additional hardware features also helped to absorb some of the work previously done in microcode.
The operand specifier m icroflows offer an exam ple of microcode space compression facilitated through hardware enhancements. The REX520 microflows group similar specifier types together and provide a general routine for each. The general routines are parameterized in hardware to resolve the differences in specifier types within each group.
In this way, the size of the m icroflow is minimized, and the performance characteristics of a dedicated microflow are preserved .
Each general specifier microflow performs the function appropriate to t he specifier mode by using a parameterized access type and data length . For example, a microinstruction in the specifier flows may make an access-type dependent memory request. If the 1-box supplied a read access type, the memory request hardware initiates a read. If the
!-box parsed a result store specifier, the write access type parameter causes hardware to initiate a write.
Operand data length is supplied in a sim ilar fashion .
42
Vol. 2 No. 2, Spring 1990
Digital Technical journal
An Oven;iew of the VAX 6000 Mode/ 400 Chip Set
The general microflows leave operands in fixed locations in the E-box MD file. The exact location in the MD file is also supplied by the !-box.
Parameterizing the microflows w ith the access type, data length, and MD file location allows the generalized specifier microflows to be shared.
The MD file parameter creates an independence between the specifier and execution microflows.
The same specifier microflow can then be used for each speci fier mode, independent of w here it actually appears in the instruction . The execution microflow also needs to know nothing about the kind of specifier that supplied the operand.
Tbe F-cbip
The floati ng point accelerator chip, F-chip, is the companion processor chip to the REX520 for floating point operations in
VAX
6000 Model 400 systems. A photomicrograph of the chip is shown in Figure 5.
The F-chip implements all VAX base instruction group floating point instructions and the longword
Figure 5 F-chip Photomicrograph
Digital Tecbnicaljournal Vol. 2
No. 2, Spring 1990
43
VAX 6000 Model 400 System integer multiply instruction. The data types imple mented by the F-chip are F-floating ( 1 -bit sign, 8-bit exponent, 24-bit fraction), D-floating ( I -bit sign,
8-bit exponent, 56-bit fraction), G-floating ( ! -bit sign, 1 1 -bit exponent, 53-bit fraction), and integers
(8-bits, 16-bits, and 32-bits).
Unlike the VAX 6000 Model 200 floating point chip, the F-chip employs a uniformly pipelined architecture that has performance which is inde pendent of the operand values. Early in the design it was decided to use a pipe l ined m icroarchitecture in order to use the same execution core in future designs, such as the VAX 6000 Model 400 vector processor floating point unit. Consequently, the execution core of the F-chip was designed to include low latc.:ncy for scJiar applications and high throughput for vector pron:-ssing (' The core exe cutes most instructions in four cycles. Double precision and integer multiply instructions take five cycles, and divide operations take 13 to 24 cycles, depending upon the datJ type .
Microarchitecture and Implementation
The F-chip is composed of an interface section and a five-stage execution core. A block diagram of the
F-chip is shown in Figure 6. The F-chip interfaces to the rest of the VAX 6000 Model 400 system through the D-bus , eight status and control lines, and four bus control signals. The imerface section receives the opcode from the REX520, the operands from the CPU chip, and cache and memory data on the
D-bus. The interface section decodes the opcode and receives the required number of operands. It assembles the different pieces of the operands and supplies formatted operands to the execution core.
After the execution of the floating point operation , the output interface transfers the formatted result back to the CPU chip.
The execution core consists of a divider, which is bypassed in all operations except division, and four pipelim:-d stages that are uniformly utilized in the execution of all instructions. Each stage has a frac tion data path, control , and sign and exponent data
44
J
D I V I D E R
I N PUT I N TERFACE t t
*----
CONTROL
1---
FRACTION t
I
DETECTION
LOGIC
I
I ,
M U LTIPLIER t
AR RAY
:t
I
+
ADDER
I
[3
I
A D D E R
I � ':
II
L t
MULT I PL I E R
RECODE
LOGIC t
EADING
I t
NORM A L I Z E R
I
1 OR
S T I CKY BIT
I
1
+
�
+
OUTPUT I NT E R FACE
-
CONTROL
t t
CONTROL t
1-
CONTROL t
1-
CONTROL
� t
EXPONENT
LOGIC
+
1---
EXPONENT
PROCESSOR t
-
EXPONENT
PROCESSOR t
-
EXPON E N T
PROCESSOR t
f-4-
EXPONENT
PROCESSOR l
I
D I V I D E R
STAGE 1
STAGE
2
STAGE 3
STAGE 4
Figure 6 F-chzp Execution Core
Vnl. 2 Nn. 2, Spn·ng /C)')() Digital Techllicaljournal
An Overoiew of the VAX 6000 Mode/ 400 Chip Set paths. The fraction data path, the most complex portion of the chip, is 60 bits wide.
The divider fraction data path consists of the hardware divider array and q uotient logic. The divider array implements an iterative radi.x-2 SRT nonrestoring d ivision scheme, and generates three signed quotient bits per cycle.7 The quotient and remainder are driven on consecutive cycles to stage 1 of the pipeline.
The stage 1 fraction data path consists of a 60-bit adder, the multiplier recode logic, and the fraction detection logic. Stage 1 receives its inputs from the i nterface section, or from the divider for divide operations. In add and subtract operations, stage 1 is used primarily to comp u te the difference between the exponents of the two operands. In parallel, stage
I is also used to determine the frac tion difference for exponent d ifferences of 0 and I .
FR
1
FRACTION
�
�
FRACTION RES U LT
2
SIGN 1
SIGN
2
EX PONE
1
I
EX
I
O N ENT
I
A D D E R
�
I
:I
S E LECT
I
I
CONTROL
SIGN AND
EXPONENT
II
S H I FTER-R
II
L 1
D/STICKY B I T
I r B
>
I
•
SHIFT AMOUNT
NORMALIZER
I
I
SIGN A N D
EXPONENT
•
I
I
I
ADDER
�
I
SELECT
+
A D D E R
CONTROL
SIGN A N D
EXPONENT
J
I
CONTROL
I
SIGN A N D
EXPO N E N T
SIGN
� �
EXPON ENT
R ES U LT R ES U LT
Figure 7 F-chip Hardware for Addition and Subtraction
2
(See Figure 7.) For the effective subtract operation, stage
1
uses an exponent difference prediction scheme in which the least significant two bits of the two exponents are examined to determine whether
0 or 1 bits of alignment are required.
If the actual exponent difference is 0 or 1 , stage
I selects the adder result. Otherwise, stage 1 passes the original operands to the next stage for align ment. In multiplication flows, the stage 1 adder is used to compute three-times the multiplicand, and the recode logic generates the recoded multiplier bits for stage 2. In divide operations, stage 1 is used to assimilate the redundant quotient and remainder vectors into a two's complement form.
The stage 2 fraction data path consists of a dynamic 57-bit right shifter, unified leading one or sticky-bit detection logic, and a multiplier array.
(The sticky-bit indicates whether a " 1 " was shifted out of the data path during alignment operations.)
The shifter is used in add-like operations for align ing the fractions. The leading one or sticky-bit logic determines the normalization amount during sub tract operations and integer-to-floating-point con versions. This logic also determines the sticky-bit during alignment in effective subtract flows. The multiplier array implements a radix-8 modified
Booth algorithm and consists of nine carry save adder (CSA) rows. It is traversed once for single precision formats and twice for integer and double precision formats. The hardware organization for the multiply operation is shown in Figure 8 .
Stage
3 of the pipeline consists of a 60-bit adder and a 57-bit dynamic left shifter for normalizing the intermediate results. The fraction adder is used in multiply operations to assimilate the product sum and carry vectors, and in add-like flows to add or subtract the aligned operands.
Stage
4
of the pipeline consists of a 60-bit adder used for rounding and negation of the final result, and the exception detection logic. The exception handling is done w ith a PLA which detects and sig nals overflow, underflow, zero results, and invalid operands.
The control is hardwired (no microcode is used) and distributed among the pipeline stages. Each stage has autonomous control implemented with comrol decoder PLAs. Each stage supplies control information to the next stage one cycle ahead of the data. This enables the data path control signals to set up before the data is propagated to the i nput of the stage. The distributed control scheme permits simultaneous execution of a divide and four other instructions in the pipeline.
Digital Technical journal Vol.
2
No. 2,
Spring
1990
4 5
46
VAX 6000 Model 400 System
FRACTION DATA PATH w
(')
�
I
M U LT I P L I E R
ADDER
3X M U LTIPLICAND
I
RECODER/IPPR
R ECODE
TC
SF
N w
�
(f)
M
" w
(') w
�
(f)
{1
{1
I
[
SELECT s
M U LTIPLIER
1 -9 ROWS
ADDER
PR
�FP c
ADDER ( R N D/N EG)
C F
IPP
I
I
I
I
I
I
K E Y :
SF, C F
T C
I P P
S, C
PR
FP
SUM A N D CARRY OUTPUTS
TWO ' S COM P L E M E N T VECTOR
I N ITIAL PARTIAL PRODUCT
SUM AND CARRY INPUTS
PRODUCT
FRACTION PRODUCT
Figure 8 F-chip Multiplication Fraction
Data Path
The exponent data path in each stage is 1 3 bits wide. Each stage has a 1 3-bit adder and detection logic for detecting zero operands, exponent differ ences, and exception conditions. To compute the absolute value of the exponent difference i n add, subtract, and convert operations, the stage
I expo nent data path has an additional 1 3-bit adder and selection logic.
The microarchitecture and the hardware orga nization of the F-chip were chosen to efficiently implement the three basic operations add/subtract, multiply, and divide. As an example of this effi ciency, the adder in stage 1 is used i n both the effective subtract flow and the multiplication and division flows. This flow overlap permits the execu tion of effective subtraction in only four steps, independent of the data, as opposed to five steps without the overlap.
For multiplication, this organization enabled the use of a radi.x-8 algorithm , w hich requ i res the computation of three-times the multiplicand . The implementation of all other operations was realized with the addition of minimal logic.
1be VC Chip
The vc chip, or the backup cache controller and vector interface chip, implements the second level of a two-level cache structure and the interface to an optional vector processor. A photomicrograph of the vc chip is shown in Figure 9.
Cache Control Functions
The REX520 contains a 2KB primary cache data and tag store. The vc chip contains the tag store and control logic to perform reads, writes, and invali dates to a 128KB secondary, or backup, cache. The backup cache is direct-mapped and write-through.
The vc chip backup cache tag store is organized such that one tag and four valid bits correspond to every four-octaword (64 -byte) block of the cache.
Each valid bit corresponds to a one-octaword sub block, as illustrated in Figure 10. When a cache tag miss occurs on a read, a block is allocated, a sub block is filled, and the corresponding valid bit is set.
When a cache tag compare is successful but the valid bit is not set, a sub-block is filled from memory and the corresponding valid bit is set.
The Backup Cache Data Store
The bach.CJp cache data store is built with off-the shelf CMOS static RAMs, which are located on the module. The discrete cache data RAMs are orga nized as 8 bytes wide (the width of the D-bus) by
16, 384 locations deep. I n addition, there are eight bits of parity, with one bit corresponding to each data byte. Fourteen bits of the VAX physical address are needed to access the cache, as shown in Fig ure 1 1 . When the backup cache returns data to the
REX520 primary cache, it returns one quadword, the fill size of the primary cache.
The Invalidate Filter Bus, /-bus
As mentioned, the chip set implements a special purpose bus, the !-bus, that allows the memory subsystem filter invalidates. In addition to this bus, the vc chip also implements a copy of the REX520 primary cache tag store for invalidate flltering.
Vol. 2 No. 2, Spring 1990 Digital Technical journal
An Overview of the VAX 6000 Mod.e/ 400 Chip Set
VC CHIP
TAG STORE
Figure
9 VC Chip Photomicrograph
VALID BITS
FIRST VALID
BIT
BACKUP CAC H E
OCTAWORD
0
}
ONE S U B BLOCK
( F I LL S I Z E)
SECOND VALID
BIT
OCTAWORD 1
ONE BLOCK
(ALLOCATION
S I Z E)
T H I R D VALID
BIT
OCTAWORD
2
FOURTH VALID
BIT
OCTAWOR D 3
Figure 10 Tag, Valid Bits, and Backup Cache Store
Digital Tech-nical journal
Vol. 2 No. 2, Spring l<J<)O 47
VAX 6000 Model 400 System
48
ADDR ESS BIT
29 28
I I
CACHE ENTRY TAG
1 1 1 0
1/0
SPACE. NOT CAC H E D
BAC KUP CACHE I N DEX
4 3
LJ
0
U N US E D
Figure 11 VAX Physical
Ad£lress and
Backup
Cache RAMs
When an address is sent to the vc chip on the
!-bus, the VC chip notifies the memory interface as to whether or not the address produced a hit in either cache. The memory subsystem broadcasts the invalidate onto the A-bus only if a match was detected. This mechanism significantly improves performance in mult iprocessor systems by reduc ing the overall traffic on the processor buses.
The Vector Interface
The vc chip implements the interface logic that a l lows an optional vector unit to be connected to the scalar processor module. The vector unit has the potential to signific:mtly increase the perfor mance of many analytical applications.
If the vector unit is present in the system, the
REX520 scalar CP chip decodes vector instruc tions and passes operand and control information to the vector module through the VC chip vector interface. The vector unit, using its own memory interface, accesses vector data from memory.
The vector and scalar modules are connected through the vector interface bus (VIB). The VC chip is the sole master of the VIB. The V I B is asyn chronous to the vc chip and synchronous to the vectOr module. The vc chip implements the asyn chronous control logic required for transferring data between the two modules. The decision to operate the V I B asynchronously with respect to the
VAX 6000 Model 400 module was made in order to simplify the design of the vector module.
The RSSC Chip
A variety of support logic is requ ired to com plete the functionality of a VAX CPU module. The
VAX 6000 Model 400 system support chip
(RSSC) integrates on a single chip the common core of functions necessary to support the VAX system environment. A photomicrograph of the RSSC is shown in Figure 1 2 .
The RSSC provides the operating system W•ith the hardware primitives needed to implement the boot and console routines, and with several necessary timing mechanisms. The RSSC is designed to inter face directly with the VAX 6000 Model 400 chips, and is based on the system support chip, or sse s
The decision to base the RSSC chip on the previ ous sse design was made because we could take advantage of an existing core of logic which already implemented the required functions. The sse was fabricated in the CMOS- I process and had been thor oughly debugged and qualified for the MicroVAX
3500/3600 system.9 The challenge faced b y the
RSSC team was to design a new pad ring and bus interface unit
(Bll ) that would interface the existing sse core to a much different and faster interchip environment .
RSSC Functions
The functions that the R SSC provides for the VAX
6000 Model 400 module can be grouped into two categories: boot and console code support, and timer functions to support the operating system.
Some of the important console and boot code support functions contained in the RSSC are the
ROM/EEPROM interface,
CPU halt-request protec tion, ARTs standby with programmable baud rates, I K B of
RAM , and input and output ports which interface to several console and module-level switches and LEOs. The
RSSC also implements three timer functions: the bus timeout counter, a I 0-milli second interval timer, and a time-of-year clock.
1be CLK Chip
Clock generation for the VAX 6000 Model 400 chip set was placed in a separate chip, the CLK chip, to better control interchip skew. By separating the clock, we eliminated power, pin count, area, and noise problems from the other possible home of the clocks, the REX520. Three main factors were taken into account in designing the CLK chip. These fac tors were the number of clock edges, clock skew, and signal integrity. A photomicrograph of the CLK chip is shown in Figure
13.
In determining the optimal configuration for the number of clock edges, t he CMOS m icro architecture was examined closely. The number of clock edges per microcycle determines the granu larity available to the designer. Increased granular ity can simplify the design process by reducing the need for self-timed design techniques. However, the usefulness of partitioning the design cycle is limited. Granularity on the order of a gate delay is of
Vol. 2 No. 2, Spring 1990 Digital Technical journal
An Overview of the VAX 6000 Model 400 Chip Set
Figure 12 RSSC Photomicrograph little use, whereas granularity on the order of a microcycle imposes the use of self-timed circuits. A decision was made to distribute four 14-ns pulses phased in time by 7 ns to facilitate the design of regular arrays, data path structures, and IIO func tions. This distribution made master-slave clock generation and single-phase generation easy, and complementary clocks for latches more available.
C lock skew was m inimized through logic and circuit techniques. The clocks are generated from a
143-M Hz crystal oscillator. The frequency is imme diately divided by two to generate an even duty cycle 7 1 . 5-MHz signal and its complemem (PHI and
PHLBAR). The divide circuitry, which comprises four 0-type flip-flops, is shown in Figure 1 4 . data output . the Q slave and 04 is the QB slave.
CLK is the clock input, 0 is the data input, Q is the latched true data output, and QB is the latched complemem
C L K is the 1 4 3-MHz crystal oscillator clock . The circuit consists of a 0-type flip-flop w ith one mas ter, 0 1 , and three slaves 02, 03, and 04 . 02 pro vides local feedback for the divide-by-two function in order for 03 and 04 to match identically. 03 is
03 allows the i nversion in the feedback path to be absorbed and the Q and QB to see the same num ber of gate delays. This logic technique works for all dividers used i n the design. However, the
Digital Technical journal
Vol. 2 No.
2,
Spring
1990
49
VAX 6000 Model 400 System
Figure 13 CLK Chip Photomicrograph dividers feed a series of large ourpur drivers in which skew could not be controlled through logic techlliques. In these cases, skew was minimized by matching the capacitance on each node. The end result was a design in which the skew between any two edges generated by the greater than 0 . 3 ns.
CLK chip was not
Signal integrity was carefully considered in the design. The clocks had to be distributed to eight different loads, and the total skew between any two edges could not exceed 0.
5 ns. This design implied that the CLK chip package and the clock module interconnect could not contribute more than 0.2 ns of skew. To minimize the electrical impact of the package, three power pins, three ground pins, and two signal pins were used for each driver.
This pin design reduced the inductance seen by each driver by approximately a factor of two. In
CLK
Figure 14 CLK Chip Divide Circuitry
50
Vol. 2 No
.
.!, 5/Jrin[?, 1()()0 Digital Technicaljounwl
An Overview of the mX' 6000 Model 400 Chip Set addition, the clocks were radially distributed, and diode termination was used on each clock leg. The length of each leg was identical.
Conclusions
The VAX 6000 Model 400 chip set represents a suc cessful mapping of the ECL-based VAX 8700 system microarchitecture into a CMOS-based VLSI chip set.
The fully folded pipeline, on-chip cache and control store, and read and run/write and run strategies of the processor chip, combined w ith a h igh perfor mance floating point processor and a second-level cache, enabled the VAX 6000 Model 400 to exceed the original performance goals for the system.
Acknowledgments
The authors would l ike to acknowledge the follow i ng people for their contributions to the VAX 6000
Model 400 c h ip set deve lopment effort: Randy
Allmon, Brad Benschneider, Mary Jo Butler, Sandy
Carroll , Gerry Cheney, Larry Commodore, Beth
Cooper, Amnon Fisher, Nannette F itzgerald,
Annette Flohr, Moshe Gavrielov, Ed Gomes, Pau l
Gronowski, Bill Grundmann, Ellen Kagan, David
Kravitz, Liam Madden, Vijay Maheshwari, Steve
Martin, Karen McFadden, Roger Meeks, George
Mills, Mike Minardi, Millind M ittal, Dave Morgan,
Victor Peng, Jeff Pickholtz, Jim Reinschmidt, Doug
Sanders, Katie S iege l, Rebecca Stamm , Pete
Starvaski, Bob Supnik, Bill Wheeler, Jeff Winston, and Bill Upham.
References
1 . P. Sullivan et a l . , "The VAX 6000 Model 400
Scalar Processor Module," Digital Technical journal, vol . 2, no. 2 (Spring 1990, this issue):
27-35 .
2 . T. Leonard, ed. , VAX A rchitecture Reference
Manual
(Bedford: Digital Press, Order No.
EY-34 59E-DP, 1987).
3. T. Fox et al., "The
CVAX 78034
Chip, a
32-
bit Second-generation
VAX
Microprocessor,"
Digital Technical journal, vol .
1 ,
no.
7
(August
1988): 95- 108.
4 . S.
Mishra, "The VAX
8800
M icroarchitecture,"
Digital Technical journal, vol.
1 ,
no.
4
(February
1987): 20-33.
5. ]. Bartoszek et a l . , " VAX 6000 Model 400 Physical
Technology," Digital Technicaljoumal, vol. 2, no. 2 (Spring 1990, this issue): 52-63.
6. B . Benschneider et a!. , " A Pipelined 50-M H z
CMOS 64b Floating Point Arittunetic Processor,"
IEEE journal of Solid State Circuits, vol. 24 , no. 5 (October 1989): 1317- 1323.
7.
D. Atkins, " H igher Radi,-x Division Using Esti mates of the Divisor and Partial Remainders,"
IEEE Transactions on Computers, vol. C-17
(October 1968): 925-934.
8. ]. Winston, "The System Support Chip, a Multi function Chip for CVAX Systems," Digital
Technical journal, vol . 1 , no. 7 (August 1988):
1 2 1 - 1 28.
9
G . Lidington, "Overview of the M icro VAX
3500/
3600
Processor Module," Digital Technical journal, vol .
1 ,
no.
7
(August
1988): 79-86.
Digital Tecl.micaljoumal
Vol.
2 No. 2, Spring
1990
5 1
john
T.
Bartoszek
Robert]. Han nem ann
Stephen P. Hansen
Robert]. McCarty john
C.
Sweeney
VAX 6000 Model 400
Physical Technology
The physical realization of the VAX 6000 Model 400 microprocessor design offered a number of significant challenges at both the chip package and the module Levels.
In meeting the requirements for a robust and manufacturable midrange imple mentation, the
VAX
MOO
Model
400 physical technology approach broke new ground for Digital, and, in some cases, for the industry. New developments included the first tape-automated bonding (JAB} interconnected semiconductors, extensive board-level physical simulation, and the use of advanced testability features on a microprocessor-based midrange product.
This paper provides details of the physi cal technology used in the
�'lL-Y
6000 Model 400 project to achieve system-level product goals.
Introduction
As shipped beginning in July 1 989 , the VAX 6000
Model 400 microprocessor chip set is one of the fastest and highest performance complex instruc tion set computer (CISC) CPLJs offered by the indus try. Its microcycle time is 28 nanoseconds (ns).
Such performance makes the chip set one of the most demanding in terms of physical technology design at the chip package and single-board computer (SBC) level. This paper details the require ments that drove the VAX 6000 Model 400 physical technology and describes the resulting technology solutions. The range of solutions included the design process, module assembly, and advanced test features.
The
VAX
6000 Model 400 project was the first
Digital microprocessor-based system effort that re4u ired developers to use large-computer design tools and processes. During the project, a number of firsts, for either Digital or the industry, were achieved.
•
The use of advanced rape-automated bond ing (TAB) technology in a manufacturing environment
•
The development of innovative surface-mount able chip packages th:H provide high lead count and a controlled electrical environment for the custom
CMOS CPU and bus interface chips
•
The design and development of an advanced printed wiring board (PWB) technology that all owed over 5000 i nches of i n terconnecting wiring on only four routing layers of a 9-inch by
1 1 -inch board, with two d ifferent controlled impedance levels
•
The extensive use of electrical and thermal s imu lation at the chip package and mod ule levels
•
The employment of an advanced surface-moum technology (SMT) that allows the use of mixed component styles (50-mil SMT devices and fine pitch, 25-mil devices, and a limited number of through-hole mounted devices); and attachment of both active and passive components on both sides of the board
•
The use of advanced rest techniques, most nota bly an innovative continuity transistor strucwre
(CTS) for assembly verification and boundary scan design for the core CPU chips
AU of the project's technology developments will be employed on follow-on products, in both mid range and entry-level systems. Ensuring this kind of technology extensibility was our explicit goal.
In the remainder of this paper, physical tech nology refers to the chip packaging and inter connection technologies at the ind ividual device and module levels.
52 Vol. 2 No. 1, Spring !'.NO Digital Technical journal
VAX 6000 Model 400 Physical Technology
Technology Requirements
The physical technology requirements that drove the VAX 6000 Model 400 technology effort arose from two primary sources: architectural and per formance requirements, and manufacturing and reliability related needs.
Architectural and Perfo
rma
nce
Requirements
The CMOS-2 semiconductor process technology and the microarchitecture were chosen to satisfy the system performance goal of at least six VAX units of performance (VU Ps). That choice established the baseline physical technology performance and elec trical requirements.
One processor design goal in particular had a significant effect on the physical interconnect tech nology. The processor was to operate at a CPU microcyclc time of 28 ns at the SBC level. Although the nominal target of the chip set design was opera tion at 40 ns, early indications were that the CMOS-2 process would allow significantly faster operation .
Therefore, the rest of the physical technology needed to support the 28-ns goal.
Table 1 sununarizes the major features of CMOS-2 and the VAX 6000 Model 400 architecture that drove the physical technology requirements. Chip packaging and module technology requirements were affected in two ways:
•
The wide data bus could result in many drivers switching simultaneously. Coupled with the rel ativel y fast d river rise times, simultaneous switching significantly increased package elec trical performance. Our major concern was the high current draw from local (within the pack age) power planes. The electrical performance required of the chip packages clearly indicated that simple cerquad packages would not work.
The primary shortcoming of the cerquad pack ages was the high inductance of the signal and power connections. Performance levels required multilayer ceramic packages with features that were at the limit of available ceramic technology.
A w ireability estimation technique used early i n the program indicated that four to sLx routing layers would be needed in the PWB, using 5-mil line widths and 5-mil spacing between lines, to route the core CPU chips with associated cache.
•
The relatively high power dissipation require ments contained in the initial design specifica tion posed considerable development difficu lty.
The core chip set was expected to be used in low-end products with limited air-flow capabil ities. However, a junction temperature limit of 85 degrees Celsius at the DEC standard 102 Class
B environmental limits had to be maintained .
How we addressed these problems is described in the later sections, Chip Packaging and Module
Technology.
Table 1 VAX 6000 Model 400 CMOS-2
Features Affecting Interconnect
Technology
Driver rise time
Clock cycle
- 4 phase cycle
Data bus width
Cache access time
Max. chip pin count
Max . chip power
1 ns
28 ns
64 bits
7 clock phases
(49 ns)
224
6 watts
Table 2 lists the key CMOS-2 chips developed for the VAX 6000 Model 400 processor and the key characteristics of these chips that affected the physical architecture of the design.
Table 2 VAX 6000 Model 400 Chip Set
Characteristics
Chip
Lead
Maximum Lines
Count Switching
(Actual) (Approximate)
Power
(W)
P-chip/DC520
F-chip/DC523
224
224
VC-chip/DC592
XDP/DC550
224
224
XCA/DC551 224
81
80
60
1 00
70
6 . 1
4 . 5
4 . 2
3 . 6
3.6
A more direct measure of the electrical perfor mance required from the VAX 6000 Model 400 physical interconnect system can be derived by examini ng the timing budget allocated to some of the signal transfers that occur within the CPU.
Three of the most critical are the clock distribution system, the cache access loop, and the data bus between the core CPU chips. The clock signals need to be valid and synchronized within 0.5 ns at the end points of all clock lines. The clock electrical performance relied heavily on the uniformity of the distribution system.
Digital Technical journal Vol. 2 No.
2. .vJring 1990
53
VAX 6000 Model 400 System
The cache access loop timing was slightly more complicated. The cache access loop timing budget that was established for the 28-ns version of the CPU is shown in Table 3. The interconnect system pri mari ly affects the address bus (A-bus) settling time
(20.5 ns).
Table 3 Cache Access Loop Timing Budget
Address bus settled
Buffer
RAM access
20.5 ns
7.0 ns
21 .0 ns
Total 48 . 5 ns
Manufacturing and Reliability
Requirements
Certain manufacturing and reliability goals estab lished for the VAX 6000 Model 400 product also influenced the physical technology selection and design. The major goals were as follows:
•
The use of TAB as the off-chip interconnect tech nology, which was viewed as an appropriate entry point for TAB
•
An all surface-mount technology (SMT) module assembly approach
•
A robust test strategy, including significant use of fault diagnosis using boundary scan and other test features
•
Maintenance of the ability to perform engineer i ng changes using etch cuts on the PWB by restricting signal routing to the second and ninth layers, with clock lines only on the remaining two signal layers
•
Achievement of the required 85 degree Celsius junction temperatures
Most of these goals were met :
•
The worst-case A-bus and data bus (D-bus) settling times were 19.6 ns and 20.2 ns, respec tively. These settling times met the timing requirements as defined in Table 3.
•
A nearly total surface mount assembly was achieved, with only connectors, oscillators, and erasable progr anun able read-only memories
(EPROMs) in through-hole configurations.
•
Boundary scan was designed into the core CPU chips. Observe-only scan latches were used at the chip boundaries as opposed to the original target of full scan at all logic boundaries.
•
Test transistor structures for interconnect verifi cation at module assembly were implemented for the custom chips.
•
Separate, system-specific micro heat sink designs allow all chips to operate at a maximum of
85 degrees Celsius, with one exception : the
6. 1 watr P-chip reaches 89 degrees Celsius under absolute worst-case conditions. A lthough not specifically meeting the defined goal, overall system reliability and operations were not felt to be impacted to any significant degree by aUow ing this exception.
Some goals were not met .
•
T he interchip routing included over nal nets and over 1 800
1000 sig routed nets within the
PWB. This large number of nets required the use of the clock routing layer to complete the inter connect of the CPU module. Although the PWB technology supported 5-mil lines and spaces, the signal integrity constraints of t he system required more than 5-mil spaces between signal lines. In addition, the ability to do etch cuts on a
5-mil line was determined to be too difficult and risky to pursue. Engineering changes were per formed by doing cuts in the surface etch, which connects each surface-mount pad to its associ ated via, which in turn connects to an internal signal layer.
•
TAB tape and packages were designed, proven, and sourced for six of the 224-pin CMOS-2 chips in the VAX
6000
Model 400 processor. However, for manufacturing logistics and line-loading rea sons, TAB is currently used in only a subset of these devices. A series of wire-bonded backup packages was designed and is now in use for the remainder of the 224-pin devices.
Nonethekss, the physical interconnect technology used did result in package designs and a module design that supports
7
VUPs, 224 1/0, fine-pitch devices, with associated cache, operating at 28 ns.
The details of the chip packages, the PWB tech nology and module design are presented in the following sections. Also described are the test struc tures required to satisfy the performance needs and other established goals of the VAX 6000 Model 400
CPU product.
54
Vol. 2 No. 2, Spring /'J'JO
Digital Tecbnicaljournal
VAX 6000 Model 400 Physical Technology
Chip Packaging
In March
1989,
Digital's Semiconductor Intercon nect Technology Group (SCIT) completed quali fication of the Model 400 processor advanced chip packaging technology.
This qualification consisted of two primary ele ments, which are discussed in the sections below.
The first element is the initial implementation of our internally developed perimeter tape-automated bonding (PTAB- 1 ) process. The process serves as the interconnection medium from the semiconductor device to the chip package. The second element is the chip package itself, which is an advanced 224-
Jead multilayer ceramic ( M LC) surface-mount com ponent. The TAB/MLC combination provides an effective chip package solution for the VAX 6000
Model 400 products. Moreover, TA B/MLC serves as a technology springboard for further TAB-based packaging solutions currently in development for future CMOS systems.
A fundamental decision was made early in the
TAB technology program. To reduce risk, we wanted to introduce TA B in an environment that allowed a convenient wire-bonded backup to be developed i n parallel. The TAB/MLC solution proved ideal for this approach, and a set of w ire-bonded packages was designed and qualified. Subsequently, several of the VAX 6000 Model 400 chips were moved to the backup wire-bonded packages for manufacturing logistics and line-loading reasons.
Wire-bonded technology is not described in this paper because the package designs are very similar to the TA B/MLC packages detailed below.
Tape-automated Bonding Technology
The connections from a semiconductor chip to the package have traditionally been made with wire bonding technology. In this process, free-floating wires are placed individually from the aluminum chip termination pads to the package internal con nections. Although this technology is versatile, it has some limitations in high pin count, high-density applications. Bonding becomes very time-consum ing at high pin counts; package and manufacturing tolerances become critical; and package choices with wire bonding have increasingly narrowed to newer and more expensive technologies, such as precision thin films.
From
1980
to
198 1 ,
exploratory work was per formed within Digital on alternative interconnec tion technologies. In
1982,
the decision was made to pursue TAB technology. The rationale for this deci sion included TAB 's easy testability, compatibility with a wide variety of packaging formats (both sin gle chip and multichip ), and improved pin density.
The technology also had the potential for electrical enhancements with multiconductor TAB tape, and indications were that the industry as a whole would embrace TAB technology.
Unl i ke the w i re-bonding process, TAB utilizes photolithographically defined copper conductors stabilized by a dielectric film made of polyamide.
As shown in Figure I , the TAB tape consists of an inner set of connections for the chip-to-tape or inner lead bonds (ILB). External to the dielectric material is the outer lead bond (OLB) region where the tape-to-package connection is made. The leads then fan out to a set of test pads. These pads permit full electrical testing of the semiconductor device at this level of assembly. Sprocket holes serve as han dling and alignment features.
Unlike wire bonding, in which the wires may be bonded directly to the aluminized bonding pad on the chip, the planar TAB tape requires a raised pillar, or bump, on each bond pad. The bump, usually made of gold, is typically
25
microns high. In addi tion to acting as a standoff, the bump provides a surface that is appropriate for the bonding of the gold-plated tape. Figure
2
shows a bumped and bonded device. The bump process may be thought of as an extension to the wafer fabrication process in which the construction of the bond pads is con tinued vertically.
Figure I TAB Tape Configuration
Digital Technical journal Vol. 2 No. 2,
Spring 1990
55
VAX 6000 Model 400 System
Figure 2 TAB Bumped and Bonded Device
As shown in Figure 3, after the semiconductor wafer has completed the standard diffusion, metal lization , and passivation steps, the wafer is metal lized again over the entire surface with a sputtered barrier metallization of titanium/tungsten. In the same process step, a top seed layer of gold is applied . The total thickness of these films i's 10,000 angstroms. The barrier prevents diffusion between the aluminum pad and the gold bump. Such diffu sion would, w ith time and temperature, seriously degrade the mechanical strength of the interface.
The seed layer forms the ba'\e upon which the bump will be plated during a subsequent step.
Metallization is followed by the deposition of a thick layer, typically
_)0 microns of photo resist .
This layer is then photolithographically patterned w ith a bump mask. Subsequent etching opens a hole over the bond pads down to the barrier and seed layer. At this point, the wafer is electroplated with pure gold. The gold is deposited only in the resist openings. Finally, the resist is stripped and a series of etches are performed to remove the seed and barrier metallizations from the areas between the bumps.
After the wafer processing steps are complctccl, the devices may be electrically tested by probing the bumps. Assembly of the functional die consists of sawing the wafer, selecting the good devices, and bonding the bumped chips to the
TAB tape. The bonding process is very similar to wire bonding.
Once the TAB tape has been aligned with the bumps, a tool supplies pressure and ultrasonic energy to the interface and creates a strong gold gold metallurgical bond .
In 1985, the
VAX
6000 Model 400 processor wa� identified a-; the first product that would incorpo rate
TAB
technology. Consequently, the
TAB
pro gram 's goals were set to meet the product's needs :
•
•
•
150-micron pad pitch
2 24 pins
•
35-mm format
TA B tape
•
Assembly in ceramic package
The program was designated
PTAB
1 , the first implementation of
TAB
w ith chip pads in a peri meter or single row format. The
PTAB
1 process was formally qualified in March 1989
VAX 6000 Model 400 Multilayer Ceramic
Packaging
The custom semiconductor packages implemented for SCIT 's ZMOSand CMOS-1 -based systems have been either cofired multilayer ceramic
(MLC) through-hole pin grid array (PGA) packages, or the simpler lead-frame-based cerquad surface-mount packages. The maximum pin counts in use up through CMOS- I are 132 pins for PGA style packages and 164 pins for fine-pitch cerquads.
Figure
3
Semiconductor Wafer
56
Vol. 2 No. 2, Spring I'J'JO
Digital Tecbnicaljounwl
VAX 6000 Model 400 Physical Technology
The
VAX
6000 Model 400 P,
F, VC, X DP,
and
XCA
chips require a packaging solution consistent with their electrical and thermal performance needs as well as compatibility with a lead count of 224 pins.
The solution chosen for these components was a blend of the
PGA
and cerquad technologies: a multi layer ceramic body with 25-mil pitch perimeter leads for surface mounting. The resulting 224-pin
M LC
is depicted in Figure 4.
There are several significant features in the 224
MLC family of packages. I n wire-bond packages, maximum wire span constraints require a different layout for each chip configuration. The TAB format, however, has more routing flexibility and permits one basic package layout. The package has five internal interconnect layers with assignments, as shown in Figure 5.
In cofired MLC technology, the conductor traces are made with screen-printed, tungsten-filled vias.
To specialize the package interconnect for each chip, a programmable approach was developed in which only one via layer would have to be changed for each package design. The combination of TAB interconnect and the programmable package con cept considerably reduces design time, design com plexity, tooling costs, and lead times.
Another feature of the 224 M LC is the provision for eight chip capacitors for power decoupling. The heat-sink design for the 224-pin MLC package is
KEY:
M P l - SEAL RING AND LOGO
MP2 - VSS EXTE R N A L (GROUND)
MP3 - SIGNAL LAYER AND BON D S H ELF
MP4 - VDD EXT E R N A L (POWER)
M PS - SPACER LAYER
M P6 - VSS I NT E R N AL (GROU N D), D I E ATTACH AREA
MP7 VDD I NT E R N A L (POWER)
MPB - CHIP CAPACITOR SOLDER LANDS. LEAD
B R A Z E PADS. HEAT S I N K ATTACH A R EA
Figure 5 Package Itself with Int erna l
Interconnect Layers with
Assign men t shown in Figure 6. The thermal performance of the package is shown in Figure 7.
The 224 the ftlled die attach epoxy. Outer lead bonding of the
TAB
TAB
MLC package is assembled by mounting chip into the package cavity with silver film to the gold-plated package pads is accom plished with the same pressure or ultrasonic pro cess used for inner lead bonding. Final steps consist of lid seal, lead plating, chip cap and heat-sink attach, and lead trim. The part is then ready for electrical testing.
Module Technology
Printed Wiring Board Technology
The
VAX
6000 Model 400 processor's printed wir ing board
(PWB)
requirements offered a significant challenge to hoard fabricators. The primary differ ence between the Model 400 processor's boards and previous boards is 10-mil finished vias, resulting in a 7 to
1
aspect ratio. We were initially very con cerned whether our vendors could produce boards
Figure 4 The 224-pin MLC
Digital Technical]ounral
Vol. 2 No. 2, Spring
199()
Figure 6 MLC Package Heat Sink
57
VAX 6000 Model 400 System
20
1 5
>=' w u
� z a:
<( W
1- (L
'!2 m a: rn
_j
_j w
<( U
::::; (/) w w
I a:
1- (.9 w
9.
1 0
5
0 200 400
600 800
AIR V E LOCITY
(LINEAR FEET P E R M I NUTE)
1 000
Figure 7 Thermal Performance of Package that met our specification i n the volu mes we needed . Table 4 lists some of the primary features of the Model 400 PWB specification.
The most severe environment the boards would see was the assembly plant. Because the module used a variety of component types, it was put through several process steps, each step requiring general or localized heating to solder reflow tem peratures. We wanted to ensure that the boards would survive multiple assembly and repair cycles and still be reliable. We established a "fit for use" plan that required the boards to undergo a series of thermal cycles. In these cycles, temperatures and times were set to match intended assembly process steps. The boards were then cross-sectioned and examined for defects. Once manufacturing began building modules, a few of these were also cross sectioned. With this strategy, we could quickly determine the q uality of each lot of boards. More over, we could begin to correlate board structure with board quality in each lot. This process allowed us to assess each vendor's capability to provide boards that wou ld meet our specifications.
Because module assembly process temperatures typically exceed 200 degrees Celsius, questions arose over what was the appropriate board mate rial. The more commonly used FR4 material, with an average glass transltton temperature (tg) of
1 20 degrees Celsius, was suspected to be incapable of surviving the assembly process.
The alternative to FR4 was polyamide, which has a tg of 240 degrees Celsius. In response to the con cern over FR4 , a PWB material selection task force was convened, which was composed of board experts from throughout the company.
The task force discovered that board material actually is not the primary consideration. Either FR4 or polyamide is acceptable (both were eventually used in production). However, other board parame ters become critical when FR4 is used. Primarily, minimum barrel copper plating thickness should be one mil for FR4 boards.
Variation in barrel copper should not exceed 50 percent. In addition, there can be no smearing or
Table 4 VAX 6000 Model 400 Printed Wiring
Board Specifications
Parameter
Value
General :
Board size
Layer count
Material options
Copper foil type
Testing
9.2 inch x
1 1 .0 inch
1 0 layers: 4 signal,
4 power/ground
FR4, polyamide
Class I l l
100% data-driven
Electrical:
Characteristic impedance
DC resistance
50 ohms ± 10% for 10-m il l i nes
4 ohms max i m u m
P hysical:
Via type
Via size
Through
10 mil fin ished
13.5 mil d ri l led
7: 1 Maximum aspect ratio
SMT pads
Etch widths
50-m il pitch: 0.030 inch x
0.076 inch
± 0.001 inch
25- m i l pitc h : 0.016 inch x
0.076 inch
± 0.001 inch
Signal: 5 m i ls
Clock: 10 mils
5 mils Prox i m ity to next feature
Solder requirement
Tin/lead alloy
0.15 mil minimum
63/37 ± 10%
58
Vol. 2 No. 2, Spring [<)!)()
Digital Tecbn icaljournal
VAX 6000 Model 400 Physical Technology other defects in the vias from the drilling process.
To maintain tight control over these parameters, we included statistical process control as a vendor requirement. Table 5 summarizes the results of the task force findings.
Because of recent board quality problems and the aggressive nature of the Model 400 specification, we ran a product-specific board qualification. The goal was to verify that each vendor could consis tently produce the Model 400 boards in volume before the vendor was placed on the qualified vendor list. The four key components of the plan were as follows:
•
Incoming inspection
•
Electrical test
•
Cross-section analysis
•
Assembly verification
I ncoming inspection testing was nondestructive and covered plating thickness, plating composition, and characteristic impedance.
Using an internally developed tester, we could verify 100 percent connectivity between pads and, thus, detect both shorts and opens. We also used this test to verify prototype boards, ensuring that the boards were good before valuable prototype parts were committed to them.
D igital 's Component Evaluation Laboratory performed bare board and assembled module
Table 5 VAX 6000 Model 400 PWB Materials
Task Force Summary
Laminate:
Polyamide should be used for prototypes.
Polyamide and FR4 are acceptable for vol ume.
Long-term, vendor capability and cost may favor polyamide.
Copper:
Class 3 foil should be used .
For polyamide, standard plated-through-hole (PTH) copper is acceptable.
For FR4, minimum barrel plating thickness is 1 m i l .
Barrel plating thickness variation max i m u m is 5 0 percent. H o l e qu ality must b e good .
Process Control:
Vendor should provide test coupon cross-sections with each lot.
Vendor should institute statistical process control.
Vendor site should be monitored regul arly. cross-sectioning. This testing process uncovered inadequate or nonuniform plating, pad-to-copper plating separations and misregistration. The infor mation gained from this procedure was given to the vendor as a basis for corrective action on future board production.
Assembly verification was final proof that the board would make a reliable module. Each board went through the ful l assembly and test process, including burn-in, to ensure it could survive the process and pass all functional tests.
Surface-mount Assembly Technology
The surface-mount assembly technology (SMT3) used for the VAX 6000 Model 400 processor is the latest in a series of electronic assembly technologies developed by Digital since 1 985 . The SMT3 tech nology allows double-sided mounting of high lead count and fine-pitch devices on a printed wiring board, with surface-mounted passives and mixed component styles. The VAX 6000 Model 400 pro cessor uses essentially all of the SMT3 features.
High pin count and fine-pitch devices presented new problems to the surface-mount attach process team. The small , tightly spaced leads require a smaller pad and less solder than their 50-mil pitch predecessors. The smaller, more fragile pins can become misaligned and no longer coplanar where they meet the board surface.
Our primary goal was to find the correct pad size and solder volume for attaching 25-mil pitch com ponents. Once the fine-pitch pad size and solder volume were determined, it became apparent that the correct solder amount could not be delivered to both ftne-pitch and standard pads in a single oper ation using the standard solder-paste screen approach. The smaller, fine-pitch screen openings could not consistently pass the correct amount of solder. The solution was to use a laminated stepped stencil that places a thinner solder deposit on the fine-pitch pads than on the rest of the pads.
The initial approach for attaching fine-pitch devices was to use the existing vapor-phase mass reflow process. However, if pin noncoplanarity exceeded 2 mils, some pins would not be soldered adequately. The best that could be guaranteed was
4 mils.
The development team turned to solder-in-place, which uses a thermode fixture to place the compo nent , push the pins into the paste, and reflow the solder, forming a good solder connection for each pin. An advantage of this process is that it does not
Dig ital Technical journal
Vol. 2 No. 2, Spring 1990
59
VAX 6000 Model 400 System hear the entire board, and the pn.:viously attached components, ro solder-reflow temperatures.
The surface-mount module process was b y necessity developed concurrently with the Model
400 product design. As the design progressed, a problem with the primary assembly equipment developed. Suppliers for h igh-volu me thermode pick, place, and solder equipment did nor keep pace with our schedule requirements. We were then faced with the choice to develop equipment inter nally or switch the process to a more developed technology.
The Midrange System Manufacturing Group's most readily available backup process was vapor phase mass-reflow. This pron:ss guards against coplanarity problems by including careful inspec tion of all fine-pitch components before commit ring them to a board . The pins that do nor solder are manually repaired. Since it was nor possible to develop solder-in-place equipment internally in time to meet our schedule, the program decided to use vapor-phase mass-ref! ow.
Module Design
Several board design requirements combi ned to make the design task challenging. These require ments included signal integrity constraints, finer layout and routing grids, and short dispersion etch.
Figures 8 and
9 show the complete module. Most of the module area is composed of the six large
224-pin devices that form the processor core and interface to the XMI corner. These devices had critical placement requirements. Their 25-mil pin pitch forced very dense etch runs.
To attain maximum use of available routing area, the design used tenth-mil grid instead of the previ ously used one-mil grid. The new grid allowed opti mum etch channel placement. Similarly, vias were placed on a 25-mil grid with 50-mil spacing. This allowed the designer more flexibility in placing vias so the available space was used most effectively.
The VAX 6000 Model 400 processor introduced new routing parameters and stringent signal integrity constraints. The density of the design required 10-mil vias. Signal integrity considerations
60
Figure 8 Processor Module - Side One
Vol 2 No. 2, Spring
1990
Digital Tecbnicaljournal
VAX 6000 Mode/ 400 Physical Technology imposed component spacing and electrical connec tion requirements that resulted in components being closer than suggested by current standards.
Surface etch was needed to keep connections short.
Components on side two were placed underneath fine-pitch devices on side one for electrical prox imity. New manufacturability rules were generated tO cover these situations as the design progressed.
Because surface layers are not used to route sig nals, these layers were not designed to have good signal integrity characteristics. However, a signal has to travel a short distance from its pin p�1d to its dispersion via which connects to an inner routing layer. Therefore, it is essential to keep surface etch as short as possible to minimize the distance the sig nal travels outside a controlled impedance environ ment. To meet this distance requirement, each critical component was placed very precisely and its dispersion pattern was individually designed.
S ignal integrity considerations placed other con straints on the design. Critical signals had to be kept less than a specified maximum to meet perfor mance specifications. To avoid skew problems, clocks were routed equal lengths to within a tight tolerance. Table 6 details the resulting design parameters.
Test Technology
Assembly and Test Process Development
Assembly and test process issues were tracked throughout the development and selection of the
Model 400 physical technology. The manufacturing impact of each physical technology choice was quantified in a spreadsheet analysis of cost and qual ity metrics.
These metrics were estimated by using integrated assembly and test process models for each possible physical technology implementation. Two issues became evident during the physical technology selection process.
Figure
9 Processor Module -Side Two
Digital Technicaljournal
Vn/. 2 No. 2, Spring 1')90
6 1
62
VAX 6000 Model 400 System
Table 6 VAX 6000 Model 400 Processor
Board Design Statistics
Parameter Value
General :
Board size (i nches)
Board thickness ( i nches)
Layers
I n itial route area (square inches)
Rou ting vias
Dispersion vias
Total components
Total component pins
Total used component pins
Total networks
Total etch length (inches)
9
.
2
X
1 1 . 0
.093
10
99
1 699
38 1 3
623
4976
42 1 8
1 005
5002
First, the high number of signals and the fine pitch of those signals in all of the possible product implementations significantly increased the risk of manufacturing defects, such as shorts between signals or open faults along a signal. transistor structure
(CTS)
A continuity is designed into each of the
VLSI
(very large scale integration) devices to help test and diagnose these open faults.
Second, high-speed operation and reduced phys ical access would make diagnosis of processor fail ures difficult. To alleviate this problem, test features and a test system were developed. T he test feature was a form of boundary scan, called observe boundary scan
(OBS).
The test system, the
VAX
6000
Model 400 scan monitor, utilized the system test environment.
OBS in a
Continuity Transistor Structure
Because the devices are surface-mounted to the module, a large percentage of the manufacturing defects were expected to be open faults between the module and the chips. Typically, these open faults are difficult to detect and diagnose because they usuaUy require the development of a set of complex test vectors that will be applied to the chip. To simplify the test for open faults, a continu ity transistor structure
(CTS) is designed into each chip. The
CTS tests for open faults by using simple instruments such as voltage sources and current meters on an in-circuit tester
(ICT).
The Module 400
CPU module is placed on a bed of-nails fe<.ture chat gives the tester electrical access to at least one point on each internal module signal network. The tester then applies digital stimulus to the internal module and verifies correct contact by reading current flow.
T he
CTS is shown in Figure 10. Pins
1 through N represent aU signal pins on the chip. The design uses minimum-sized transistors. The
CTS design does not require any dedicated pins because the test pin is a normal device signal pin. There is no perfor mance penalty because the transistors are placed in paraUel to the normal system logic, which results in a negligible load on those signals.
The use of the
CTS in the processor module manufacturing process has been very successful.
The
ICT very quickly isolates open connections to the device pin and differentiates them from device test pattern failures. T his process allows the open connection to be repaired rather than replacing the device.
Further,
CTS testing allows prototype modules to be fully tested for assembly defects, even if the
V LSI in-circuit test patterns are not available. This advantage is possible because designers can fully develop
CTS tests without any knowledge of the
VLSI device internal structure or function.
Observe Boundary Scan
Modules that pass
ICT testing are then tested at a
"system like" test station. Self-tests and boatable diagnostics are run, and the
VMS system is booted. If a module fails any of these tests, skilled technicians diagnose the failures by attaching logic analyzer probes to the module. Because of the fine-pitch surface-mount devices and the high-speed opera tion, it is very difficult to attach logic analyzer probes to many nodes on the processor module.
OBS allows the
CPU module to be observed as it executes
VAX macrocode on board self-test stimu lus at the module's full clock rate. T his additional observation is used by the scan monitor to help test and diagnose module-level faults in a system test environment in stage-one manufacturing.
TEST PIN (ALSO A NORMAL
DEVICE SIGNAL PIN)
N-CHA N N E L MOSFET
::: r3
N - C H A N N E L MOSFET
G N D
P I N N
· · ·
Figure
10
Continuity Transistor Structure
Vol. 2 No. 2, Spring J<)<JO Digital Technicaljournal
VAX 6000 Mode/ 400 Pbysical Tecbnology
Designing OBS into a custom VLSI device is more complex than adding CTS , but is still relatively sim ple. OBS is simply a parallel- load, serial-shift register with one bit of the register on each device pin.
Although it is not negligible, the OBS uses a rela tively small amount of silicon area and also does not affect product performance. The total area used by both the CTS and OBS was estimated at about one to two percent of the chip area. Unlike CTS, which does not require any dedicated device pins, OBS uses two dedicated pins on each device in which it is implemented. To fully utilize the OBS test fea ture, the VAX 6000 Model 400 scan monitor was designed and built.
The scan monitor controls and reads the OBS on the Model 400 module. A host computer system interfaces with the monitor. The scan moni.tor control program (SMCP) operates the scan moni.tor, makes pass/fail decisions on the data received , and diagnoses failures. SMCP also includes many features that allow it to perform as a virtual logic analyzer, including waveform displays that high light faulty behavior, as we.ll as full triggering functionality.
Conclusion
The aggressive performance goals and advanced semiconductor technology used for the VAX 6000
Model 400 processor meant a signi.ficant develop ment effort for packaging and interconnect techno logy. The technology requirements included high lead count, electrically tailored single-chip pack ages, very dense controlled impedance printed w iring boards, a state-of-the-art surface-mount assembly process, and advanced test features.
The physical technology achievements in the
VAX
6000 Model 400 project represent an effort in the packaging and interconnect disciplines more akin to mainframe and supercomputer develop ments than to traditional microprocessor-based system approaches. The accomplishments of the efforts include:
•
Development and implementation of an advanced TAB technology for the high lead count custom chips
•
Design of an innovative semi customized ceramic single-chip package that combines the best fea tures of surface-mount devices and traditional pin grid arrays
•
Development, sourcing, and qualification of very dense printed wiring boards with multiple controlled impedances
•
Achievement of a technology set capable of
28-ns clock cycles through the use of full elec trical simulation at the device, package, and module levels
•
Development and implementation in manufac turing of the SMT3 module assembly technology, which allows double-sided mounting, high lead-count fine-pitch surface mounting, surface mounted passive components, and mixed com ponent types
•
Introduction of innovative testability features, including the continuity transistor structure for assembly verification and observe boundary scan for diagnosis in engineering debug and module manufacturing
As intended at the outset of the project, these technologies will be employed on a significant num ber of follow-on midrange and low-end products.
Acknowledgments
A large number of people contributed to the success of the VAX 6000 Model 400 physical technology program. It is not possible to name all these impor tant contributors in this space. Therefore, the authors want to acknowledge that the VAX 6000
Model 400 project could not have been undertaken nor have succeeded without significant efforts by members of the SCIT Physical Technology Group,
Semiconductor Assembly Group, Midrange Systems
Engineering Group, Corporate PWB Group, Mid range Systems Integration Group, and the SCIT
Semiconductor Engineering Group.
Digital Tecbnicaljournal Vol. 2 No. 2, Spring
1990
63
Richard
E.
Calcagni
Will Sherwood
I
VAX 6000 Model 400 CPU
Chip Set Functional
Design Verification
Tbe VAX 6000 Model 400 system is Digital's first VLSI
CPU to employ a ful�y micro pipelined architecture. The
CPU chip set for this system posed verification challenges
Jar beyond those of previous designs. The major problem was the large number of complex control sequences and combinations that could exhibit design errors.
A single verification strategy would not suff'icient(v handle this complexi�y. Therefore, t:et�(ication engineers developed a multipronged approach for simulation modeling and functional clesign verij'ication. They also employed
CPU diagnostic programs, hand-generated tests, and directed pseudo-random techniques to
verify
that the design conformed to the
VAX architecture. These techniques helped them find bugs prior to committing the design to masks. As a result, the first-pass versions of the
CPU chip set successfully booted em operating system. Simulation also minimized chip rework and delays in bringing the product to market.
The Design Verification Project
The VAX 6000 Model 400 chip set verification pro ject had two goals : find implementation bugs in the design and verify that the design performed as a VAX system. The design verification tasks involved about
25 person-years of effort in the areas of system microcode, custom V LSI (very large scale integr::t tion) chips, and the VAX 6000 Model 400 sc::tlar pro cessor module.
The chip set verification team coordinated a set of simulation models origin::tlly written by the chip logic designers. At various stages of the project, models were available at the gate/transistor, behav ioral, and architectural performance levels. The verification team used these models to run a wide assortment of both basic and sophisticated tests.
The use of simulation models is described in more detail in the next section.
The CPU design was partitioned into functional units, and one or more units were assigned to each member of the verification team. A list of specific tests or testing activities was produced for each sec tion of the
CPU chips' specifications. This list was
;wgmented by project-wide brainstorming sessions.
These sessions were used to analyze obscure or sub tle combinations of events in the design . Often, the thinking process would identify that a bug existed before any testing had been done. Verification engineers used the Jist to create tests for the functional units of each chip. These tests were implemented in either microcode, macrocode, pin-stimulus, or some combination of the three. Tests were imple memnl in the priority determined by the design ream's identification of the most complex areas of the design and those most susceptible to bugs.
Simulation Models
Functional design verification using software simu lation is inherenrly s low in a design as large and complex as a VAX CPU.
1
To use resources most efficiently, the verification team specified and coor dinated a project-wide modeling methodology that incorporated a number of different modeling levels, trading off detail versus other factOrs such as speed.
These trade-offs allowed us to match the testing of each phase of the design to a model that met its specific needs and characteristics. Thus, simula tions were only as detailed as necessary for a parti cu lar test situation, and the overall efficiency of the verification effort was increased. Most levels of modeling could contain different levels of descrip tion detail for different areas of the design, which further optimized simulation performance.
There were three major phases of the design: architecture, detai led block diagram, and logic/
64
Vol. 2 No. 2, Spn·ng 1')<)0 Digital Technical journal
VAX 6000 Mode/ 400 CPU Chip Set Functional Design Verification transistor schematic. To match these design levels, architectural, behavioral , and structural models were written . Each model section was written by its designer and then integrated into a whole-chip model, a system model, or both models for chip and system testing. In addition the design method ensured that the overal l organization of each modeling level was faithfu l to the hardware design level it represented.
Architectural Model
The architectural-level model is the highest level of modeling for this project.
!.I 1
This model describes only the control algorithms and abstract data paths in the microarchitecture. The VAX
6000
Model
400
CPU architects wrote this model in PASCAL for exe cution performance reasons. The PASCAL program avoids most of the simulation-oriented overhead because it is a standalone program .
I n the architectural model , actual microcode i s used. However, because much o f the detail o f the m icroarc hitecture is abstracted in this mode l , cm tches (additional simulation aids) are required to execute the microcode flows. Model simulations are dri ven from inst ruction traces. Opcode, operand , and address information are extracted from user programs and system software running on actual
VAX systems. Special fields in each microword make use of the information from these traces and direct the flow of microcode execution accordingly . These special fields are only used for simu lation and are not incl uded in the actual microcode implemented in hardware. The use of trace data extracted from real VAX systems permits actual machi ne loads to be reproduced, and the architect can evaluate implementation trade-offs from these reproductions.
Signals in the architectural model are correct to within a machine cycle, which al lows execution t imes to be accurately measured . Microarchitec tural parameters, such as cache or translation buffer size, can be easily adjusted to anal yze their effects on system performance. More complex design fea tures, such as bus protocols and pipeline control algorithms, can also be modi fied relatively easily.
The model describes most of the hardware sec tions that wil l be in the final design. Therefore, i t serves as a prototype system debugging tool to predict and tune system performance. The model also uncovers design flaws before implementation begins. About a hundred bugs were found at this preliminary s tage by using the architectural model.
Behavioral Model
The behavioral level model describes each chip sec tion's logic i n detail. It is written in the DECSIM behavioral modeling Janguage.'u' 7 DECSIM pro v ides a high- level computer-hardware description language that executes procedurally rather than using event-driven algorithms . Models written in such a language generally simplify both model and design debugging.
The modeling methodology req uires that, i n the behavioral model , every signal in the design be explicitly modeled, with its timing accurate to the clock-phase boundary. This met hodology maxi mizes the probability that timing problems will be found at this level of simulation . (Note: No addi tional logic timing verification was done on this project .) Each designer writes the section's model in parallel with writing the chip's specification. The writing must be done before detailed schematics are started. This method ensures that the model accu rately represents the real hardware behavior.
The model executes hierarchically . The system clocks are advanced at the beginning of each phase, then the top-level routines for each section are called . These routines, in turn, call the routines for each subsection. The subsection routines do the actual work.
The DECSIM behavioral model is the basis for many model variations. These variations range from the simplest single-CPU mod ule ro elaborate multi processor versions that include peripherals and I/O adapters.
The basic version of the behavioral model is non ported; i .e. , the model is implemented as a single. self-contained hierarchy of procedures that does not use or connect to any other models. The non ported model contains detailed descriptions of the three custom
VLSI chips: CPU processor, secondary cache controller, and floating point accelerator.
This model also i ncludes a representation for back'Up cache RAMs connected to the DAL (data and address lines). Also modeled are a system support chip and a simple memory that can return data as fast as the protocol allows.
This nonported model was used for extensive testing of the microcode, microarchitectu re, and logic design of the
VLSI chips and their i nterfaces.
A majority of the verification tests did not need detailed memory timing or
I/0.
Tests could be run faster on the nonported model than on one that simulated the actual memory access delays. These delays would have increased the testing time with out adding value to the testing process.
Digital Tecbnica/]ournal
Vol. 2 No. 2, Spring /')')() 65
VAX 6000 Model 400 System
Another version of the behavioral model is ported. This model was constructed to test the
DAL and the interaction of the core chips with the system support chip (RSSC) and bus interface stan dard cell (REXMI) chips.H Ported models have ports at the boundaries of procedural model compo nents. Ports are used where chip pins or similar boundaries appear in the design. Ported model components are i ntegrated for simulation through a structural wire list derived from module (printed wiring board) schematics. The main ported model consists of the core CPU chips with a ported repre sentation of the DAL . This model connects to ported models of the other chips on the VAX 6000 Model
400 module, which in turn connect to memory and
IIO models. Several different combinations of these ported models were used for various specific verification test applications as shown in Table I .
The combined ported models run several times slower than the less complex nonported model.
The slower time is offset by increased testing granu larity. The ported models also allow asynchronous behavior, which in turn allows chip- and module level interactions to be tested.
Table I lists more usages of the ported core chip set behavior model as compared to the nonported architectural performance model.
Structural Models
The structural models were derived automatically from the designers' transistor-level schematics. The wirelists, or network descriptions, were translated for the two simulator systems: DECSIM MOS and the ZYCAD simulation engine 9' 10 1 1 DECSIM MOS is a transistor-level simulator based on RSIM and ESIM that models R-C delays, undefined-state initiali zation, and charge sharing.1 2'1 u �
The hardware accelerated ZYCAD system abstracts transistors into a three-state, gate-level model. The DECSIM MOS model was used for standalone chip sections and whole chip simulations to find initialization and charge-sharing bugs. Both DECSIM MOS and ZYCAD were used to find logic and schematic bugs. Both systems used pin-level pattern stimulus that was
Table 1 Combinations of Behavioral Model Configurations
Number of Module
Ported Modules Abstraction
No
No
Yes
Yes
Yes
Yes
Yes
Yes
2
2
Nu mber and Level of Memory Peripherals Applications
Performance
(Microcycles/
Second)
Architectural level; 1 Abstract memory per-cycle detail
Behavioral level for 1 Abstract memory su pport chips; permod u l e phase detail
Behavioral level for support chips module
1 Detailed memory module
Behavioral level for
2 Detailed memory support chips modules
Behavioral su pport 1 Detailed memory chips module with gatelevel bus i nterface
Gate level for 1 Detailed memory bus i nterface chips mod ule with gatelevel bus interface
None
None
Architecture debugging and performance tuning
600
CPU verification 7 and generating chip test patterns
2.5 None Self-test code debugging and bus interface verification
None Mu ltiprocessor verification
1 RL02 disk Booting VMS
(high level) system
2
None
None
Bus interface verification and generating chip test patterns
Mod u le verification
Gate level for 1 Detailed memory bus interface chips modu le with gatelevel bus interface
Gate level for 2 Detailed memory bus interface chips mod ules with gatelevel bus interface
Bus adapter System verification in multiprocessor mode
0.5
66
Vul.
2 No. 2, Spn·ng 1990 Digital Tecbnical]oumal
VAX 6000 Mode/ 400 CPU Chip Set Functional Design Verification generated from the nonported behavioral model.
Signals traced in the behavioral model matched the boundary of the section of logic or chip being simulated at the gate level. Test results were com pared on a cycle-by-cycle basis. The tests uncovered many bugs in the logic design implementation.
Gate-level Fault Simulation
In addition to ZYCAD true-value simulation, single stuck-at fault simulation was done. Fault simulation measured verification and manufacturing test coverage, and provided guidance for verification engineers to enhance tests. The fault simulation effort for the CPU processor chip alone was almost six months long. As a result of this effort, five new tests were written, and manufacturing fault coverage was subsequently increased from 83 per cent to 94 percent.
Verification Strategies
CPU chip set verification engineers had two explicitly stated and somewhat overlapping goals.
We had to prove that the hardware design intent adhered to the VAX architecture standard in every respect, and that the logic implementation adhered to the intent.
' 5
We strongly believed that any bugs in prototype hardware (first-pass silicon for the custom VLSI chips) would negatively impact our ability to meet time-to-market for the product. Bugs found at a later stage of the design process are more expensive to fix for custom VLSI chips. It is expensive because we are severely restricted in our ability to isolate and work around bugs in the hard ware. Therefore, for custom VLSI chips, verification explicitly meant proving the design and finding the bugs in simulation. No single verification strategy or technique can find all of the bugs in something as complex as a VAX CPU. Therefore, a breadth of veri fication strategies were flexibly applied .
In addition to technical strategies, the verifica tion team cultivated a "bugs are good " philosophy throughout the project.
16
Past experience has shown us that bugs will always creep into the design of something as complex as a VAX CPU.
Instead of being viewed as mistakes or failures, bugs were celebrated because a bug found in simulation was a bug that didn' t make it into prototype hard ware. This subtle shift in how the finding of a bug was regarded had, we believe, a strong motivational impact on members of the design and verification teams and increased the probability of finding bugs during verification.
Existing Design Verification Tests
The VAX architecture has undergone a number of implementations since the first VAX- 1 1 /780 system was designed in the mid- 1970s. Over the years, a substantial body of knowledge regarding the key areas and problems associated with designing a VAX processor has been accumulated from various VAX implementations. We put these past lessons to use in the VAX 6000 Model 400 verification effort. We actively sought out bug lists, test plans, and actual test code used by previous VAX system design teams. One key example of this is HCORE, a self checking VAX macrocoded diagnostic program.
HCORE specifically focuses on the high-risk areas that are common across VAX designs. The HCORE test program was originally developed from another basic field-diagnostic program. It was later modified many times, throughout several projects, to focus on testing potential high-risk instructions and functional areas that had been identified in past designs. Existing design verification tests (DVT) such as this are almost always VAX macrocoded tests. Macrocoded tests transport more easily across implementations than microcoded tests because the microword formats are usually different from implementation to implementation.
We derived three benefits from using existing
DVTs. They provided a strong level of confidence in the basic functional operation of the design . Sec ond, they found any functional bugs that might be hiding in obscure or seldom used areas of the VAX instruction set . Third, when used with demons
(explained later), they were useful in finding bugs in very implementation-specific areas, such as error recovery logic.
Custom Design Verification Tests
Although all VAX CPU designs implement the same architecture and run the same software, the hard ware and firmware implementation details of each are unique. Therefore, generic VAX diagnostic tests did not necessarily cover the specific critical paths and functions in the VAX 6000 Model 400 design.
Existing DVTs often could not provide a clear picture of what had and had not been covered . To solve these problems, we used custom DVTs to test specific, obscure, and hard-to-get-at areas of the design. There were several techniques for imple menting custom DVTs:
•
Handwritten macrocode
•
Handwritten microcode
Digital Teclmtcaljournal
Vol. 2 No. 2, Spring 1990
67
VAX 6000 Model 400 System
•
Manipulation of pins and imernal signals under simulation comrol
These techniques could be used individually and in combination, within a single custom DVT. For example, although custom DVTs for the instruction fetch and parse logic (!-box) were written primarily in microcode, a custom macrocoded instruction stream was written to give the 1-box something to parse. Explicit manipulation of pins in simulation was used to generate asynchronous events, such as interrupts, when necessary.
Custom DVTs provided confidence in areas of the design that could not easily be tested with existing
DVTs. We made the tests as focused and efficient as possible. However, in doing so, the generation of such tests required large amounts of development time and people resources. Although these tests uncovered several bugs in all areas of the design, we now believe that many of these same bugs could have been found with less labor-intensive methods, such as pseudo-random tests. At the time, the pri mary adva n tage of custom DVTs was the clear indication they gave that specific functional areas of the design had been tested and were working as specified .
Pseudo-random Design Verification Tests
Each new VAX CPU design aspires to improve on the price or performance of the previous design.
Improvemems are sought by pushing the limits of available technology to package hardware into smaller and , if possible, less expensive spaces. At the same time, a decrease in the cycle time or an i ncrease in the work done per cycle in the func tional design is also sought. In particular, this last item has substantially increased design complexity by imroducing techniques such as pipelining and special-case hardware. As a result of this complex ity, we often encoumer very obscure bugs when debugging new VAX implememations. These bugs involve unamicipated interactions in the logic, between seemingly unrelated functional areas, and interactions dependem on intricate combinations and sequences of evems. We were concerned about these types of bugs because it is extremely difficult to write tests for unanticipated problems. The method we chose to address these problems was pseudo-random testing.
The intent of pseudo-random testing is to exer cise the design in ways that are likely to find bugs without necessarily knowing in advance what those bugs are or where they might be. Pseudo-random testing implies simulating many cycles and trad ing off test efficiency to address the problem of unanticipated bugs.
It is absolutely necessary to automate the pseudo random test process, both test generation and test scoring, as much as possible since pseudo-random tests are much longer than focused tests.
A powerful tool already available for pseudo random testing VAX designs is the VAX architectural exerciser tool suite (AXE and
MAX).17
Originally intended as hardware prototype verification tools,
AXE and MAX have proven to be even more effec tive as design verification tools in a simulation environment. They provide a virtually inex haustible source of unique, interesting macrocode test cases, and require a minimum of intervention and effort by the user.
Although AXE and MAX provide some control over test case parameters, they still aspire to be gen eral, architecturaUy focused exercisers. We also wanted pseudo-random test case generation that could be targeted at specific, risky areas of the implememation. These areas were the most likely locations for unanticipated bugs. Custom pseudo random exercisers were developed for these areas.
These exercisers provided very detailed control over test case parameters, yet retained many of the features and advamages of AXE and
MAX .
A powerful techni
� ue for pseudo-random test is the use of demons.' ' A demon is a n y a u tomated imervention of a simulation model's normal execu tion behavior. For example, a bus demon can imer ject one or more bus commands, error conditions, or interrupts at random intervals in order to aggra vate normal system operation. By doing this, a dense environment of unusual or uncommon event combinations can be created to stimulate the design with worst-case situations. Demons typically slowed model execution by a factor of ten, but they often found bugs that had not been considered by the designers or architects. Without demons, it could have taken many months of field testing to find and characterize these bugs, if they would have been found at all.
Pseudo-random testing with A.,'\E,
MAX, custom exercisers, and demons was used throughout the development cycle of the CPU chip set. All four uncovered obscure interaction bugs, as expected.
Unfortunately, they did not find them all. Some unanticipated bugs slipped through verification and into the first-pass silicon stage. These bugs were eventually found after running many more cycles in real prototype hardware. The failure to find all
68
Vol. 2 No. 2, Spring 1990 Digital Tecbnicaljournal
VAX 6000 Model 400 CPU Chip Set Functional Design Verification unanticipated bugs in the simulation stage illus trates a fundamental problem in the use of pseudo random testing. The effectiveness of testing is closely coupled to the number of cycles run, and simulation speed severely restricts this number.
Our application of pseudo-random testing to the problem of unanticipated bugs was largely success ful for this project. However, we learned that we must do more in the future to increase the efficiency and scope of these tests. To provide this increase, we are looking toward more d irected pseudo-random testing.
Booting the VMS Operating System
A major milestone in the development of any new
VAX CPU is booting the VMS operating system on prototype hardware. Not only does this demon strate significant functionality in the design, but it also provides a platform from which further testing can proceed. As previously stated, increased com plexity in these designs can produce very subtle bugs. Often, such bugs do not even appear until the hardware is run under a heavy system load in a large multiprocessing or I/O-intensive environment. Suc cessfully booting VMS on prototype hardware is necessary before any such system load testing can begin. The sooner such resting begins, the better the chance of finding subt le bugs. For this reason, boot ing the VMS system in simulation was an important goal of the VAX 6000 Model 400 chip development and verification effort.
At first glance, it would seem impossible to boot the VMS system on a simulation model in a reason able amount of time. Simu lation speeds on the fastest of our models were many orders of magni tude slower than actual hardware. However, careful analysis of the macrocode modules involved in the boot process revealed that by optimizing or remov ing large, iterative sections of code, we could sub stantially reduce the number of cycles in the boor path without losing significant coverage for veri fication. For example, the primary bootstrap mod u le contains code that creates a bit map of all physical memory in the system . The code tests each memory location for errors. By reducing this code to test fewer memory locations, the number of cycles executed is vastly reduced.
Another key optimization involved speeding up simulated transfers from clisk. At several points dur ing the boot process, code and data are pulled into memory from a mass storage device (usually a disk).
For simulation, a special " turbo" disk model was written by the verification team. This model artificially processes model requests for block data transfers and performs rhe transfers instantly. This technique eliminated wasting simulated CPU cycles while waiting for transfers to complete.
With model and code optimizations in place, booting the VMS system to the point of printing the
VMS banner and starting the process scheduler was actually achieved in simulation. It took approxi mately seven CPU days on a host V�'< 8800 system .
One model bug and one real design bug were found during this effort . Booting the VMS system in simu lation requi red a large amount of verification resources. The process was worthwhile in terms of the bugs that were found and the confidence it gave us that we could boot VMS on first-pass hardware.
Analysis of the Functional Bugs Found
To demonstrate where the verification project was successful and where it needed improvement, we discuss here the bugs that were found both before and after the chip set first-pass design was commit ted to masks. (Note: The milestone of this commit ment is called PG , for mask data preparation's pattern generation .) Only the bugs pertaining to architecture, microcode, functional design, and logic design are discussed. Layout and circuit prob lems, as well as modeling and tool bugs, are outside the scope of this paper.
During the design period, several hundred bugs, with a variety of complexity levels, were found in all sections of the design. These bugs were found through the verification techniques described in previous sections and are detailed in Table 2 .
The prototype chips were first tested on a Takeda
3381 chip tester. This tester allowed prototype chips to be tested in a standalone environment. The chips were then i nserted into a prototype CPU module, which was part of a custom-designed engi neering tester. The prototype module provided a
Table 2 Methods Used to Find Design Bugs before First-pass Silicon
Verification Process
Number of
Bugs Found
Custom DVTs, DVT reviews, microcode assertions
Existing macrocode test programs
Pseudo-random macrocode tests
Boot VMS operating system on the behavioral model
Total of bugs found prior to first-pass si licon
273
1 71
90
535
Digital Tecbnicaljournal
Vol. 2 No. 2, Spring 1?90
69
VAX 6000 Model 400 System true system environment for the chip set 1H During p rototype debugging, 1 1 bugs were found. The characteristics of these bugs are shown i n Table 3.
None of these bugs was a "show-stopper" in terms of prototype debugging or field testing. In fact, most of them were so obscure that the proba bility they would appear during normal use was very low. Several generalizations and conclusions can be drawn from the types of bugs found and how they were found, particularly whether they could have been found earlier.
•
Bugs 1 , 2, 6, and
9 were simple, but were missed because of overlooked test coverage. The sim plest test, if identified and written, would have found them. We learned that we needed more discipline and thoroughness in generating lists of tests as guided by test coverage indicators.
•
Bugs
4, 5, and
11 were found on the simulator after prototypes were available. It is questionable w hether these bugs would ever have been noticed, much less isolated, on shipped systems because the conditions triggering them were obscure. For example, six conditions, including a parity error interrupt, had to happen simul taneously to reveal bug 4 . It was acceptable to find these bugs after PG because the impact from such bugs on prototype debugging was minimal.
•
Bug
3
was found using prototype hardware rather than simulation. A new version of
MAX
was released duri ng the prototype debugging time frame. This version had some new test coverage features, and we decided to run them
on the hardware at-speed as opposed to using simulation. One of the new features stimulated the conditions for bug 3. Had this lV1AX feature been available p rior to PG , we would have encountered the bug in simulation w ithin the first few tests. Typically,
90 percent of the bugs uncovered by using the 1V1AX tool are found with the first 1
00 tests generated.
•
The verification methodology for bug 10 was correct, but had not been followed . Appropriate gate-level comparison testing would have found this bug prior to
PG .
•
Bugs 7 and 8 were extremely obscure and diffi cult to find . The first symptom of these bugs was a series of unexplained system crashes over a period of several weeks. The situation was finally resolved by attaching a logic analyzer to a system and waiting for days for the right combination of events to trigger the failure. Unfortunately, it is highly unlikely that we would have found either bug i n simulation, even with a broader scope of directed-random verification, given the limited amount of simulation that was done. The architectural, microcode, and pipeline combi nations required to trigger these bugs were just too complex.
Conclusions
Although there were 1 1 bugs in the ftrst-pass chip set, the verification efforts were considered quite successful. Prototype debugging was never stopped because of a bug, and system field testing was able
Table 3 Bugs Found after Pattern Generation
Bug
Number Chip Type Bug Type Complexity
6
7
8
9
10
1
2
3
4
5
1 1
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
Floating point accelerator
Cache controller
Microcode
Microcode
Logic
Microcode
First order
First order
Sequential
Second order
Microcode
Microcode
Second order
First order
Logic/microcode Second order
Microcode
Logic
Logic
Second order
First order
Sequential
Logic Second order
How Found
Inspection
Inspection
MAX on prototype
Pseudo-random on simu lator
Pseudo-random on simulator
Self-test on prototype
Field test system crash
Field test system crash
Prototype debugging
Chip tester comparison
System verification test on simu lator
70
Vol. 2 No. 2, Spring
1990
Digital Technical journal
VAX 6000 Model 400 CPU Chip Set Functional Design Verification to complete on schedule. The modeling and veri fication methodologies contributed to this success.
•
All logic, functional , and microcode bugs could be reproduced in the behavioral model. There fore, the bugs found after PG could have been found w ith simulation had the appropriate com bination of events been tested.
•
Complex pipeline activity, an area of concern from the beginning, was the primary problem area in the design. Compromising design com p lexity to make thorough verification more achievable should be considered.
•
In general, the tool or test that first exercised a chip section or functional area with a bug found the bug. We believe this indicates that it is more p roductive to first debug using existing macrocode tests or automatically generated tests before writing new tests.
•
Although the effort to boot the VMS operating system on the simulator was comparatively large, one bug was found that would not have been found through other means. We had not considered the combination of events that caused the bug. Finding this bug clearly showed the benefit of running this application on the simulator. Further, it gave us confidence that the
VMS operating system would boot on the first pass prototype hardware to ensure that proto type debugging could proceed unimpeded.
•
The flexible and wide-ranging modeling meth odology served the design team well. The source code of the CPU chip set model has been reused in system verification and chip set application development projects at least ten times. The cor porate standard DECSIM logic simulator made this modeling effort savings possible.
Acknowledgments
The verification of the CPU chip set design was a team effort performed by the Semiconductor
Engineering Group's m icroprocessor verification team. Members of this team included David Asher,
Rick Calcagni , Liza Hudepohl, George M ills, Rick
Pekkala, Ed Rocha, Will Sherwood, and C h ris
Spear. Part of our success is attributed to our CAD tool development groups, including the SEGCAD
DECSIM team and the VAX Architecture Group AXE/
MAX team. We would also like to thank Mike Uhler for his technical guidance throughout the project.
References
1 . ] . Basmaji et a l . , "The Role of Computer-aided
Engineering in the Design of the VAX 6200
System," Digital Technical journal, vol. 1 , no. 7
(August 1988): 47-56.
2. C. W iecek, "The Simulation of Processor
Performance for the VAX 8800 Family," Digital
Technical journal,
1 987): 100-1 10. vol. 1 , no. 4 (February
3 . C. Wiecek and S. Steely, "Performance Simu lation as a Tool in Central Processing Unit
Design," Performance Evaluation Review, vol.
1 1 , no. 1 (August 1 979): 4 1 -47 .
4 . B. Moses and K . DeGregory, " Performance
Evaluation of the VAX 6200 Systems," Digital
Technical journal, vol. 1 , no.
7 (August 1988):
64-78.
5 . M. Kearney, " DECSIM : A Multilevel Simulation
System for Digital Design," Proceedings of
ICCAD (October 1 984).
6. R. Gries and ]. Woodward, "Software Tools
Used in the Development of a VLSI VAX
Microcomputer," Proceedings of the MICR0-17
Conference
(October 1 984).
7 . W. Sherwood, "An Interactive Simulation
Debugging Interface," Computer Hardware
Description Languages and Their Appli cations, Breuer and Hartenstein, editors,
(Amsterdam: North-Holland Publishing, 1 98 1 ) :
1 37- 144.
8. P. Sullivan et a!. , "The VAX 6000 Model 400
Scalar Processor Module," Digital Technical journal,
27-3 5 . vol. 2, no. 2 (Spring 1 990, this issue):
9. B . Milne, " Put the Pedal t o the Metal with Simu lation Accelerators," Electronic Design, vol. 35, no. 21 (September 1 987): 39-52.
10. M. McMahon, "Accelerators for Faster Logic
Simulation: The ZYCAD Approach,"
Proceed ings of
VLSI and Computers, First Interna tional Conference on Computer Technology,
Systems and Applications (COMPEURO '87 Cat.
No. 87CH24 17-4 , 1 987): 98 1 .
1 1 . D . Jenkins and S . Morton, "Transistor-Level
Logic S imulation Using the ZYCAD Logic
Evaluator," Proceedings of the Conference on A utomated Design and Engineering for
Electronics ( 1 985): 1 54- 163.
Digital Technical journal Vol. 2 /Vo. 2, Spring
1990
7 1
VAX 6000 Model 400 System
1 2 . C. Terman, " Timi ng Simulation for Large
Digital MOS Circuits," Computer-aickd Design of VLSI Circuits and Systems,
JAI Press, 1 986).
(Grc<:nwich :
1 3 . C. Terman,
" RSI M - A
Logic-Level Timing
S imulator,"
International Conference on
Computers and Design
( 1 983).
14 . C . Terman, "Simulation Tools for Digital LSI
Design," Technical Report
TR-304,
MIT/LCS
(Cambridge: MIT Laboratory for Computer
Science, 1 983).
1 5 . T. Leonard, ed . , \1,4X A rchitecture Reference
Manual (Bedford : Digital Press, Order No.
EY-3459E-DP, 1 987).
16. D. Clark, " Bugs are Good: A Probkm-oriented
Approach
to
the Management of Design Engi neering,"
Research - Technolof!Y Management
(forthcoming 1990).
17. D. Bhandarkar, "Architecture Management for Ensuring Software Compatibility in the
VAX Family of Computers," IEEE Computer
(February 1982): 87-93.
18. W. Sherwood , "The
V LSI VAX
C h ip Set Micro architecture,"
Microarchitecture of VLSI
Com
puters,
P. Antogneui, ed . ( 1 985): 103-1 26.
72
ViJI. 2 No. 2, Spring 1990
Digital Tecbntcaljournal
john
Larry
W.
Croll
T.
Camilli
A nthony]. Vaccaro
Test and Qualification of the
VAX 6000 Model400 System
Computer-aided design simulation, which is used in the design of the VAX
6000 family, finds most problems during the hardware design phase. Simulation, however, cannot test a complex system running under system software control. For the
VAX
6000
Model
400 system, a qualification process was designed to completely test the interaction of the system s hardware and software components. The benefit of such a process is clearly shown in the results. Nearly all the problems found in the qualification stage could not have been found in the simulation process. The testing and qualification of the Model 400 was a multigroup effort.
This paper describes the methods and tools of three Midrange Systems Engineering groups who were involved in the project.
The VAX 6000 Model 400 system is the third in the
VAX 6000 series. The Model 400 is designed to enhance CPU performance using the same platform, that is cabinetry, buses, and power systems, as all models in the
VAX
6000 family. The basic architec ture of the VAX 6000 Model 400 is unchanged from that of the earlier Model 200 and 300 systems 1 2' 1
The Model 400 is distinguished from earlier models primarily by two additions:
J. new processor design, which offers over twice the performance of the original VAX 6000 Model 200 processor, and by the addition of a vector coprocessor. '·' The interfaces in the Model 400 to the common platform remain the same. However, the processor is an entirely new design, and is the first Digital system to use semi conductors designed with the CMOS-2 process 6
A l l VAX 6000 systems use a common design pro cess that relies heavily upon simulation ro detect and correct design errors. This simulation process is designed to ensure that first-pass hardware, or the initial engineering prototypes, will run the operat ing system at speed.- The VAX 6000 Model 400 processor is an excellent example of t he effec tiveness of simulation techniques. The elapsed time from power-on of the first prototype to reliable login under both the VMS and LJLTRIX operating systems was less than six weeks, w hich is sub stantially less time than has been seen for previous
VAX processors.
Current computer systems are very complex, especially when hardware and software interac tions are considered. Simulation cannot adequately test hardware executing under operating system control . A companion system test and qualification process, executed on h ardware prototypes, is required to thoroughly test the complete system and ensure its reliability.
A key goal of the
VAX
6000 common-platform design strategy was to allow new processor tech nology to be brought to the market quickly. To achieve this goal, we had to mini m ize the time required for system test and qualification without compromising the quality or reliability of the final product. By reusing common platform compo nents, we could primaril y focus on testing the new components.
This paper addresses the system test and qual ification process used for the VAX 6000 Model 400 system. This process was designed to maximize test effectiveness and minimize test time. The first cus tomer shipments of the VAX 6000 Model 400 system occurred onl y six months after the introduction of its predecessor, the
VAX
6000 Model 300 system .
System Test and Qualification Process
The VAX 6000 Model 400 System Integration Group is responsible for the overal.l design and manage ment of the system test and qualification process.
This group comprises engineers who reside within the hardware design group, but have not directly participated in the design of the components being tested. Therefore, any problems found during the test period can be rapid ly communicated and resolved, w hereas the possibility of " testing to
Digital Tecbnicaljournal
\11/. 1 No. 1, Spring J<J90
73
VAX 6000 Model 400 System implementation" versus " testing to specification" is avoided.
A distributed test process was developed for the
VAX 6000 Model 400 system that used the resources and expertise of a variety of groups within Digital.
The test process also allowed many tests to be exe cuted in parallel to minimize time. Additionally, some of these groups have specialized test facilities that are required to satisfy standards imposed by
D igital and various government regulatory bodies, e.g. , the FCC and U L .
The groups that participated in testing the
VAX 6000 Model 400 system, together with a brief description of each group's function, are shown in
Table 1 . Each group played a valuable role in testing a particular aspect of the system. Initially, a very aggressive test schedule of six months for the entire test and qualification process was planned. How ever, clue to some delays in planned prototype availability, all testing had to be completed in about five months to allow systems to ship as scheduled in
July 1 989. The expertise of all the groups involved was required to meet this schedule. However, space
limitations prohibit an in-depth discussion of each function. Therefore, we will examine in-depth the roles of three of the Midrange Systems Engineering
Groups - the
VAX
6000 Model 400 System Integra tion Group; the
VAX
Architecture Verification
Group; and the M idrange Systems Evaluation Engi neering Group. We w i l l also describe the tests and tools used in the qualification of the
VAX
6000
Model 400 system.
System Integration Group
Planning for the system test and qualification of the
VAX 6000 Model 400 system began approximately one year prior to scheduled availability of the first prototype systems. During this period, the overall qualification plan was developed, and individual test plans were solicited from each group that would participate in the testing. Each plan was reviewed by the System Integration Group for test coverage, as well as for minimization of overlaps and duplication in the component plans. In parallel, the System Integration Group developed plans for hardware-specific system design verification tests
Table 1 Organizations I nvolved in System Qual ification
Organization
System I nteg ration
VAX System Arch itecture
M id range Systems Evaluation
Eng inee ring
VAXcluster Validation
Mechanical Tech nology
E lectromagnetic Compatibility
E ngineering
P roduct Safety Laboratory
Diagnostic Qual ity Assurance
M an ufacturing P roduct
Verification
Central Characterization G roup
System Performance Analysis
G roup
Customer Services Systems
Engineering
Software Qual ity Mai ntenance
Systems Reliabil ity E ngineering
Function
Manage qual ification process; perform DVT, system test, reliabil ity confidence test; manage field test
Exercise new system to find discrepancies with the VAX arch itectu re
Test a variety of system configu rations for proper operation , concentrating on 1/0 configu rations
Test complex VAXcluster envi ronments with new products to find p roblems in new or existing VAXcl uster com ponents
Demonstrate successful product operation while exposed to specific environmental conditions (e. g . , vibration, h u m id ity, altitude)
Test new p roducts for electromagnetic compliance to various government regu latory requ i rements (e. g . , FCC , VDE)
Test various aspects of new product safety and ensure compliance to safety standards (e . g . , U L , GSA)
Test various diagnostic programs to ensu re correct operation
Provide the necessary data to verify and improve the man ufacturing process to enable consistent production of a qual ity product
Characterize performance of industry and appl ication-specific workloads
Model and measure performance of m i d range products
Develop service delivery and training p rograms, test d iagnostic and repair features of products
Test software-layered product operability with new hardware and operating system versions
Predict, test, and analyze hardware and system reliabil ity
74 Vol. 2 No. 2, Spring
1990
Digital Tecbnicaljournal
Test and Qualification of the VAX 6000 Model 400 System
(DVTs) and for external field test. Prowtype plans specified the number, distribution, and cost of the prototyped systems that would be required during the test period.
The System I ntegration Group also acted as the problem-reporting center. In implementing a dis tributed test process, two functions are essential.
There must be a central focus that disseminates information regarding observed problems to all test groups. Second, an established method for tracking status and resolution of these problems must be maintained. An internally designed system maintained a complete audit hiswry of each prob lem. There were 1 2 1 problems reported during
VAX 6000 Model 400 testing. Each step in the reso iution process was tracked for each problem in the problem database. This database was available to all test groups. The database was supplemented by weekly cross-functional meetings, at which repre sentatives from engineering, manufacturing, and customer services reviewed and updated each problem's status.
The complex VAX 6000 Model 400 project sched u le was developed and tracked by the group. Due to the number of groups involved in testing and the delays involved when problems were found, the critical path tended to be very dynamic during the test phase. A project management tool, which was developed within Digital and which used PERT, tracked status against milestones and modeled dif ferent scenarios to prevent overall schedule slip page as changes occurred.
The System Integration Group performed four major types of test. These were system test, reli ability confidence test, design verification tests, and field test. The following sections describe each of these tests.
System Test
There are two forms of system testing, directed and random. Most testing groups use directed tests, which test hardware or software features, or follow a strict test sequence. Directed tests seek specific results and are well defined .
The System Integration Group performed many directed tests on the Model 400 system. Some of these tests were done to satisfy the requirements of external regulatory agencies, or internal Digital development standards. Other directed tests include system DVT tests, which are discussed in more detail later in this paper.
Many aspects of complex systems cannot be ade quately tested i n a directed fashion . For example, an operating system and a processor can operate in a nearly i nfinite number of states. It is impossible to design a series of tests to verify each of these states.
Random tests exercise the system in more com prehensive ways than directed tests. They do not seek specific results. I nstead, random tests attempt to push the system into as many different states as possible, as quickly as possible. Greater test cover age results from these tests, but problem diagnosis and isolation are more difficult.
Because random testing does not look for specific results, it is effective only if done for extended periods of time. Even if identical test scripts are run repeatedly, system activity becomes unpredictable over time, due to events such as network activity or disk fragmentation. This unpredictability is impor tant because it means more system states are being exercised.
The System Integration Group developed a ran dom test package, called the Systems I ntegration
Test Package (SITP), to test the VAX 6000 Model 400 system. This package consists of a comprehensive collection of test programs and a script-driven mechanism that controls their execution. SITP is diverse and flexible. The test programs were obtained from many sources. The System Integra tion Group also wrote some test programs to exer cise specific aspects of the Model 400 system that were not fully exercised by other tests.
The test programs used with SITP are high level.
Each high-level test uses many lower level functions within the system. Many of these programs are run together, with varying test parameters and run times. The programs are self-checking. If an action does not complete properly, the program nores the error immediately. The program does not attempt to identify the cause of an error; rather, it gathers as much information about the error as possible. This information is later examined by a test engineer.
SITP is easy to use, restarts automaticaUy after system crashes or power failures, and includes mon itoring tools. Periodic reports , with details about system activity and error log data, are generated by the test package. With this i nformation, the test engineer can gauge the effectiveness of the tests and adjust them as necessary. The test engineer can also control and monitor tests on many different machines. Machines can be located locally and remotely.
A number of SITP scripts were developed to provide different workloads for testing the Model
400 system. Each set of scripts emphasized a dif ferent type of system activity. Some were com-
Digital Tecbnicaljournal Vol. 2 No. 2, Spring
1990
75
VAX 6000 Model 400 System pure intensive, some I/O intensive, and some stressed parallel and multiprocessing activity. The scripts were modified to suit system configurations as needed.
SITP and the test scripts were installed on all the
Model 400 system prototypes in the system integra tion lab . Tests under the control of SITP were run on the prototypes as prototypes were avail:lble.
Because the prototypes were heavily used during daytime hours for various debugging tasks, SITP tests were run overnight and on weekends. Tht: test scripts were designed to run for a specific number of hours and thm stop. The prototype was then available for the next user. This procedure allowed otherwise idle prototype hours to be used in system testing and ensured a clean shutdown of the tests. In this way, test data could be retrieved without inter ference from other prototype users.
SITP was used on the earliest Model 400 system prototypes and was continually used throughout the qualification period. as prototype time was avai lable. Scripts wen: t:�ilored to cause test con centration in specific areas and were modified as necessary to suit various prototype configurations.
Typical
SlTP runs would last for 16 or 19 hours
(overnight), or
58 or 60 hours (over weekends).
Processor, memory, and 1/0 configurations varied from run to run, and depended on test needs and equipment availability.
The overall results from system testing were very positive. Over 6700 CPU hours were accumulated on various prototypes and configurations. Many errors were encountered during this period, but most were due to SITP bugs (SITP was still under development for most of this period) or to errors in setting up test scripts. Hardware errors occurred in peripheral devices, principally disks and com munications devices, and were corrected as they occurred.
Of the mort: serious problems found, one was a hardware problem that would cause a system hang.
The problem was identified as a bug in a bus inter face chip on the CPU module, which was operating in an untested mode. It was resolved by modifying the system console to ensure that this mode was never used. An error was found in the
VMS machine check handler, which was corrected in a subse quent release of the VMS operating system.
Five other serious bugs were found in the new
CPU modules. Although none of these bugs were found by the System Integration Group's testing, each took time to invcstipte, resolve, and test the fixes. As a result, there was less time available on protOtype machines for other testing. Two of these bugs were fixed by modifications to the CPU mod ule. The other three required changes to the proces sor chip. As corrected processor chips became available, SITP was used tO ensure the fixes had not introduced further bugs.
It is interesting to note that fou r of these five problems occurred in system areas not simulated during hardware design. Of these four , two occurred in the handling of external system events.
One was in system reset handling. The other was in handling "control/P" interrupts. Conrrol/P is the standard method an operator uses to get the atten tion of the system console on VAX systems H Two bugs were caused by interactions between the new processor and other system components. These interactions were not simulated during hardware design. The fifth bug was not found during simula tion because of a deficiency in a simulation test tool.
Reliability Confidence Test
To accumulate uninterrupted run-time on the
Model 400 system, five identically configured sys tems were set up in an isolated area. The machines were isolated to protect them from outside inter ference while rhe confidence test was running.
The purpose of this test was to derermine the actual reliability parameters of the Model 400 system hardware and to compare the results to the system's actual reliability requirements. A second ary goal of the test was to determine the long-term system reliability, both for the hardware and oper ating system software.
The duration of the test was planned for SL'X weeks, which was sufficient to show the hardware reliability. Once this six-week period was over, we planned to continue to run the machines in the same environment with the same workload for as long as possible to accumulate further system run-time.
The test started at the beginning of May
1989, when enough CPU modules became available to populate the five machines. The formal test period ended two months later, in late June. Three of the machines continued to run for two and a half months, until mid-September. Of the other two machines, one machine was needed for other pur poses, and another's CPU modules were removed to change the configurations of the remaining three.
The systems ran identical SITP scripts that con centrated on exercising the new CPUs. Tests included compute-intensive programs, programs that explicitly tested various aspects of the new
76
Vol. 2 No. 2, Spring /'J'.JO Digital Tecbnicaljoun1.al
Test and Qualification of the VAX 6000 Mode/ 400 System
CPUs (e.g. , multiprocessor cache coherency), some decomposed parallel applications, and tests that generated many VMS processes.
The overall results of these tests were very good.
The systems demonstrated a hardware reliability of over a year between hardware failures.
Only two module failures occurred. One had a bad cache tag store, which was discovered very early in the test. As a result of this discovery, the test process for the CPU module's cache control chip was changed. The other module failure was a float ing point chip fai lure. One specific test program began generating wrong answers late in the formal period of the confidence test. This module was removed for repair.
Of the other failures that occurred during this test, all were attributable either to test script or set-up errors, or to software or hardware prob lems which were corrected prior to shipment to customers.
Once the formal test period was completed, the three machines that continued to run u ntil m id
September exhibited no new failures and eventu ally accumulated a year's run-time. The reliability of the VAX 6000 platform and the Model 400 CPU was successfully demonstrated.
Design Verification Tests
Part of the System Integration Group's responsibil ity is to ensure that parts of the system not covered by tests from other groups are tested. In general, these parts are specific either to the new hardware or to how the new hardware fits into the existing system. These tests are called design verification tests (DVTs).
A complete test plan for the Model 400 system
DVTs was written and reviewed by the group. The list of DVTs performed is shown in Table 2 .
The Model 400 DVTs complemented those tests performed on the new hardware components. The parts of the system tested were those in which other testing was weak or nonexistent. These tests were conducted i n a formal manner, w ith a w ritten sequence of events and formal reports of results.
Any problems found were noted and reported to the relevant development groups for analysis and eventual correction .
The DVTs executed for the Model 400 uncovered two system bugs and some minor documentation problems. Both bugs were related to power failure recovery. One was in the console and one in the
VMS operating system . Both bugs were eventually fixed. The minor documentation problems in the system installation guide were also fixed. All other design verification tests found no problems with the system.
Field Test
Field tests are made on prototypes of new products provided to customers. The purpose of field test is to gain experience with the new products before production; new products are actually used as opposed to tested.
Table 2 VAX 6000 Model 400 System Design Verification Tests
Test
Keyswitch
Voltage marg i n
Thermal
Power-fail/battery backup
I nterlocks
X M I saturation
Queue contention
M u lti processing
Load test
System instal lation
Boot
System configuration
Remote services console
Console input
Fu nction
Verifies front panel fu nctions with new hardware/software
Verifies proper operation over allowable voltage range
Verifies proper operation over allowable temperature range
Tests battery backup operation with hardware and software
Tests memory interlock fu nctions
Tests system operation u nder very heavy bus loads
Tests proper operation under very heavy worst-case memory loads
Tests proper operation of all multiprocessing features
Runs system for extended period under heavy com pute and 1/0 load
Verifies overall manufacturing process, and i n stallation documentation
Verifies that system boots all operating systems from all devices
Verifies that all allowable configu rations work properly
Verifies that remote services console hardware and process works with new system
Verifies that console term in al hardware and software work properly
Digital Tecbnicaljournal Vol. 2
No.
2, Spring
1990
77
VAX 6000 Model 400 System
For VAX system products, field test historically has lasted a m inimum of four months. This period was determined from tracking problem reports from field test sites. General ly, a month was required for the new product to be installed and for usage to reach a level where meaningful testing occurred. The next three months provided useful data about the new system. After three months, the amount of useful data declined.
In field testing the VAX 6000 Model 400 system, we shortened this four-month period to three. The plan was to check on field test results two months into the test. If field test was not progressing well at this point, we were prepared to extend the test period.
The System Integration Group did two things to eliminate the field test startup time. First, because the Model 400 system was an upgrade from earlier
VAX 6000 systems, Model 200 systems were shipped to each field test site in advance of the
Model 400 field test start. These systems were installed and running approximately a week before the official field test started . A considerable amount of time was saved in site preparation and system installation.
Second, the startup of the field test sites was performed by system integration engineers, w ho brought Model 400 CPU modules and new VMS software to each site. These engineers supervised the installation of the new CPl 's in the previously installed systems, installed the VMS operating sys tem, and ensured that the systems were available to the customer before leaving the site.
A total of seven field test sites were started up in early April 1989. Six of these sites were located in the United States. Five of these sites were instal led and turned over to the customers within two days.
The sixth site was ready in four days. The seventh site was located in Europe, and was started up by mid-April .
Once the site systems were running, the System
Integration Group maintained regular contact with each site. Each site was assigned a "captain" (a sys tem i ntegration e ngineer), w h o polled the site weekly, talked to the users, and received first-hand information about machine usage. This method was used instead of the traditional dial-up problem reporting method for two reasons. First, technical problems existed in making reliable connection to the dial-up system. Second, many people are reluc tant to report problems, unless the problem is so large that work stops or is severely impaired .
Overall, field test went smoothly. Most of the problems that occurred were minor and easily corrected . Two problems arose because the Model 400 system was different from the other VAX systems in use at two of the sites. One site reported m inor dif ferences in results from a benchmark program, which was due to differences between run-time libraries used for full VAX architecture implementa tions and subset implementations.H The other prob lem resulted because a customer program was referencing an internal processor register that does not exist in subset implementations.
Field test progress was assessed two months into the test, at the beginning of June. The results of field test were then exa m i ned, together w i t h the data obtained from other qualification testing. Since no major problems had been found, we decided to pro ceed with plans to ship the Model 400 system as scheduled in mid-July.
Some improvements could have been made in the field test process. First, site audits prior to instal lation of the prototypes were not very thorough.
Many of the sites were not running the neces sary software revision levels. Therefore, the new machines could not be immediately put into
VA.Xcluster environments with existing machines.
Second, some pieces of hardware were missing and had to be supplied later. Better communications with the site prior to shipping prototypes would have reduced these problems.
The System Integration Group wrote some moni toring software that was to be installed on each field test machine. Because this was written at the last minute, it was not properly tested and did not work properly. The problems were fixed, but the moni toring software was not run at all the sites. Finally, how to get data from the monitoring software back to the engineering groups was not well defined.
Thus, usage data obtained from field test machines was spotty. These tools and methods are being improved for use in the field test of future products.
VAX Architecture Verification Group
The Architecture Verification Group ensured that the Model 400 CPU conformed to the VAX architec ture specification.H
Test Process Overview
The architectural verification process consists of running two programs - AXE (architecture exer ciser) and MAX (multi-instruction architecture exerciser) - in various modes for a given number of test cases. The tests are simple to run. However, the test programs are quite complex and required many years to develop. The Architecture Verification
78 llri/. 2 No. 2,
Spn'np, /l)'JO
Digital Tecbnicaljournal
Test and Qualification of the VAX 6000 Mode/ 400 System
Group maintains and enhances these test p rograms and the databases used to verify the architecture.
When one of the tests fails, the group identifies the problem and helps resolve it. Once the problem is fixed, the group repeats the tests.
AXE andMAX
The VAX instruction set consists of over 360 instruc tions and 21 addressing modes. Most modes are valid for up to six operands per instruction . Con ceptually, both AXE and
MAX divide an instruction's context into several components. These com ponents include opcode, operand specifiers, oper ands, page protection and validity, and processor status long word (PSL). For each component, valid and invalid values are pseudo-randomly selected to create a test case. The exerciser continues to create unique cases for as long as it is run.
The VAX architecture has a clearly defined excep tion and instruction restart structure. Much of the
VAX architecture's complexity is in those opera tions. Therefore, both AXE and MAX favor values that cause faults. Each program establishes a situa tion with faults, starts the instruction or sequence, verifies that the fault occurs, ftxes the fault, restarts the instruction, and verifies that it completes cor rectly. Upon completion, AXE or
MA)( compares the results from the unit under test to a known good reference, and reports any differences. The known good references contain the correct results of each test case. These references have been accumulated over the years of testing VAX systems and have changed as the VAX architecture changed.
AXE AXE is the older and simpler exerciser. It cre ates test cases that consist of a single instruction. A typical instruction stream would be:
ADDF3 R 1 , R 2 , 5 " ' 1 R 1 . 0 0 0 0 8 0 0 0 , R 2 = 4 6 2 4 68 1 1
When this instruction is first executed, either a reserved operand fault (on
Rl)
or a reserved addressing mode fault (short literal destination) should be reported. AXE w ill fix whichever fault is reported and restart the instruction.
Assuming the reserved addressing mode fault was reported, the instruction might then look like this:
A D D F 3 R 1 , R 2 , R 3 R 1 . 0 0 0 0 8 0 0 0 , R 2 : 4 6 2 4 68 1 1
If the reserved operand fault is reported, AXE will change R 1 to a valid floating point value and restart.
When the instruction completes, AXE compares the i nstruction stream and the relevant data to reference data. In this example, the relevant data includes the three general purpose registers.
Both AXE and
MAX ignore a machine state defined to be unpredictable for a given condition.
Therefore, allowable differences between imple mentations do not cause erroneous fai lure reports.
Limiting AXE's testing to single instructions pre cludes meaningful testing of the pipelining that is common in today's
C P U designs.
MAX overcomes this l imitation .
MAX currently acts as an adjunct to
AXE. However, it will eventually replace AXE .
MAX
MAX is similar to a compiler in that it creates complete instruction streams. However, MAX does not have source code to ensure that the resultant machine code is logically consistent.
To test how a
C P U handles inter-instruction data dependencies,
MAX must create test cases with instructions that share registers and memory loca tions. Creating sensible instruction streams can be difficult. For instance, the result of one instruction could be used as part of an address calculation for a subsequent instruction. However, the likelihood is slim that the result of a randomly selected arith metic calculation w ill be used within the test case's virtual address space.
MAX first creates and executes sensible cases i n logical steps. I t then assembles and executes each case as a whole.
After selecting the fi.rst instruction, MAX executes the instruction, including fault restarts , and saves the final results.
MAX next selects and places the second instruction in memory following the first instruction. Where possible, it uses the results of the first instruction for operands and operand specifiers of the second instruction. MAX selects new values for operands and specifiers for which old values cannot be used.
MAX includes the new values in the initial state of the entire case. It then executes only the second instruction. This process repeats until an instruc tion stream of the desired length is created. At this point, the entire stream is executed. Once the stream is run, the results of all of the instructions are compared. This comparison is made against the results of the single-step execution and the results of the known good reference.
Test Process
The minimum testing requirements for shipment of VAX systems were developed from experience
Digital Tecbntcal]ournal Vol. 2
No. 2, Spring 1990 79
VAX 6000 Model 400 System gain<.:tl from testing all previous VAX systems. The requirements are a compromise between the num ber of bugs l ikely to be found and the time required for the tests. The testing program consists of over
200 million test cases.
The use of AXE and 1'<lAX in the syst<.:m qualifi cation process is a continuation from the CPU design process. Both exercisers were used <.:xten sively to test simulations of the
CPU d<.:sign. How ever, this testing is limited by simulation speed.
Test ing on hardware at speed is necessary for more thorough coverage.
Testing of rhe VAX 6000 Model 400 started in late
April. To complete verification in less than two months, four machines were used. The machines executed different test cases in parallel.
The VAX 6000 Model 400 CPU was the ftrst VAX
CPU for which no hardware bugs were found by
AXE or MAX during final qualification. This result is a testament ro the careful design and extensive test ing during the simu lation and chip debug phases of the CPU design.
However, a bug was found in the VMS operating system's floating point emulation code that calcu lates the
400 CPU
POL
Yx instructions.H Because the Model is the first thar did not include the POL instruction in the CPU microcode, it was the first
CPU that required use of the VMS emulator for this instruction. For this reason, the bug had never been seen before.
The Architecture Verification Group traced the bug and confirmed that a patch to the emulator fixed the bug. The group then reran all of the test cases that exercised the floating point emulator to confirm that no new bugs 'ivcre introduced. The patch was included in the VMS version 5.2 manda tory update.
Midrange System Evaluation
Engineering Group
The Midrange System Evaluation Engineering
Group tests new products to isolate design faults and configuration incompatihilities. This group examines various system configurations, rests inter action between components, and tests special situa tions. such as power failure and recovery.
Test Process Overview
System evaluation typically begins when the first hardware and software prototypes become avail able and continues through to product shipment.
Evaluation planning often begim six to nine months prior ro the actual pron:ss with the creation and review of a test pla n . The rest plan is based on product specifications and information from devel opment groups. It identifies the test tools, configu rations, and strategy the System Evaluation Group will use.
The System Evaluation Group exposes new products to a wide range of hardware and soft ware configurations. This exposure is achieved by combining a stable, well-equipped, and versatile laboratory with specialized rest software and test procedures. These tests a re done before system shipment to customers, when modifications can be made at minimal expense. The group complements the qualification efforts of the product develop ment groups by allowing these groups to focus attention on product-specific engineering issues.
Test Strategy and Test Software
The overall VAX 6000 Model 400 test strategy involved the installation of the new processors in a wide range of system configurations. The group subjected these configurations to a series of inter active load experiments, which used i nternally designed and developed software tools. These tools concurrently stressed aU configured processors, memory arrays, 110 adapters, and communication devices. Stress in this context means heavy loads in terms of 1/0 bandwidth used for a given data bus, and minimal idle CPU time for processors. This test strategy has proven successful since it was first used in the mid- 1 970s for PDP- I I systems.
The processor and memory tests exercise VAX instructions, the floating point processor, and the cache subsystem. Configurations with large mem ories are tested with a specially developed memory exerciser. This program references memory in the least efficient manner to force high page fault and low cache hit rates.
1/0 adapter tests exercise specific devices. The rests include a system exerciser that can generate various !10 rates to disk, tape, and terminal devices. while verifying data integrity.
Communication tests i nclude a DECnet exerciser and an Ethernet local area network program. These tests generate a high level of network activity and check data integrity.
Configuration Selection and Test
The System Evaluation Group selects configurations according to many factors, including
•
VMS operating system restrictions
•
Bus architectures and slot placement limitations
80
Vol. 2 No. 2, Spring 1990 Digital Tecbnical]ournal
Test and Qualification of the VAX
6000 Model 400 System
•
Power and packaging restrictions
•
Marketing requirements
The maximum number of each supported option is tested within laboratory resource limits. Because the Model 400 CPU is a higher performance proces sor for the existing
VAX
6000 platform, it was tested within established VAX 6000 family configuration guidelines. The System Evaluation Group chose nine Model 400 system configurations to test from one up to six processors, 32 to 256 megabytes of memory, and from two to six
VAXBI channels.
An important part of configuration testing for a new processor involves verification of proper sys tem initialization and operating system boot using various load paths. With the Model 400 system, this testing meant loading the operating system through different VAXBI channels, disk or CI adapters, and load devices. Several problems were noted with bus adapter initialization and self-test while testing certain VAX B I channel configurations. These configuration-dependent problems were corrected by modifying console and operating system software.
As shown i n Figure 1 , VAXB I channels on the
Model 400 were also configured with all currently supported VAX 6000 system
1/0 adapters. Although testing with large and diverse system configurations poses logistical challenges, these envirorunents will often succeed in exposing device compatibility problems.
The device compatibility problems found during the Model 400 evaluation occurred either during operating system initialization or when interactive workloads concurrently exercised all system devices. One such problem resulted from the increased processing speed of the Model 400 pro cessor and would occur only when an adapter was tested in a specific configuration under a certain workload.
More specifically, this problem was due to a race condition between a
VMS application level program issuing
I/O requests and the adapter hardware pro-
MODEL
400
PROCESSORS
1 TO 6 PER SYSTEM
MEMORY ARRAYS
1
TO
8
PER SYSTEM
(32 TO 256MB)
X M I
I
I
CONSOLE
LOAD
DEVICE
1 PER SYSTEM
Cl
ADAPTERS
1
PER SYSTEM
2 TO 6 VAXBI C H A N N E LS
I
VAXBI
I
PARALLEL
PORT
ADAPTERS
4 PER VAXBI
8
PER SYSTEM
I
ASYNCHRONOUS
COM M U N ICATION
ADAPTERS
2 PER VAXBI
3 PER SYSTEM
DEC net
COM M U NICATION
ADAPTERS
2 PER VAXBI
4 PER SYSTEM
�
LAVC/LAN VA Xcluster
DISK
ADAPTERS
2 PER VAXBI
12 PER SYSTEM
TAPE
ADAPTERS
2 PER VAXBI
4 PER SYSTEM
I
U N I B U S
ADAPTER
1 P E R VAXBI
1
P E R SYSTEM
SYNCHRONOUS
COM M U N ICATION
ADAPTERS
2 PER VAXBI
2 PER SYSTEM
Figure I Adapters/Options Tested during VAX 6000 Model 400 Evaluation
Digital Tecbnical]ournal
Vol. 2 No.
2, Spt"ing
1990
8 1
VAX 6000 Model 400 System cessing and returning a response. With slower machines, the adapter hardware won the race; but with the faster Model 400 processor, the host won hy issuing commands faster than the adapter hard ware could process them. The problem was cor rected with a modification to the device driver software.
Complete configuration test coverage for the
Model 400 processor also required VAXcluster/local area VAXcluster (LAVC) and DECnet/local area net work (LA�) evaluation using each of the supported
CI and DEC net communication adapters. To accom plish thest: tests, a 1 3-node VAXcl uster was estab lished composed of t:ight VAX host systems, four
HSC mass storage servers, and the VAX 6000 Model
400 system under test.
Primarily, the VAXcluster/LAVC and OECnet/LAN testing verified the functional compatibility of the
Model 400 processor with the VAX 6000 series C I and NI adapters. Cluster and LAN activity were used simply as adapter loads. Interactive experiments were designed to emphasize stress at the local sys tem level . Cluster-level verification was deferred to another group.
Evaluation of the VAX 6000 Model 400 system also verified the Model 400 system-level power-fail/ warm-restart capability in large configurations with high compute and 1/0 loads. These tests ensured that battery backup units would maintain supply voltages for the guaranteed duration. Further, these tests ensured that error reporting and recovery pro cedures operated properly.
Interactive Test Method
The interactive test method first selects test soft ware. It then modifies parameters such as VMS queue 1/0 (QIO) request size, number of outstand ing conunands, and device mode of operation.
Using this method, three system workloads were generated.
The first workload used small VMS QIO request s izes and maximum QIO request queue lengths to achieve high l/0 rates. This workload minimized idle CPU time and maximized time on the inter rupt stack.
The second workload used large VMS QIO request sizes w ith sequential disk accesses and device loop back to generate high bus utilization rates (bytes per second). This workload saturates 1/0 buses and interconnects by generating large amounts of direct memory access activity.
The third workload combined CPU/memory and
I/O adapter test software w ith a d istribution of small , moderate, and large VMS QJO request sizes.
Memory arrays were also configured in a non interleaved mode to degrade memory access time and aggravate potential bus timeout conditions.
The System Evaluation Group ran interactive test experiments for a minimum of four hours and a maximum of three days. A typical experiment lasted 18 hours. Most long duration experiments were performed using the combined CPU/memory and l/0 adapter workload . The longer run-times provided adequate time for adapter exercisers to step through a preprogrammed range of trans fer sizes.
Following each test, VMS error counters, test soft ware status reports, and VMS error log entries were examined for system or device errors. The group then characterized problems in terms of their fre quency, repeatability, and the system environment in which they occurred. The system environment incl uded detai led information regarding hardware configuration, the software test tools and parame ters used, and module, operating system, device driver, and firmware versions.
Although some of the problems noted during the
Model 400 evaluation were easy to reproduce and occurred frequently, others were intermittent in nature and not so easily induced. For example, the configuration-dependent problems occurred each time a specific configuration was tested, whereas a particular device compatibility problem was inter mittent and required long duration runs before appearing.
The detailed problem descriptions, along with pertinent error log or crash dump data, were pro vided to the appropriate development groups for analysis. Also, descriptions were logged in the pro ject-specific problem reporting databases to ensure that problems were properly tracked and resolved.
The System Evaluation Group then worked with the development group to fix the problem.
From a total of six problems noted during the
Model 400 evaluation, four were configuration dependent and corrected through modifications to the VMS operating system or console microcode.
Two problems were device compatibility bugs that occurred during operating system initialization and interactive testing. These problems were fiXed through modifications to the VMS operating system or device driver software.
When the VAX 6000 Model 400 system evaluation was completed, a final report was distributed. The report summarized the configurations and test tools used, the specific experiments performed, and the current status of all problems that were identified.
82 Vol. 2
No. 2,
Spring /<)<)0 Digital Tecbnical]ounwl
Test and Qualification of the VAX 6000 Model 400 System
Summary
System qualification is the last stage in the system development process. The qualification process for the Model 400 system was designed specifically to take into account computer-aided design and simu lation used by the hardware design process.
As our experience with the Model 400 system has shown, nearly all of the problems found during system qualification were in areas of the system that could not be simulated. As a result, the qualification process is most effective when focusing on testing those parts of the system that cannot be simulated because of their complexity, which presents both a chaUenge and an opportunity.
The challenge is to design test processes and tools that can adequately test a complex system in a reasonable time. With SITP and other test tools, such as those used by Midrange System Evaluation
Engineering, we have made a significant start i n developing these tests. However, there i s still much room for improvement, and work is continuing in this area.
The opportunity is to shorten the qualifica tion process. Because first-pass hardware is more robust, more system testing can occur earlier. Also, better test tools enable us to provide more test coverage in less time and with fewer resources, both in prototypes and number of people. We took advantage of this opportunity in the Model 400 system qualification to cut the length of field test by 25 percenr, thereby bringing the Model 400 system's new technology to market faster.
References
1 . B. Allison, "An Overview of the VAX 6200 Family of Sys terns," Digital Technical journal, vol. 1 , no. 7 (August 1988): 10-18.
2. B. Allison, "The Architectural Definition Process of the VAX 6200 Family," Digital Technical journal, vol . 1 , no. 7 (August 1988): 1 9-27.
3. R . Gillett, " Interfacing a VAX Microprocessor to a High-speed Multiprocessing Bus," Digital
Technical journal, vol . 1 , no. 7 (August 1988):
28-46.
4. P. SuUivan et al. , "The VAX 6000 Model 400
Scalar Processor Modu le," Digital Technical journal,
27-35 . vol. 2 , no. 2 (Spring 1990, this issue):
5 . D. Slater et a l . , "Vector Processing on the VAX
6000 Model 400 System , " Digital Technical journal,
1 1 -26. vol . 2 , no. 2 (Spring 1990, this issue):
6. H . Durdan et al., "An Overview of the VAX 6000
Model 400 Chip Set," Digital Technical journal, vol. 2, no. 2 (Spring 1990, this issue): 36-5 1 .
7 . ] . Basmaji et al . , "The Role of Computer-aided
Engineering in the Design of the VAX 6200
System," Digital Technical journal, vol. 1 , no. 7
(August 1988): 47-56.
8. T. Leonard, ed. ,
VAX
Architecture Reference
Manual (Bedford : Digital Press, Order No.
EY-3459E-DP, 1987).
Digital Tecbnicaljournal Vol. 2 No. 2, Spring
1990 83
Thomas
C.
Furlong
Michael].
K.
Nielsen
Neil
C.
Wilhelm
Development of the
DECstation 3100
The DECstation 3100 is the jlrst member of Digital's family of high-perfonnance
ULTRIX workstations.
Built with the R2000 chip set from .HIPS Computer Syste�m�
Inc., and highly integrated
110 and graphics subs_ystems, the Dtestation 3100 i m ple ments 12 mips of RISC-based computing, workstation
110, and e.ycellent bit-map graphics on a single module. The DECstation 3100 workstation runs Digital's ULTRIX opi-�ating system (compatible with UNIX software) as well as
DEC windows software,
TCP!IP, DECnet software, and Network File Services
(NF!:J).
The workstation can be configured with 8MB to 24;HB of parity-protected memory, monochrome or 8-plane color graphics, 15-ilu'h or 19-inch monitors, and SCSI disk and tape devices. This paper describes the DECstation 3100 product, the design effort, details of the s_ystem, and measured benchmark performance.
System Overview
Packaged in a desktop system box, the DECstation
3 100 workstation is implemented as a single mod ule that contains CPU/I'PU, separate instruction and data caches, memory control logic, Ethernet and small computer systems int<.:rconnect (SCSI) con trollers. four serial lines, and video display logic.
Connectors on the module accept as many as 12 memory modules of 2 megabytes (MB) apiece as well as a single monochrome or color frame buffer module. The box optionally contains 3.5-inch,
104 MB SCSI disk driv<.:s. An SCSI connector on the back of the system box supports the attachment of additional SCSI devices, such as the 332MB disk drive, the 95 MB tape drive, and the 600M B CDROM disk drive.
Background and Project Goals
Digital 's research and developmen t groups in
Palo Alto had used a UNIX operating system in a client/server computing env ironmen t for more than three years. The clients were various VAX workstations. The servers were VA X systems and
RISC-technology research machines, called Titans, which were developed by Digital's Western
Research Laboratory.
The major frustration in this environment was the lack of proct'ssing power at the workstation.
Computer room servers dt'livered up w 1
2 million instructions per second (mips), but most office workstations delivered only
I mips. This disparity caused many workstations to be used only as rather expensive tt'rminal emulators for the larger machines. The slow workstations also meant that window applications were often bogged down in screen update activity, and that NFS performance was limited not by device or network speeds but by the workstation's processing power.
The primary goal of the DECstation
)
100 project team was £O produce a fast RISC lJLTRIX workstation that would bring processing and windowing perfor nunce to the user's desk at a competitive price. The product would run the ULTRIX operating system,
DECwindows products, and network software for both TCP/IP and DEC net networks.
In early May of 1988, we received approval tO build an ULTRIX workstation that featured increased processing power. The remaining design goals were time to market, packaging, reuse of existing designs, and system cost and price. The aggressive schedule called for the first workstation to ship in mid-January 1989. We were asked to use the new desktop system hox designed for product to be later announced as the VAXstation
3 100. We were also asked to reuse any hardware or software elements of the VAXstation 3 100 that we possibly could. Our own view of the market place caused us to choose an entry-level price of approximately $ 10 K .
At the end of the project, we achieved all of our goa ls. We shipped the first system in the desired box on the very day we had promised. We reused most
84
Vol.
2 No. 2, Sprin[i fl)')O Digital Technical journal
Development of the DECstation 3100 of the existing ULTRI X , windowing, and network software, and supported the same internal disk drives as the VA Xstation :1 100. We held the entry level price within 20 percent of the S lO K goa L
System Cost Issues
We needed to control the high-cost items of proces sor, caches. and memory . We also had to resist the tendency to add things, either because other groups requested them or because we had ourselves wanted them.
The processor choice was based more on cost per mips than the absolute cost of the processor chip set i tself. The cost per mips consideration led us to select a RISC processor instead of a VAX processor in order to obtain at least a two-to-one performance advantage. Since we were building a product for users of the LJLTRIX operating system, the lack of
VAX instruction compatibility was not an issue. We chose the R2000 chip set from M I PS Computer
Systems as the best CMOS HISC technology on the market.
Even though caches contribute significantly to system performance, we still considered the use of small caches to reduce cost. Simulations of system designs indicated that the higher expense of large caches was necessary to achieve fast desktop performance.
Our most difficult challenge was determining how to implement memory . We did not want to burden the entry-level systems with more memory than was necessary, but we did want the opportu nity to add memory to systems that could use i t . We considered memory on the system module, mem ory on daughter cards, memory on commodity single in-l ine memory modules (SIMMs), and finally memory on custom-designed SIMMs. In the end, we chose the custom-designed SIMMs because of their density, cost, and configurability.
Basic Project Rules
To succeed at this project we kept the size of the team to the minimum and isolated the team from outside influences. We settled on a minimal product focus team and a minimal design team. The five person product focus team would manage the pro ject while the three-person hardware design team would build the machine. Everyone would work from the Palo Alto base.
The first machine was running two months after the project start dare. 13y that rime the design team had expanded to about twenty people developing the electronics package, diagnostics, software, and documentation. Many designers, particularly software engineers, temporarily relo cated from New Hampshire to part icipate in the project. Researchers from Digital's two Palo Alto research laboratories gave generously of their time in reviewing the design, developing new software, and testing prototype systems.
The product focus ream built extended support groups for functions such as manufacturing, mar keting, sales, and appl ication development .
Basic Design Rules
From the beginning we agreed upon some basic design guidelines. We strictly adhered to these deci sions throughout the design phase of the project.
We would develop all functions on one system module. Anything that did nor fit would not be part of the product.
We would do no ASIC or other IC design; the schedule did not allow for it. So we would use lots of random, low-cost, standard logic functions, PALs in particular. Opportunities for lower cost integra tion would be saved for follow-on products.
We would build dumb J/0 controllers. This deci sion eliminated the use of secondary processors, microcode, and hardware coordination of intel ligent devices. Various-sized buffers on every con troller would allow devices to run at their own speeds independent of general processor activity.
Software drivers running on the genera l CPU would interact with industry- or Digital-standard con trol.ler chips for the I/O subsystems.
We would build dumb graphics - no pipeline, no graphics processors, no rendering chips . bur simply a frame buffer configured as part of the main mem ory address space. The onl y exception was to add a color plane mask for use by the color software.
We would aim for ease of manufacturing by keeping the option choices low. The only choices to be made in manufacturing the system box were how many memory modules to insert. which frame buffer module to insert, and whether to add one or two i nternal disks. This decision kept the total manufacturing permutations down to 30. (Earlier
Digital workstations were up around 1000.)
We would try to let experienced computer users upgrade and service their own workstations. The choice of system box limited this capability since the VAXstation 3100 box was not originally designed for user access. We simplified the original box design and left much of the box empty to make the electronics more accessible.
Digital Tecbnicaljoumal
Vol. 1
No
.
.2, Spring 1990
85
VAX 6000 Model 400 System
Conflicts and Resolutions
At project starr, the design team resisted the idea of offering both monochrome and color graphics options. We bel ieved in the value of provid ing color wherever and whenever possible on workstations: customers prefer color, and it provides additional functionality. However, we were not confident we could develop both graphics options and still remain on schedule. We decided to usc a simple frame buffer along with a single, configurable video output design. We would bear the cost of an unnec essary VDAC in the monochrome system, but we only had to design one system .
Although the hardware design team was confi dent that cost/performance trade-offs were the right ones, other project members voiced their con cern . Some issues were memory size and I/0 and graphics performance. Woul d the maximum mem ory size of
24 MB he large enough for applications such as computer-aided design {CAD) and model ing� How would Ethernet and disk performance compare to VAXstation and competitors' work stations� Would the choice of frame buffer matched with a fast RISC processor deliver adequate graphics performance, particularly in color systems�
These questions were answered once the soft ware development was complete and performance measurements could be made on late prototype systems. Sec the Product Qualification section of this paper.
Another topic we debated was whether to anow peripheral devices in the box , which was clearl y dt�signed for such devices. Various combinations of disks and tapes in the box presented three prob lems: more complicated options in manufacturing, a heavy drive plate and complex cabling for the user to remove during system upgrade, and a power problem if an internal rape drive was present.
Eventually we permitted onl y the internal 3. 5-inch disk drives and simplified the drive mounting plate to be easier to remove.
Product Qualification
S i nce the entire project time was only eight months, we needed to maximize the test time of the DECstarion 3 100. The hardware was an entirely new design. The software was a port from a VAX base to a RISC base, and much of the lower level graphics software was completely new code.
To test modules and the hardware system, we sent many early prototypes to a local testing lab oratory for stress testing. While running both diag nostics and the ULTRI X operating system, we shook the systems, power-cycled them, temperature cycled them, even submitted them to rainfall due to an environmen tal fai lure. We recorded every failure and traced it back to its source. Many of these failures led to changes in diagnostics, compo nents, placement, and mechanical solutions. This early stress resting did not uncover any problems nor seen elsewhere, but with a small number of machines, it validated all problems seen in the other test situations.
To test the total system with its software, we i nven ted a quali fication team nicknamed the
" wrecking crew," a group of about twenty-five senior engineering researchers and developers.
They agreed to accept early protOtypes and to sub ject them w heavy use for a period of three months.
Their goal was to break the systems, as often and in as many ways as they could. During the wrecking period , we constantly installed the latest soft ware changes, replaced diagnostic ROMs, added hardware, and moved systems from person to per son to allow everyone to try different configura tions. Each crew member was responsible fur an exhaustive test of a subset of the ULTRIX commands and utilities.
The wrecking crew was a huge success. Team members reported 785 problems, complained mightily and usefully, ported CAD tools, window applications, and compilers in their spare time.
Th<.:ir constant demand for the performance they expected exposed many bugs that were artificially limiting performance. Best of all, the wreckers almost doubled the number of software experts knowledgeable about the workstation from an early stage and thus contributed significantly to system quality.
Processor Subsystem Details
The DECstation 3 100 CPU consists of the M I PS
R2000 integer processor, the R2010 floating point coprocessor, a nd fou r R 2020 write buffers. The chip set operates at 16.67 megahertz.
In the DECstation 3100, the R 2000 chip set runs in " Little End ian" mode. In other words, bits within bytes and words are counted from right to left , and the low-order bit is the rightmost bit in a word.
" Little Endian" mode means that the integer data format of the OECstation 3100 is identical to the i nteger data format of any VAX processor. The floating point data format is compliant with IEEE standards.
The R 2000 CPU implements the instruction set, processor registers, virtual memory, and interrupt
H6
Vol. 2 No. 2.
Spring i'J'JO
Digital Technicaljournal
Development of the DECstation 3100 system as defined by the R 2000 architecture. The
CPU maintains the direct-mapped, write-through data cache. Each cache is 64 kilobytes (KB) in capac ity with a 4 -byte l ine size. The tag and data stares of each cache are byte-parity protected, and cache parity errors transparently generate cache misses ro reload the cache from memory.
The R 20
10 floating point coprocessor i mp le ments the I EEE arithmetic functions and coproces sor registers defined by the R 2000 architecture.
The R 2020 write buffer implements a four-stage write bu ffer for the CPU . This write buffer allows the C PU to write to its write-th rough cache without stall ing the CPU as long as the write buffer is not ful l .
Graphics
Graphics on the DECstation 3 100 is implemented in a tightly integrated subsystem. Frame bu ffer mem ory is a region of memory in the processor address space - 256 KB in a monochrome system and 1 M B in a color system. Less than half of the monochrome frame bu ffer and three quarters of the color frame bu ffer are displayed on the workstation monitor.
The remaining frame buffer memory may be used for storage of graphics data structures such as fonts.
The frame bu ffer memory is not parity protected.
At boot time, the lJLTRIX operating system detects the size of frame bu ffer memory and whether the system is monochrome or color.
Because frame bu ffer memory is cacheable and addressable in the same way as the dynamic ran dom access memory (DRAM), the software is able to ach ieve extremely high performance without any special-purpose graphics hardware.
A color plane mask allows processor writes to the color frame bu ffer to a ffect only specific bits of a pixel. This design al lows modi fication of a given plane of the color frame buffer using only write cycles, which i nc reases performance significantly.
The graphics progra mmable cursor supports a
16-hy- 16 pixel, two-plane cursor. The cursor can take two forms : a 1 6-hy- 16 bit pattern or a crosshair whose l i nes may extend to the edges of the visible raster or may be clipped to a programmed region.
The cursor in a color system may have up to three colors, and the cursor in a monochrome system may have up to th ree gray-scale values.
Memory
The DECstation 3 100 supports 8MB to 24 M B of byte parity protected memory in 4 M B increments. The memory system includes both the DRAM array and a video random access memory (YRAM) frame buffer.
The video frame bu ffer has the same memory access characteristics as memory and may be cached if desired. The memory system supports byte, half word, word writes, and word read s.
The memory system control logic is optimized for m inimum memory read latency, at a sl ight cost in memory write latency. On a memory read, the
CPU incurs a five-cycle stall in the absence of mem ory refresh contention. The memory system can sustain five-cycle reads, which results in a peak read bandwidth of 13.3MB per second .
Memory writes to an empty write buffer com plete in eight cycles, but do not stall the CPU . Suc cessive memory writes complete at the rate of six cycles, and the CPU stalls whenever the write buffer is ful l . The memory system can sustai n six-cycle writes, which results in a peak write bandwidth of
1 1 . 1 MB per second.
The DRAM and VRA M arrays are implemented with SIMMs. Each DRAM array contains 2 M B of memory on a double-sided modu le. The VRAM arrays contain either 1 megabit (Mb) (monochrome) or 8Mb (color) of frame buffer memory on single sided modules.
Ethernet
The Ethernet interface on the DECstation 3 100 con sists of a CMOS controller chip and a 64 KB buffer.
The controller chip manages transmission and reception of packets through ring descriptors and packet bu ffers located in the Ethernet bu ffer. The buffer is time-multiplexed between the controller chip and the workstation CPU .
Connection to the Ethernet is by a thick-wire or
ThinWire cable. A push-button switch on the rear of the system box selects the appropriate connector.
SCSI
The DECstation 3 100 supplies a small computer system interconnect (SCSI) as the interconnect for storage peripherals. The workstation's SCSI i nter face consists of a gate array controller chip and a
1 28 K B buffer. The controller chip manages the SCSI bus through selection, DMA data transfer, and dis connect commands. The interface supports com mand d isconnect/reconnect and synchronous data transfers at 4 MB per second on the SCSI bus. The buffer is time-multiplexed between the controller chip and the workstation CPU .
An SCSI connector on the rear of the DECstation
3 100 system box allows connection of external SCSI peripheral devices. Digital offers a 332 M B disk , a
95 MB tape, and a 600 MB CDROM reader. Each of these devices is packaged with power in its own sidecar box .
Dtgitul TeclmicaljourrUll Vol. J No. 2.
Spring 199()
87
VAX 6000 Model 400 System
Table 1 Comparison of RISC System Performance
D h rystones/second
Li npack single precision
(M FLOPs)
Unpack double precision
( MFLOPs)
Stanford small i nteger bench mark
(seconds)
Digital Review's CPU 2 benchmark suites (seconds)
X L I B graphics performance rate
DECstation
3 1 00
22.7K
3.7
1 . 6
0 . 1 1 5
6.91
4.9K
Sun
4/1 1 0
1 2.8K
0.95
0.57
0. 220
1 8.99
NA
Sun
4/260
1 6 .7K
1 . 3
0.89
0 . 1 50
1 3. 7 1
0.7K
MIPS
M/1 20-5
24.7K
4.0
2 . 1
0 . 1 1 8
N A
NA
Serial Lines
Four serial line interfaces are present and are pro grammable from SO to 9600 bits per second. The serial transmittt:rs art: double buffered , and the receivers shart: a 64-entry F IFO. The workstation uses one serial line for the kt:yboard and another for the mouse. One serial line, designed for modem use, supports data-terminal-ready/data-set-ready
(DTR!DSR) control signals.
Software
The DEC:station 3 100 runs the standard software expected by w;ers of N I X operating systems as well as software that allows easy networking and windowing of VAX and 3 100 systems. Digital's
ULTRIX operat ing system is compatible with
Bt:rkeley v4 .3, AT&T System V, and is compliant with POSIX standards. OECwindows software runs on the DEC:station 3 100 in both the monochrome and color configurations and integrates searnlessly with DEC:windows running on VAX systems with
VMS operating systems. ULTRIX supports DE\.net,
TCP/IP, and NrS . Compilers include t he C,
FORTRA N , and PASCAL compilers adapted from the compilers from MIPS Computer Systems.
Performance
Table 1 l ists key performance measu res of the
OECstation 3100 workstation. For comparison pur poses, the table also lists the performance of other
R ISC systems, namely the Sun 4 / 1 10 and 4/260 and tht: MIPS M / 1 20-S. l:tble 2 lists the SCSI and network subsystem periphera ls 1/0 performance of the DE< :station 3 100.
Table 2 DECstation 3100 1/0 Performance
Subsystem
Peripheral 110 Performance
RZ55 file reads
(with read-ahead microcode) ·
RZ55 file reads
RZ55 writes
NFS file reads
(with read-ahead
RZ55 microcode)
NFS file reads
N FS file writes
I P/UDP 1 500-byte packets
I P/U DP 64-byte packets
I PfTCP end-to-e nd
930KB per second
330 KB per second
320KB per second
450KB per second
330KB per second
320KB per second
800 K B per second
1 000 packets per second
350KB per second
•
RZ55 read-ahead microcode has not yet been released by Digital Equipment Corporation.
Conclusion
One of the delights of the DECstation 3 100 project was that we were building the machine that we ourselves had wanted for a very long time. By using a smal l, focused core design team and resisting incremental additions, we achieved the aggressive time-to-market goal .
88
Vol. 2 No. 2,
Spring 1990 Digikll Tecbnicaljournal
Larry
B.
Weber
I
Compiler Optimization in RISC Systems
Compiler optimization detennines the level of RJSC system performance. The archi tectural design of compilers from MIPS Computer Systems, Inc., combined with sup port tools facilitates compiler optimization and overall system throughput. The compiler design takes advantage of small and high-speed cache memory to enhance performance. The cord tool positions the program in memory to ensure that the most frequently used memory locations never compete for the same cache locations.
Portability is crucial to compiler effectiveness. MIPS compilers implement many industry-wide extensions to the standard languages to make them compatible with other implementations.
RISC
(reduced instruction set computer) system performance embodies many components. In addi tion to the performance of individual instructions, the processor architect must consider how the compiler combines the instructions, how system vendors construct the memory system, and how the user writes programs. Of particular importance is how well the compiler optimizes programs for a given hardware architecture. In addition, program portability is essential to ease the burden of moving applications to new systems. By considering all such aspects of system performance, the processor architect can use the full potential of a RISC system.
Traditional CISC (complex instruction set com puter) processors were developed without signifi cant information on how high-level programming languages would use them. In contrast,
RISC archi tects make trade-offs between microprocessor structures and compiler complexity, with the goal being overall system performance.
The compiler is the key link between the archi tect and the system user. Therefore, it is essential that the processor architect completely understand the compiler and its capabilities.
Compilation is the process of converting high level source code w ri tten in a progr amm ing language into machine code for a target machine.
This process must consider translating the instruc tions correctly into machine language, as well as into optimal machine language. Often, the high level language masks the primitive level of the target machine by providing programming tools that do
Reprinted with the permission of f:.SD Magazine,
December
1989, Digital Design Publishing, Westborough,
MA
0158 1 . not directly correspond to the machine's features.
Compilers must also deal with programs that do not take best advantage of these features. The optimizer is the portion of the compiler that deals with perfor mance issues.
Optimization Boosts Performance
Optimization occurs at many stages within the compiler. Some optimizations are best done at the front end of the compilation process when first processing the source program. These optimi zations are called language-dependent optimiza tions because they rely on features unique to a specific programming language. Other optimiza tions, called machine-dependent optimizations, are performed late in the process because they require detailed information about the target machine and how the program actually uses that machine. Still, other optimizations are independent of both the source language and the target machine.
Compilers from
MIPS
Computer Systems consist of several independent front ends that convert individual languages into a common intermediate code called ucode. (See Figure
1 .)
MI PS
currently supports six programming languages:
ADA ,
c ,
COBOL, FORTRAN, PASCAL,
and
PLI ; ANSI C
and
C++
will be available in 1990. The common back end performs the bulk of the optimization and generates machine code.
The common back end of the compiler uses a variety of optimization tech niques that require varying amounts of information. The compiler must gather the information from the source code and analyze it. Peephole optimizations require the
Digital Tecbnical]ournal
Vol. 2 No. 2, Spring 1990 89
VAX 6000 Model 400 System
1 986
1 986
UCODE L I N K E R
F U NCTION I N - L I N E
GLOBAL OPT I M I Z E R
}
I NTERCOMPI LATION U N I T
OPT I M IZATION
GLOBAL OPT I M I ZATION
CODE G E N E RATOR LOCAL OPTIM IZATION
P I P E L I N E SCHEDULER PEEPHOLE OPTI M I Z ATION
Figure 1 Compiler Structure least a mount of information - usually only an instruction or two. Global optimization requires the most information; it must take into account control tlow (the branching and looping stmcture of a program) and data tlow (the data usage within each section of the program). lntercompilation unit optimizations represent an extreme form of global optimization that occurs between i ndependent source files. Local optimization requires an interme diate amount of information - usually data usage within a group of consecutive statements. Table I lists optimiz:.�tions shared in the MIPS compilers.
One of four optimization levels (-00, -01 , -02, or
-0:)) can invoke the MIPS compiler. The levels indicate the relative compilation speed - not the importance - of the various optimization classes that the compiler can implement. for example, a program compiled at the -00 level, which specifics no optimization, would compile faster than a pro gram compiled at the -03 level, which offers the full range of optimizations. These optimization kvels also correspond closely to rhe components i nvoked during the compilation process.
The -00 option disables the optimization nor mally performed by the code generator and assembler. The -01 option (the default) designates minimal and fast optimization. Under this option, the code generator and assembler perform local optimizations within basic blocks. Apart from tradi tional local optimizations such as local common subexpression, expression simplification, constant folding, dead code elimination, and peephole opti mizations, the code generator performs branch and label optimization, and the assembler performs architecture-dependent pipeline scheduling. Com pilation speed does not lengthen noticeably between -00 and -01 . Thus, -00 is sddom needed and is used mostly for comparison studies.
Option -02 adds the uopt phase to the compi lation stream to perform global optimization and register allocation within the full range of individ ual procedures. Compilation time might lengthen substantially because global data-tlow analysis and coloring register-al location algorithms are invoked.
Supporting optimizations across multipl e source files is the
-03 option. This option adds the uld
90
Vol. 2 No. 2. Spring 1990
Digital Tecbnicaljournal
Compiler Optimization in RISC Systems
Ta ble 1 Optim ization Methods
Peephole opti mization
Local optimization
I nstruction sched u ling
I nstruction sel ection
Instruction su bstitution
Call/return selection
Branch-to-branch optim ization
Local subexpression elimination
Constant folding
Expression simpl ification
Dead code elimination
Local register assignment
Short-c i rcuit evaluation
G lobal optimization
I nvariant code removal
Strength reduction
G lobal register assignment
G lobal su bexpression elim ination
Shrink-wrapping register saving
Linear test replacement
Loop u nro l l i ng
Tai l recursion
Copy propagation
Redundant store elimination l nterco mpilation u nit optim ization l nterprocedural register allocation
I n-line expansion of procedures phase, which comhines separate compilation units into a single file at the ucode level. Thus the option enables multimodule programs to achieve the same degree of optimization as single-module programs.
The umerge phase selectively expands procedure calls by in-line substitution. The resulting ucode object is then sent into normal back-end optimi zation and compilation stream starting w ith uopt.
The -03 option causes uopr to perform interproce dural register allocation; uopt also benefits from the complete information in the linked ucode file to perform the other global optimizations normally associated with this phase.
Local Hardware Architecture
MIPS' architectural design facilitates compiler opti mization and overall system throughput. Important to system performance is the memory hierarchy. A split cache provides independent access to both instructions and data in a single cycle. A single com bined cache would limit the processor ro obtaining only a single instruction or data item each cycle.
The compiler design takes into consideration the effect of cache memory. On average, instruction references cache miss less frequently than data ref erences ; this observation allows the compiler to prefer slightly longer instruction sequences if they avoid extra data references. The instruction cache miss rate is lower because the hardware loads multipie contiguous words into the cache on a miss, and the sequential nature of instruction execution takes advantage of this locality.
Register optimization makes the most significant architecture/compiler performance enhancement .
Because the optimizer knows that data al located i n registers can b e accessed without delay, i t places the most frequently used variables into registers. The optimizer computes the lifetime of individual data items and replaces the memory usage with a register usage. The relatively large number of registers makes it likely that the optimizer can successfu lly promote the most frequently referenced variables into a register. (There are thirty two 32-bit integer registers and sixteen 64-bit floating-point registers. )
One of the most important architectural features used to improve performance is the instruction pipeline. Although several cycles arc required to actually complete the instruction, the processor can be viewed as if each instruction takes only one cycle because a new instruction is started each cycle. The compiler is aware of several exceptions; for exam p le, load instructions require one instruction between the load and the use of the data loaded.
This load-delay slot is used by the compiler for another instruction - effectively, the cache mem ory access of the load executes in parallel with the other instruction. This technique is called instruc tion scheduling. Other examples of parallel instruc tion execution include
•
Overlapping a branch instruction with another operation
•
Testing for division by zero while the divide is in progress
•
Executing several different floating point oper ations simultaneously (The MIPS processor has separate floating point add, mu ltiply, and d ivide u nits.)
The
MIPS architecmre does not have condition codes. Although this seems unusual compared with many other machines, this design actually improves performa nce. The architecture provides branch instructions that both rest a condition and then branch. Thus , the compiler must generate only one instruction for conditional branches rather than the two instructions usually required (one to test for the condition, then another to perform the branch).
Only comparisons that compare larger (or smaller) values between two registers, or a register and inunediate value, cannot be handled this way. The
Digital Tecbnicaljournal
Vol. 2 No. 2, Spring 1990
9 1
VAX 6000 Model 400 System compiler restrucrures most comparisons ro avoid this case, rhus decreasing average test time.
Instructions in a C ISC processor often have widely varying execution times. This di fference makes it hard to determine which of several alterna tive sequences is actually fastest. Because almost all
R I SC instructions take the same time. the optimizer can select the fastest sequence with relative ease.
Program Portability
To make a system useful, programs must be ported onto the processor. The usc of U N I X operat ing systems and standardized languages such as
FORTRAN has tremendously improved program portability. However, incorrect programs may have latent bugs that are masked by a nain: compiler.
Thus, an optimizing compiler tends to expose more of these problems. A naive
FORTRAN compiler may assign all variables to memory locations, giving variables predictable initial values. An incorrect program that relics on these i nitial values will fail when an optimizing compiler assigns a variable to a fast register that tenets to have unpredictable values.
An optimizer converts the program (as written) into ont> that is kkntical except that it executes faster. To do this, tht> optimizer must make assump tions about what the programmer intended . Often the programmer tlepcnds on experience with previ ous compiler implementations rather than the rules of the language.
The
:vi i i'S compiler suite provides a number of options to allow a program to run without modifi cation in the presence of such common errors. This permits a program to be ported quickly, giving the programmer a choice as to when to correct the problem.
Traditionally, system vendors have added unique extensions to t heir language implementations.
While these may be a boon to a programmer when writing the program , they can be a bane when it comes time to port the program. The MIPS com pilers implement many industry-wide extensions to the stanclarcl languages to make them compatible with other implementations. An important set of extensions is the support of Digital's
VAX FORTRAN extensions. Another is the inclusion of
I B
M
' s PL l extensions to permi t an independent software vendor to port a 1 .8-million-line PL l progr:tm to the
M I PS architecture.
Wherever extensions arc required,
M i l' S chooses proposed extensions for similar functions that are being considered by the standards committees. An example of this last situation is the need to represent hardw:.�re
110 structures when being refer enced by an optimizing compiler. Consider an
I/0 device register that is used to define the status of the I/0 device. The optimizing compiler would see multiple references to the address without inter vening assignments. The compiler would cleverly
(bur incorrectly) optimize all references to a proces sor register. MIPS has added the key word volatile to indicate to the compiler that this variable changes in ways that the compiler cannot detect. This exten sion was recently incorporated into the current
ANSI standard for c, bur it was added to the MIPS c compiler four years ago .
Tools to Development
Fundamental to system development is a tool set that aids in compiling, debugging, performance tun ing, and system construction (bring-up). M I PS' tool set includes those tools traditionally found in
U N I X operating systems, as well as tools unique to M I PS.
The multiple front-end, common back-end con struction of the MIPS compilers provides a consis tent set of languages to the developer. Options and flags :tre the same across all languages. In fact, cc can usually be used to compile programs in the other languages. All languages share a common linkage convention that makes it easy ro write or port programs written in two or more languages.
The l " \I I X tool make provides a convenient method of program development. This tool iden tifies which source modules of a program changed and recompiles onl y those modules. In recom piling, make provides the correct compilation options. It also provides the complete set of compi lation options; for example, the debugging option can be enabled t hrough a make target to trou bleshoot a program. Later, the debugging options can be disabled, and higher levels of optimization can be specified in compiling the production version of the program. The make rules file, which accompanies the source program modules, deter mines how the modules are to be combined into the final run program. This prior determination elimi nates the need for detailed written documentation or programmer support , making it simpler for developers to exchange source programs. a
Crucial to efficient program development is source-oriented debugger. MIPS provides an extended version of dbx which supports all pro gramming languages. (ADA has a special debugger.)
The debugger provides features such as print ing variables as they change, or replaying a debug ging session to a certain point before continuing
92
Vol 2 Nn. 1. .\jJring /'J'JO Digital Tecbnica/journal
Compiler Optimization in RISC Systems the debug session. The user can view and ed it the source, as well as see each statement as i t i s executed .
The debugger also lets the user view the program in both the high-level language in which it was written and in the generated machine language.
Thus, the user can set a breakpoint on a specified statement or instruction. Full debugging facilities are available only when the debugging option i s specified. However, the compiler (by default) main tains the l ine number tables in a compressed format in the load module even when the option is not speci fied . Retention of the line number tables per mits panial debugging without the need to recom pile the program using the debugging option. Line number information is kept for each instruction, permitting the instruction scheduler to move the instruction while keeping track of the l ine that originated it.
Specification of both debugging and optimization options can create conflicts. For example, it is possi ble for a bug to appear only when a h igher level of optimization is specified . Moreover, optimizations such as register allocation can confuse the debug ger, because variables are not at the locations that it assumes. To avoid this situation, the M I PS compil ers disable any optimization that interferes with debugging when both debugging and optimization options are specified. To debug the problem that shows up only in optimized code, a special option permits both the debugger and optimization to be enabled . This technique requi res the developer to use caution when inquiring about the contents of a variable.
To tune a program for optimal performance, the developer must learn where in a program the time is spent and why. Traditional UNIX systems provide a method called pc sampling. With this technique, the system m ust interrup t a program at regular intervals (usually 60 or 1 00 times per second) and i ncrement a per-location counter. After enough sample points are taken, a pattern of execution time emerges. This method has a severe flaw because modern processors execute 10 to 20 million instruc tions per second; this means the number of samples is less than 1 in 100,000. The method requires a very long execution time ro collect a statistically mean ingful sample.
Although
MIPS provides the pc sampling tech nique, it also furnishes a more exacting method .
Pixie is a tool that takes an executable load module and prepares it for measurement by inserting a counrer i n every basic block. A fter running the instrumented program, the counters are dumped to a file for analysis by several programs. Prof dis plays tables of interesting information such as the following:
•
Number of CPU cycles for each source l ine
•
•
Number of times each function is called
Average number of cycles in each call to a function
Figure 2 shows examples of prof output Listings.
A second program, pixstats , takes the same counts and disp lays information about the program in architectural terms, such as:
•
Number of cycles used for each instruction
•
Number of unused delay slots
•
•
Number of FLOPS
A general indication of cache locality
These two programs assume a perfect memory system, that is, no effect due to cache misses.
For a complete analysis, a cache simulator is also available that uses p ixie to provide a memory address trace to the simulator to model the mem ory system. MIPS uses this technique to plan new machine designs. Each proposed change to the system requires a detailed simulation to exhibit its effect. When a MIPS designer is convinced that a balanced and optimal point has been found, imple mentation begins. Experience w ith this technique has shown an accuracy of better than 4 percent when comparing predicted performance ro actual performance.
An Optimization to Improve Cache
Perfo
rma
nce
An area seldom add ressed in compilers is the opti mization of programs in memory to improve cache hit rates. Modern microprocessor performance has been increasing faster than the supporting memory systems. Taking advantage of this higher perfor mance without introducing costly memories has required the use of small and high-speed cache memory. Cache memory contains recently used instructions and data. H ardware substitutes cache memory if the desired word is in the cache. This substitution is invisible (other than performance improvements) to the program . Each main memory location must share a cache location with other memory locations because the cache is smaller than main memory. For a cache to be effective, it must contain enough of the program or data to ensure multiple reuse of the instructions or data.
Digital Teclmical]ournal
Vol. 2 No. 2, Spring 1990
93
VAX 6000 Mode1 400 System
�1751 cy c le
� ot-----1
TOTAL NUMBER OF
PROGRAM CYCLES cycles
\ cyc les cum \ cycles
/c, li l bytes p roced u r e ( f i l e )
/ l l/'\e _________ _
9 0
82
"
"
1> lJ l l
6
>
JO
23
H eQln
2"1
L
. /t.c
nt.-d ehdt" I l x
. . . ut ext
• C) input .
e )
8 w r i t.
; ch.Hs
I
.
/ te x
� w r i t.
e
t o t.pu L . c l i n teger
( . . /t�xtoUlJJut . c) r:..::..._-= ..:.:._ _ _ ___:_o:c:_ _:..:..,: rltt: s o:�odl;;- 1 t.
r
. . l'l g .
, /telOI�O'-ltpUt. . c )
/ t . e . l nput . c: l f l t c: J n ( . . I t xt.output
. c ) o f ( . . /tex t i :>ut . c l f h b u f
( . . / f 11bu{ . c )
�" f \ \ I I I � , ( \ l 1 r1
0 . 0 0 1 0 0 . 0 0 o.oo
100.00 o.oo 100.00
0 . 00 lOO.CO
0 . 00 lOC.OO
0 . 00
100 •
.00
0 . 0 0 1 0 0 , 00
0 . 00
1 00 . 00
0 . 0 0 1 00 . 00
0 .
00 100.00
STATISTICS DESCRIBE CALLS
TO wrile_char COMPILED FROM
SOURCE FILE lexloutput.c.
�
.
. / r e s et
/ { o p e n
I /11brk. sl
.
c l
.cl ll
"
> eycl<lll
\ cu '"
\ l ' L 0 9 1 9 . 0 9
THE MOST HEAVILY USED LINE
IS
1 20.
LOCATED IN PROCEDURE write_char, COMPILED FROM
SOURCE FILE lexloutpul.c. w r i t e d�.1r (
, . /text.output .c) w r I t <!:-cho\ r I .
. /t.ext.out.put. . c)
�oln
(
. . / t. lt iC l I n put. . c) read ch<H ( . . /taxt Input. . c )
!Ti
a i n w r ! t w r i t o o
f ( J IC!ont . p l ch.-rs
( . . /te"tOi.l t pu L . c ) c h <l r ( . . /tc"tout.pu l . c ) wri t.tt
-C h ,H ( , , /t.I!:ICLOlllp�l t . C ) writ.tt
� Lnt:eqer ( . . /teiCtoutput . c l m a l n C f l d o n L p ) w r i te i n t e q e t ( , , /t.exto'.lt.pu t . c )
:!�< � I n (Hx!
eo l n
{ o n t . p )
. . /tc.ut i n p u t . C )
J:",a i n l ! l x f o n t . p )
!'YI 1 n ( f l "fonl.pl write ' t r i :-:q (. . /text.out put. . c J w r i t ech.- rs wrJte-eh,us
{
.
.
/t.ext.out.put. . c)
(
. . /t.ttxt.output. .c) w r i t e
= ch.:tr:� ( • . /t.e"toutpu t. . c ) w r ! te chars
1
. . / t e"t.out.po t . c ) m• i n ( f ! J dont. .pl
I l l
27 12
2
21369lb
1 . 85 8 4 . 88
·------���----------,
19
LINE
36
OF texlou1put.c
HAS
6
J
BYTES OF CODE, USED
1 .795.672
2
CYCLES. OR
1.21%
OF TOTAL
��.�P�RO�G=R�A�M�C�Y�C=LE�S�·--------�
J > O . .
5 7 1 050
O . J'J
..
"
"
J8
) J
1
.
28
10
00
561855
561855
481387
3 4 8 1 5 0
3 4 8000
0 . 38
0 . 38
0 . 3 3
0 . 24
0 . 23
95.
� 6
9 5 . 8 4
96..21
9 6. . 55
96.1'}
97 . 0 2
Figure 2 Examples of prof Output Listings
A valuable optimization is to position the pro gram in memory so that the most frequently used memory locations never compete for the same cache locations. MIPS has built a tool called cord that rearranges the program to improve instruction cache utilization. This tool is made possible through the existence of precise profiling tools.
To use cord, the programmer compiles the pro gram in the usual way. Pixie is used to add counters to the program for each basic block. After exe cuting the instrumented program, prof is run with an option that creates a file containing dynamic execution information. That file is given to cord, along with the original executable module.
Cord computes the density for each function
(procedure). The density is defined as the average number of cycles executed by each i nstruction in the function. Figure 3 is a n example of eight functions, their sizes, cycle counts, and density.
Cord then creates a new executable module after sorting the functions according to density. Figure 3 also shows the order of the functions in the rear ranged program.
This sort improves cache hit rate because it places the functions that use t he most cycles in memory so they do not compete for the same cache location as other frequently executing functions.
The effectiveness of sort is helped by two other features in the MIPS architecture. First, t he caches are direct-mapped to memory so that each memory location corresponds to a single cache location. Second, the MIPS operating system places virtual pages in physical memory so that adjacent virtual pages map to adjacent cache pages. As a result, the place ment of functions by cord has very predictable effects on the cache.
Figure 3 also shows the arrangement in an unexpected way. Rather than placing the densest function (A) at the beginning of memory, it is placed farthest from the start of the cache. This arrange-
>f-
(j, z
80
70
60
50
4 0
D
30
20
1 0
OK c
B
KEY
NAME
A
B c
D
E
F
G
H
S I Z E
1 2K
24K
20K
8 K
1 6K
1 2 K
1 6K
20K
64K
MEMORY ADDRESSES
CYCLES
DEN SITY
960K 80
1 680K 70
1 200K
400K
640K
360K
320K
200K
60
50
40
30
20
1 0
Figure 3 Cache Performance Improved by cord
94
Vol.
2 No. 2, Spring 1990
Digital Technical journal
Compiler Optimization in RISC Systems ment has the effect of making the densest func tion share cache locations with the function
1 28 kilobytes (twice the cache size) away (H). Cord improves performance by as much as 20 percent to
30 percent on programs exceeding the size cache. lt works solely by improving the efficiency of the instruction cache. Methods to improve data cache accesses are not available yet because of the more random nature of data accesses and difficulty in get ting accurate data reference i nformation.
An advanced architectural simulator, Sable mod els (in C) the processor, including
TLB, pipeline, register set, and system design, incorporating the cache subsystem, main memory, and
110 interface.
Developers can customize Sable for a unique system design. Sable has been used routinely at M I PS to bring up the
UNIX operating system before hard ware is available. Simulation with Sable assures that the software is reliable and performs optimally when the hardware is actually available. Sable can be used with the debugger to provide full symbolic debugging in the simulation environment. Also,
Sable can provide the same address traces that pixie provides to analyze operating system performance.
After a system has been brought up using Sable, other tools can assist in constructing the system on the real hardware. A simple debug monitor is available to work with the symbolic debugger to provide a symbolic debugging environment on the real hardware.
Trends in Compilers
Compilers are taking better advantage of the paral lelism in today's RISC processors. Evidence of this can be seen in the schedul ing of instructions to capital ize on load and branch delays and multiple floating point units. This trend will continue as it becomes feasible to build effective multiprocessor systems. In this area, compilers that will partition a problem across multiple processors - each per forming a portion of the iterations of a loop - will be seen. A major challenge wil l be to find ways to use this kind of parallelism in nonengineering prob lems. These problems tend not to be loop intensive and will require a breakthrough in compiler tech nology for automatic parallelism.
Over the past five to ten years, the programming language C has come to the forefront as a major systems language. While C offers many advantages, it requires a user to deal with fairly primitive struc tures rather than abstractions. C++ will offer much of the flexibility of C with the added capability of data abstraction.
It is expected that future compilers w ill take advantage of optimizations that red uce cache misses. These optimizations include loop inter change, which reorders the accesses to an array to improve locality of data references, and software pipelining, which takes better advantage of over lapping memory accesses and computation.
Digital Tecbnicaljow71al Vol. 2
!Vo. 2, Spt·ing 1990
95
I
Further Readings
The Digital Technicti Jouroal publishes papers that e.\jJ/ore the technologiculjimndatitms of Digitals maj(
>r pn Jducts.
Each journal focuses on at least one product area and presents a compilation ojj;apers written by
!be engineers who det,eloped the product.
The content for t be journal is selected by the journal
Advis01y Board.
Topics covered in previous issues of t he
IJ(�ital
Tecbnicaljournal are as follows:
VAX
8600
Processor
Vol.
I, No. 1, August NH5
MicroVAX I I System vbl. 1, No. 2,
J'vlarch
1')86
Networking Products
HJ/. I . . Yo.
3,
5>eptenzber Jl}86
VAX
8800
Family
Vol.
I, No. 4, Februcny N8'
VAXcluster Systems
Vol.
I, No. 5, September 1')87
Software Productivity Tools
Vol.
I, No. 6, Februmy 1988
CVAX-based Systems
HJI. 1, No. 7, A ugust /988
Storage Technology
Vol. I, No. 8, February 1989
Distributed Systems
Vol. I, No.
9, june
1989
Compound Document Architecture
Vol. 2,
No.
1, Winter
1990
See the inside front cover of this issue for subscrip tion information .
Single copies and past issues of the Digital Technical journal can be ordered from Digital Press at a cost of S 16.00 per copy.
Digital Press is Digital Equipment Corporat ion's imcrnat.ional publisher of books for computer pro fessionals. Copies of t he new titles now available from Digital Press that are listed below can he ordered by writing ro Digital Press, Department
DTJ, 12 Crosby Drive, Bedford, MA 01 730, U . S . A .
COMMON LISP: The Language
Guy Steele Jr., Second Edition. 1990
(538.95 in soft cover. 544 .95 in clothcover)
The Matrix: Computer Networks and
Conferencing Systems Worldwide
John Quarterman, 1990 (549.95)
UNIX for VMS Users
Philip Bourne, 1990 (S2B.95)
The VMS User's Guide
James Peters and Patrick Holmay, 1990 ($23 .00)
A Beginner's Guide to VAX VMS Utilities and Applications
Ronald Sawey anu Troy Stokes, 1989 ($23.00)
VMS Internals and Data Structures:
Version 5 Update X press
Ruth Goldenberg and Lawrence Kenah,
Volumes
I ,
2 , and 3, 1989 (535.00)
VAXNMS Internals and Data Structures:
Version 4.4
Lawrence Kenah , Ruth Goldenberg, and
Simon Bate, 1988 ($75.00)
Digital Guide to Software Development
Corporate l ser Publications Group of Digital
Equipment Corporation, 1990 ($27.95)
Technical Aspects of Data Conununication
John McNamarJ, Third Edition, 1988 (S42 . 00)
Infonnation Technology Standardization:
Theory, Practice, and Organizations
Carl Cargill, 1989 (S24 .95)
Computer Programming and Architecture:
The VAX
Henry Levy and Richard Eckhouse,
Second Ed ition, 1 989 ($24 .95)
ABCs of MUMPS: An Introduction for Novice and Intennediate Programmers
Richard Walters, 1989 ($24 .95)
96 \lui. 2 Nn. 2. Spring /'J'JIJ Digital Tecbnicaljournal
ISSN 0898-90 1X
Printed in U.S.A. EY-C 197E-DP/')(} Oi
0 2 30.0
RUO Copyright
1990
Digital Equipment Corporation All Rights Reserved
பதிவிறக்கு
விளம்பரம்