Verification of the First Fault-tolerant VAX System. Digital VAX 6000, VAX TPC Benchmark A, VAX DECtp, VAX DECdta, VAX ACMS, VAX Server, VAX ft, VAX 9000, VAX cluster, VAX KODA
विज्ञापन
विज्ञापन
William F, B r u c k e r t
C a r l o s Alonso
J a m e s M. Melvin
Verification of the First
The fault-tolerant character of the V a f t 3000 system required thatplans be made early in the development stages for the verification and test of the system. To ensure proper test coverage of the fault-tolerant features, engineers built fault-insertion points directly into the system hardware. During the verification process, test engi- neers used hardware and software fault insertion in directed and random test forms.
A
four-phase verification strategy
was
hardware and software was fully tested for error recovery that is transparent to applications on the system.
The
VAxft
3000 system provides transparent fault tolerance for applications that run on the system.
Because the 3000 includes fault-tolerant features, verification of the system was unlike that ordinar- ily conducted on
V M
systems. To facilitate system test, the verification strategy outlined a four-phase approach which would require hardware to be built into the system specifically for test purposes.
This paper presents a brief overview of the
VAxft
system architecture and then describes the meth- ods used to verlfy the system's fault tolerance.
V M f t 3000 Architectural Overview
The
VAXft fault-tolerant system is designed t o recover from any single point of hardware failure.
Fault tolerance is provided transparently for all applications running under the
VMS operating system. This section reviews the implementation of the system to provide background for the main discussion of the verification process.
The system comprises two duplicate systems, called zones. Each zone is a fully functional com- puter with enough elements to run an operating system. These two zones, referred to as zone
A
and zone
B,
are shown in Figure
1, which illustrates the duplication of the system components. The two independent zones are connected by duplicate cross-link cables. The cabinet of each zone also includes a battery, a power regulator, cooling fans, and an
AC
power input. Each zone's hardware has sufficient error checking to detect all single faults within that zone.
Figure
2 is a block diagram of a single zone with one
I/O adapter. Note the portions of the zone labeled dual-rail and single-rail. The dual-rail por- tions of the system have two independent sets of hardware performing the same operations.
Correct operation is verified by comparison. The fault-detection mechanism for the single-rail
1/0 modules combines checking codes and communi- cation protocols.
The system performs I/O operations by sending and receiving message packets. The packets are exchanged between the
CPU
and various servers, including disks, Ethernet, and synchronous lines.
These message packets are formed and interpreted in the dual-rail portion of the system. They are pro- tected in the single-rail portion of the machine by check codes which are generated and checked in the dual-rail portion of the machine. Corrupted packets can be retransmitted through the same or alternate paths.
In the normal mode of fault-tolerant operation, both zones execute the same instruction at the same time. The four processors (two in each zone) appear to the operating system as a single logical
CPU.
The hardware supplies the detection and recovery facilities for faults detected in the
CPU
and memory portions of the system.
A
defective
CPU
module and its memory are automatically removed from service by the hardware, and the remaining
CPU
continues processing.
Error handling for the
I/O interconnections is managed differently. The paths to and from
1 / 0 adapters are duplicated for checking purposes.
If a fault is detected, the hardware retries the operation.
If the retry is successful, the error is logged, and operation continues without software assistance.
Digital
Technical Journal
VoI.
3
No. 1 Winter 1991
79
Transaction Processing, Databases, and Fault-tolerant Systems
ZONE A ZONE B
DUAL-RAIL
MEFAORY
CONTROL
CROSSLINK
I
CROSSLINK CABLES
Figure 1 A Dual-zone VAXft System
MEMORY
INTERFACE
CROSSLINK
CABLES
TO ZONE B
E
-
>
CPU
MODULE fl
CROSSLINK
I
MODULE
INTERCONNECT
-9
80
h
110 ADAPTER
SINGLE-RAIL
~ p r t
ADAPTER
Figure
2
Single-zone Structure of a
VMJt 3000 Sjatenz
Vol.
3
No. I Wirrtrv 1991
Digital Tecbttical Journal
Verification of the First
Fault- tolerant
VRX
System
If t h e retry is unsuccessful, t h e Fault-tolerant
System Services
(FTSS) software performs e r r o r recovery.
n S S
is a layered software product that is utilized with every
VAXFt 3000
system. It provides the software necessary t o complete system e r r o r recovery. For system recovery from a failed
I/O device, an alternate path o r device is used. All recoverable faults have an associated maximum threshold value. If this threshold is exceeded,
FTSS performs appropriate device reconfiguration.
Verij3cation
of
a Fault-tolerant
VAX
System
This section entails a discussion of t h e types of sys- tem tests and the fault-insertion techniques used to ensure the correct operation of the VAXft system.
In addition, the four-phase verification strategy and the procedures involved in each phase are reviewed.
There are two types of system tests: directed and random. Directed tests, which test specific hard- ware o r software features, are used most frequently in computer system verification and follow a strict test sequence. Complex systems, however, cannot be completely verified in a directed fashion.' As a case i n point, an operating system running o n a processor has innumerable states. Directed tests verify functional operation under a particular set of conditions. They may not, however, be used to verify that s a m e functionality u n d e r all possible system conditions.
In companson, random testing allows multiple test processes to interact in a pseudo-random o r random fashion. In random testing, test coverage is increased w i t h additional run-time. Thus, o n c e t h e proper test processes are in place, t h e need t o develop additional tests in order to increase cover- age is eliminated. This type of testing also reduces the effects of t h e biases of the engineers generating the tests. While directed testing can provide only a limited level of coverage, this coverage level can b e well understood. Random testing offers a poten- tially unbounded level of coverage; however, quan- tifying this coverage is difficult if not impossible.
To achieve the proper level of verification, t h e
VAxft verification utilized a balance of directed and random testing. Directed testing was used t o achieve a certain base level of functionality, and random testing was used t o expand t h e level of coverage.
To permit testing of system fault tolerance in a practical amount of time, s o m e form of fault inser- tion is required. The reliability of components used in computer systems has been improving, and more importantly, the number of components used to implement any function has been dramatically decreasing. These factors have produced a corre- sponding reduction in system failure rates. Given t h e high reliability of today's machines, it is not practical from a verification standpoint t o verify a system by letting it run until f i ' l ures occur.
Conceptually, faults can be inserted in two ways.
First, memory locations and registers can be cor- rupted to mimic the results of gate-level faults
(software fault insertion). Second, gate-level faults may be inserted directly into t h e hardware (hard- ware fault insertion). T h e r e a r e advantages to b o t h techniques. O n e advantage of software-. implemented fault insertion is that n o embedded hardware support is required.' The advantage of hardware fault insertion, on the other hand, is that faults are m o r e representative of actual hardware failures and can reveal unanticipated side effects from a gate-level failure. To utilize hardware fault insertion, either a mechanism must b e designed into t h e system, o r an external insertion device must b e developed o n c e t h e hardware is available.
Given t h e physical feature size of t h e components used today, it is virtually impossible t o achieve ade- quate fault-insertion coverage through an external fault-insertion mechanism.
The error detection and recovery mechanism determines which fault insertion techniclue is suitable for each component. Some examples i l l u s trate this point. For t h e lockstep portion of t.he
VAXft
3000
CPUs,
s o h a r e fault insertion is not suit- able because t h e lockstep functionality prevents corruption of memory o r registers w h e n faults occur. Therefore, hardware faults cannot be mim- the software fault-insertion technique was suitable t o test the
I/O adapters since t h e system handles faults in the adapters by detecting the corruption of data. Hardware fault insertion was not suitable because t h e
I/O adapters w e r e implemented with standard components that did not support hard- ware fault insertion.
Because t h e verification strategy for t h e 3000 was considered a fundamental part of t h e system development effort, fault insertion points w e r e built directly into the system hardware. The amount of logic necessary t o implement fault insertion is relatively small. The goals of t h e fault-insertion hardware were to
fl
Eliminate any corruption of t h e environment under test that could result from fault insertion.
For example, if a certain type of system write
Digital Technical Journal h 1 .
No. I IVinrcr 1991
81
Transaction Processing, Databases, and Fault-tolerant Systems
operation is required to insert a fault, then every test case will be done on a system that is in a
"post-fault-insertion" state.
Enable the user to distribute faults randomly across the system.
Allow insertion of faults during system operation.
Enable testing of transient and solid faults.
The fault-insertion points are accessed through a separate serial interface bus isolated from the operating hardware. This separate interface ensures that the environment under test is unbiased by fault insertion.
Even with hardware support for fault insertion, only a small number of fault-insertion points can be implemented relative to the total
number
possi- ble. Where the number of fault-insertion points is small, the selection of the fault-insertion points is important
to
achieve a random distribution.
Fault-insertion points were designed into most of the custom
chips
in the
VAxft
system. When the designers were choosing the fault-insertion points, a single
bit
of a data path was considered sufficient for data path coverage. Since a significant
portion
of the chip area
is
consumed by data paths, a high level of coverage of
each
chip was achieved
with
relatively few fault-insertion points.
The remaining
fault-insertion points could then be
applied
to the control logic. Coverage of this logic was important because control logic faults result in error modes that are more unpredictable than data path failures.
The
effect
that a given fault has on the system depends on the
current
system operation and
when
in that operation the fault was inserted. In the
3000, for
example,
a
failure
of bit
3
in a data path will have
significantly
different behavior depend- ing upon
whether
the data bit was
incorrect during
the address transmission portion of a cycle or
dur-
ing the succeeding data portion. Therefore, the timing of the fault insertion was pseudo-random.
The choice of pseudo-random insertion was based on the fact that the fault-insertion hardware oper- ated asynchronously to the system under test. This meant
that
faults could be inserted at
any time,
without correlation to the activity of the system under test.
Faults may be transient or solid in nature. For design purposes, a solid fault was defined as a fail- ure that will be present on retry of an operation.
A transient fault was defined as a fault that will not be present on retry of the operation. Transient faults do not require the removal of the device that experienced the fault; solid faults do require device removal. Since the system reacts differently to tran- sient and hard faults, both types of faults had to be verified in the VAxft system. Therefore, it was required that the fault-insertion hardware be capa- ble of inserting solid or transient faults. Solid faults were inserted by continually applying the fault- insertion signal. Transient faults were inserted by applying the fault-insertion signal only until the machine detected an error.
As noted earlier, the verification strategy utilized both hardware and software fault insertion. The hardware fault-insertion mechanisms allowed faults to be inserted into any system environment, includ- ing diagnostics, exercisers, and the
VMS operating system. As such, it was used for initial verification as well as regression testing of the system. The veri- fication strategy for the VAxft 3000 system involved a multiphase effort. Each of the following four veri- fication phases built upon the previous phase:
1.
Hardware verification under simulation
2.
Hardware verification with system exerciser and fault insertion
3.
System software verification with fault insertion
4. System application verification with fault insertion
Figure
VAxft
3 shows the f ~ ~ n c t i o n a l
3000 system in relation to the verification phases. The numbered brackets to the right of the diagram correlate to the testing coverage of each layer. For example, the system software verifi- cation, phase
3, verified the
VMS system, Fault- tolerant System Services
(RSS), and the hardware platform.
The following sections briefly describe the four phases of the VAxft verification.
Hardware Verification under Simulation
Functional design verification using software simu- lation is inherently slow in a design as large as the
VAXft
3000 system. To use resources most efficiently, a verification effort must incorporate a number of different modeling levels, which means trading off detail to achieve other goals such as speed.'
VAxft 3000 simulation occurred at two levels: the module level and the system level. Module-level simulation verified the base functionality of each module. Once this verification was complete, a sys- tem-level model was produced to validate the intermodule functionality. The system-level model
Vol.
3 No. I
Winter 1991 Digital Technical
V?rzf?cation of the First Fault-tolerant
VAX
System
I
USER APPLICATION
I
I
HOST-BASED VOLUME SHADOWING
I
TEST PHASE COVERAGE
- - - - - - - - - - - -
VMS 5.4
I
FAULT-TOLERANT SYSTEM SERVICES
I
}
PHASE 3
PHASE
4
VAXFT 3000 HARDWARE
Figure
3
Functional Layers ofthe VRYft 3000 System in Relation to the Verification Phases
consisted of a full dual-rail, dual-zone system with an
I/O adapter in each zone. At the final stage, hill system testing was performed.
More than 500 directed error test cases were developed for gate-level system simulation. For each test, the test environment was set up on a fully operational system model, and then the fault was inserted. A simulation controller was developed to coordinate the system operations in the simulation environment. The simulation controller provided the following control over the testing:
Initialization of all memory elements and certain system registers to reduce test time
Setup of all memory data buffers to be used in testing
Automated test execution
Automated checking of test results
Log of test results
For each test case, the test environment was selected from the following: memory testing,
I/O register access, direct memory access
(DMA)
traf- fic, and interrupt cycles. In any given test case, any number of the previous tests could be run. These environments could be run with or without faults inserted. In addition, each environment consisted of multiple test cases. In an error handling test case, the proper system environment required for the test was set, and then the fault was inserted into the system. The logic simulator used was designed to ver@ logic design. When an illegal logic condi- tion was detected, it produced an error response.
When a fault insertion resulted in an illegal logic condition, the simulator responded by invalidat- ing the test. Because of this, a great deal of time was spent to ensure that faults were inserted in a way that would not generate illegal conditions. Each test case was considered successful only when the system error registers contained the correct data and the system had the ability to continue opera- tion after the fault.
Exerciser and Fault Insertion
After the prototypes were available, the verification effort shifted from simulation to fault insertion on the hardware. The goal was to insert faults using an exerciser that induced stressful, reproducible hard- ware activity and that allowed us to analyze and debug the fault easily.
Exerciser test cases were developed to stress the various hardware functions. The tests were designed to create maximum interrupt and data transfer activity between the
CPU
and the
I/O adapters. These functions could be tested individ- ually or simultaneously. The exerciser scheduler provided a degree of randomness such that the interaction of functions was representative of a real operating system. The fault-insertion hardware was used to achieve a random distribution of fault cases across the system.
Because it was possible to insert initial faults while specific functions were performed, a great degree of reproducibility was achieved that aided debug efforts. Once the full suite of tests worked correctly, fault insertion was performed while the system continually switched between all fi~nc- tions. This testing was more representative of actual faults in customer environments, but was less reproducible.
As previously mentioned, the hardware fault- insertion tool allowed the insertion of both tran- sient and solid failures. The
VAXft
3000 hardware recovers from transient failures and utilizes
Digital Technical Journal Vo1.3 No. I
Winter
1331
Transaction Processing, Databases, and Fault-tolerant Systems software recovery for hard failures. Since the goal
of
phase
2 testing was to verlfy the hardware, the focus was on transient fault insertion. Two criteria for each error case determined the success of the test. First and foremost, the system must continue to run and to produce correct results. Second, the error data that the system captures must be correct based on the fault that was inserted Correct error data is important because it is used to identi@ the failing component both for software recovery and for servicing.
Although the simulation environment of phase 1 was substantially slower than phase 2, it provided the designers with more information. Therefore when problems wcre discovered on the prototypcs irsed in phase 2, the failing case was transferred to the simulator for further debugging. Thc hardware verification also validated the motlels and test pro- cedures used in the simulation environment.
System Software Verification with Fault
Insertion
In parallel with hardware verification, the
VAxft
3000 system software error handling capabilities were tested. This phase represented the next higher level of testing. The goal was to verify the
V U tionality of the 3000 system as well as the software recovery mechanisms.
Digital has produced various test packages to verlfy
VAX
functionality. Since the
v&sft
3000 system incorporates a
VAX
chip set used in the
VAX
6000 series, it was possible to use several standard test packages that had been used to verify that system!
Fault-tolerant verification, however, was not addressed by any of the existing test packages.
Therefore, additional tests were developed by com- bining the existing functional test suite with the hardware fault-insertion tool and software faiilt- insertion routines. Test cases used included cache failure, clock failure, memory failure, intercon- nect failures, and disk failures. These failures were applied to the system during various system opera- tions. In addition, servicing errors were also tested by removing cables and modules while the system was running. The completion criteria for tests included the following:
Detection of the fault
Isolation of the failed hardware
Continuation of the test processes without interruption
System Application Verification with
Fault Insertion
The goals for the final phase of the
VAXft
3000 verification were to run an application with fault insertion and to demonstrate that any system fault recoverv action had no effect on the process integrity ant1 data integrity of the application. The application used in thc testing was based on the standard DebitCretlit banking benchmark and was implemented using the 1)ECintact layered product.
The bank has
10 branches, 100 tellers, and 3,600 customer accounts ( 1 0 tellers and 360 accounts per branch). Traffic on the system was simulated using terminal emulation process (VURTE) scripts representing bank teller activity. The transaction rate was initially one transaction per second
(TI'S) and was varied up to thc maximum
TI'S rate to stress the system load.
The general test process can be described as follows:
1. Started application execution. The terminal emu- lation processes emulating the bank tellers were started and continued until the system was operating at the desired
TPS rating
2. Invoked f a ~ ~ l t
A
fault was selectecl at random from
;i table of hardware and softwarc faults. The terminal emulation process subrnittetl stimuli to the npplication before, during, ant1 after fault insertion.
3. Stopped terminal emulation process. The appli- cation was run until a quiescent state was reached.
4
I-'erformetl result valiclntlon 1 he process integ- rlty and data integrity
o t
the application wcre val~dated
All the meaningful events were logged and time- stamped during the experiments. Process integrity was proved by verifying continuity of trans;iction processing through failures. The time stamps on logs allowed these two independent processes to be correlated.
The proof of data integrity consisted of using the following consistency rules for transactions:
1. The sum of the account balances is equal to the sum of the teller balances, which is equal to the sum of the branch balances.
2.
For each branch, the sum of the teller balances is equal to the branch bal;ince.
84
Vol.
. j l\'o
1
W i t i t o '
1991 Digital Technical Journal
Verificcrtion of the First Fault-tolerant VXXSysterrz
3. For each transaction processed, a new record must be added to the history file.
Application verification under fault insertion served as the final level of fault-tolerant validation.
Whereas the previous phases ensured that the vari-
011s components required for Fault tolerance oper- ated properly, the system application verification demonstrated that these components could oper- ate together to provide a fully fault-tolerant system.
The process of verifying fault tolerance requires
;I strong architectural test plan. This plan must be ware support for testing may be required. The veri- fication plan must tlelnonstrate cognizance of the capabilities ant1 limitations at each phase of the development cycle. For example, the speed of sim- irlation prohibits verification of software error recovery in a simulation environment. Also, when
;I system is implemented with
VI.SI
technology, tlie ability to physically insert faults into the system by means of an external mechanical mechanism may not be adequate to properly verify the correct system error recovery. These and other issues must be acldressed before the chips are fabricated or adequate error recovery verification may not be possible. Inadequate error recovery verification directly increases the risk of real, unrecoverable faults resulting in system outages.
The verification plan for the
VAXft
3000 system consistetl of the following phases ant1 objectives:
I. Hardware simulation with fault insertion verified error detection, hardware recovery, and error data capture.
2. System exerciser with h i ~ l t the coverage of the hardware siniulat ion cffort.
3.
System software with fault insertion verified software error recovery and rcporting
4.
System software verification with fault inser- tion verified the transparency of the system error recovery to the application running on tlie system.
The test of any fault tolerant system is to survive a
real
fault while running a customer application.
Removing a module from a machine may be an impressive test, but machines fail as a result of modules falling out of the backplane. The initial test of the
VAXft
3000
system showed that the sys- tem would survive most of the faults introduced.
Tests also revealed problems that would have resultetl in system outages i f left uncorrected.
System enhancements were made in the areas of system recovery actions and repair call out.
Whereas some of the problems were simple coding errors, others were errors in carefully reviewed and documented algorithms. Simply put, the collective wisdom of the designers was not always sufficient to reach the degree of accuracy desired for this fault-tolerant system.
As the
VAXft product family evolves, perfor- mance and functional enhancements will be avail- able. The test processes described in this paper will remain in use, so that every future release of software will be better than the previous onc.
1:he combination of hardware and software fault insertion, coupled with physical system disruption allows testing to occur at such a greatly accelerated rate, that all testing performed will be repeated for every new release.
References
1. J. Croll, L. Camilli, and
A .
Vaccaro, "Test and
Qualification of the
jiiuc
6000 Model 400 System,"
1990):
73-83
2. J. Barton, E Czeck, Z Segall, and
D.
"Fault Injection Experiments Using FLAT (Fault
Injection-based Automated Testing,"
IEEE
Trans- actions on Computers,
vol.
39,
no.
4
(April
1990).
3.
R. Calcagni and
W.
Sherwood,
"VAX
6000
ivlotlel
400
cru
Chip Set Functional Design Verification:'
Digital Technical Journal,
vol. 2, no. 2 (Spring
1990): 64-72.
Digilal Techrticnl
Journal
Vol
.$
iVo
1
IVirr1c.r 1991
85
डाउनलोड
विज्ञापन