ACAS PROGRAMME ACASA PROJECT Work Package 1 Final

ACAS PROGRAMME ACASA PROJECT Work Package 1 Final
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
____________________________________________________________________________________________
ACAS PROGRAMME
ACASA PROJECT
Work Package 1
Final Report on
Studies on the Safety of ACAS II in Europe
ACAS/ACASA/02-014
Edition
Edition Date
Status
Class
ACASA/WP-1.8/210D
Version 1.3
:
:
:
:
1
March 2002
Released Issue
EATMP
March 2002
Page i
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
DOCUMENT IDENTIFICATION SHEET
DOCUMENT DESCRIPTION
Document Title
ACAS PROGRAMME ACASA PROJECT
Work Package 1
Final Report on
Studies on the Safety of ACAS in Europe
EWP DELIVERABLE REFERENCE NUMBER:
PROGRAMME REFERENCE INDEX:
1.3
VERSION:
March 2002
EDITION DATE:
Abstract
ACAS II is an airborne avionics system designed to reduce the risk of mid-air collision. Its carriage
by a majority of aircraft within Europe is mandatory.
ACASA is a set of studies that has investigated several areas related to ACAS II operations in
Europe.
Work Package 1 of the ACASA project investigated safety. The safety of ACAS was assessed
purely in terms of the degree to which it achieved its stated aim of preventing mid-air collisions.
ACAS
CONTACT PERSON:
Authors:
Keywords
RVSM
TCAS
Garfield Dean
TA
TEL: 33-1-6988 7587
UNIT:
RA
EEC Brétigny
Thierry Arino, Ken Carpenter, Stephan Chabert, Harry Hutchinson, Thierry Miquel,
Beatrice Raynaud, Kevin Rigotii and Eric Vallauri (CENA, QinetiQ, Sofréavia)
DOCUMENT STATUS AND TYPE
STATUS
CLASSIFICATION
Working Draft
o
General Public
o
Draft
o
EATMP
þ
Proposed Issue
o
Restricted
o
Released Issue
þ
ELECTRONIC BACKUP
INTERNAL REFERENCE NAME:
HOST SYSTEM
Microsoft Windows
ACASA/WP-1.8/210D
Version 1.3
MEDIA
SOFTWARE
Type: Hard disk
Media Identification:
March 2002
Page ii
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
DOCUMENT APPROVAL
The following table identifies all management authorities who have successively approved the present
issue of this document.
AUTHORITY
NAME AND SIGNATURE
ACASA Project
Manager
DATE
4 March 2002
Garfield Dean
ACAS Programme
Manager
9 March 2002
John Law
20 March 2002
Director Infrastructure,
ATC Systems &
Support
Guido Kerkhofs
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page iii
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
Work Package 1:
Studies on the safety of ACAS II in Europe
WP-1.8: Final Report on Full System Safety Study
Version: 1.3
March 2002
Prepared by Thierry Arino, Ken Carpenter, Stephan Chabert,
Harry Hutchinson, Thierry Miquel, Beatrice Raynaud,
Kevin Rigotti and Eric Vallauri
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page iv
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
Document control sheet
Title:
WP-1 – Studies on the safety of ACAS II in Europe
Authors:
CENA, QinetiQ, Sofréavia
Reference:
ACASA/WP-1.8/210D
Pages:
Cover & reverse + xxiv+ 63
version
date issued
comments
0.1
19 December 2001
Final draft
0.2
20 December 2001
Inclusion of extra discussion in chapter 9
Tidied up and reordered Conclusions
0.3
20 December 2001
Editorial corrections
Format improvements
Changes in Abstract and E6.3.6 for GD
0.4
21 December 2001
Changes following comments from TA
1.0
21 December 2001
Delivered issue
1.1
27 December 2001
Editorial changes requested by GD
1.2
18 January 2002
Editorial and format changes
Elimination of ‘professional risk ratio’ from Exec. Summary
Clarification of 5.3.3.1
1.3
21 February 2002
ACASA/WP-1.8/210D
Version 1.3
Editorial changes to merge in ACASA Project Report:
Released Issue
March 2002
Page v
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
Table of contents
Executive summary ................................................................................................................................... xi
List of tables .............................................................................................................................................. xxi
List of figures............................................................................................................................................. xxi
Acronyms.................................................................................................................................................. xxii
Definitions ............................................................................................................................................... xxiii
1
Introduction................................................................................................................................. 1
1.1
Objectives .......................................................................................................................... 1
1.2
Background and context .................................................................................................. 1
1.3
Previous ACAS II safety studies ..................................................................................... 2
1.4
The need for further studies, for Europe ....................................................................... 4
1.5
ACASA project ................................................................................................................. 5
1.6
Scope of the document...................................................................................................... 5
1.7
Structure of the remainder of the report ....................................................................... 6
2
Summary of WP-1 ...................................................................................................................... 7
2.1
Introduction to WP-1 ....................................................................................................... 7
2.2
The WP-1 work packages ................................................................................................ 8
2.2.1
WP-1.1: Development of tools required to calculate ‘logic risk ratio’ for Europe...... 8
2.2.2
WP-1.2: Performance limitations of small aircraft ...................................................... 8
2.2.3
WP-1.3: Effect of surveillance performance on the ACAS II logic............................... 8
2.2.4
WP-1.4: Quality of the ACAS II surveillance performance .......................................... 9
2.2.5
WP-1.5: Multiple encounters ........................................................................................ 9
2.2.6
WP-1.6: ACAS II collision risk ‘event tree’ .................................................................. 9
2.2.7
WP-1.7: Human reaction to ACAS II alerts and displays............................................. 9
2.2.8
WP-1.8: The final report ............................................................................................. 10
2.3
The work carried out...................................................................................................... 10
2.3.1
WP-1.1: Development of tools required to calculate ‘logic risk ratio’ for Europe.... 10
2.3.2
WP-1.2: Performance limitations of small aircraft .................................................... 11
2.3.3
WP-1.3: Effect of surveillance performance on the ACAS II logic............................. 12
2.3.4
WP-1.4: Quality of the ACAS II surveillance performance ........................................ 12
2.3.5
WP-1.5: Multiple encounters ...................................................................................... 12
2.3.6
WP-1.6: ACAS II collision risk ‘event tree’ ................................................................ 12
2.3.7
WP-1.7: Human reaction to ACAS II alerts and displays........................................... 12
2.4
Results concerning the ICAO standards ...................................................................... 13
3
Tools and methods .................................................................................................................... 14
3.1
Risk ratio ......................................................................................................................... 14
3.2
Risk ratio for the airspace, or for the aircraft............................................................. 14
3.3
Estimating the performance of the logic ...................................................................... 15
3.4
A European ACAS safety encounter model ................................................................. 16
3.5
Data for the European ACAS safety encounter model ............................................... 16
3.6
NMAC rates .................................................................................................................... 17
3.7
ACAS II surveillance...................................................................................................... 18
3.8
Pilot response .................................................................................................................. 19
3.9
Aircraft performance and type ..................................................................................... 19
3.10
Estimating the performance of the whole system........................................................ 20
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page vi
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
4
Performance of ACAS II in European airspace .....................................................................22
4.1
The performance of the ACAS II logic .........................................................................22
4.1.1
Scenarios......................................................................................................................22
4.1.2
ACAS – unequipped .....................................................................................................22
4.1.3
ACAS – ACAS ..............................................................................................................22
4.2
ACAS II induced risk......................................................................................................23
4.2.1
Introduction .................................................................................................................23
4.2.2
ACAS – unequipped .....................................................................................................23
4.2.3
ACAS – ACAS ..............................................................................................................23
4.2.4
Discussion....................................................................................................................23
4.3
25ft altitude data .............................................................................................................24
4.3.1
Altitude reporting format .............................................................................................24
4.3.2
ACAS – unequipped .....................................................................................................24
4.3.3
ACAS – ACAS ..............................................................................................................24
4.3.4
Conclusion ...................................................................................................................24
4.4
The total effect of ACAS II on the risk of collision......................................................25
4.4.1
Risk ratios ....................................................................................................................25
4.4.2
Discussion....................................................................................................................25
4.5
Factors affecting the efficacy of ACAS II.....................................................................26
4.5.1
Factors investigated.....................................................................................................26
4.5.2
Human factors..............................................................................................................26
4.5.3
Equipment upgrade directly benefiting ACAS equipped aircraft ................................27
4.5.4
Equipment upgrades indirectly benefiting ACAS equipped aircraft ...........................27
5
The effect on ACAS safety of imperfect surveillance ............................................................29
Introduction.....................................................................................................................29
Tools and methods ..........................................................................................................29
5.2.1
Monte Carlo approach.................................................................................................29
5.2.2
Risk ratio comparison..................................................................................................30
5.2.3
Simulation hypotheses..................................................................................................30
5.2.4
Surveillance error model .............................................................................................30
5.2.5
Scenarios......................................................................................................................31
5.3
Robustness of the ACAS II logic to imperfect surveillance ........................................32
5.3.1
General ........................................................................................................................32
5.3.2
Influence of missing surveillance data ........................................................................32
5.3.3
Influence of large errors in the range measurements..................................................32
5.3.4
Influence of the global surveillance error model ........................................................33
5.4
Conclusion........................................................................................................................33
5.1
5.2
6
The effect on ACAS safety of imperfect pilot response .........................................................34
6.1
Methods and data sources ..............................................................................................34
6.2
Actual pilot behaviour ....................................................................................................34
6.2.1
Preventive RAs .............................................................................................................34
6.2.2
Corrective RAs without pilot reactions........................................................................34
6.2.3
Pilot reactions to RAs ..................................................................................................35
6.2.4
Specification of the pilot models based on the airborne data .....................................36
6.2.5
Conclusion ...................................................................................................................38
6.3
The effect of non-standard behaviour on the logic ......................................................39
6.3.1
ACAS II simulations.....................................................................................................39
6.3.2
Conclusion ...................................................................................................................40
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page vii
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
6.4
6.4.1
6.4.2
6.4.3
7
Total system performance with real pilots................................................................... 41
Pilot behaviour............................................................................................................ 41
Risk ratios.................................................................................................................... 41
Conclusion................................................................................................................... 42
The benefits of Phase 2 of the ACAS mandate....................................................................... 43
Introduction .................................................................................................................... 43
Methods and data sources.............................................................................................. 43
7.2.1
Aircraft performance classes ...................................................................................... 43
7.2.2
Scenarios ..................................................................................................................... 44
7.2.3
Aircraft class distribution ........................................................................................... 45
7.3
Value to logic risk ratio of the extension of the ACAS II mandate ........................... 46
7.3.1
Ground point of view................................................................................................... 46
7.3.2
Airborne point of view................................................................................................. 47
7.4
Consideration of the whole system ............................................................................... 48
7.4.1
Introduction ................................................................................................................. 48
7.4.2
Levels of ACAS equipage ............................................................................................ 48
7.4.3
Ground point of view................................................................................................... 49
7.4.4
Airborne point of view................................................................................................. 49
7.5
Conclusions ..................................................................................................................... 50
7.6
Recommendations........................................................................................................... 50
7.1
7.2
8
8.1
8.2
8.3
8.4
8.5
Multiple encounters .................................................................................................................. 51
Introduction .................................................................................................................... 51
Methods and data sources.............................................................................................. 51
The frequency of multiple encounters .......................................................................... 51
The efficacy of the ACAS II logic in multiple encounters .......................................... 52
Multiple encounters and the safety of operations with ACAS II............................... 52
9.1
9.2
9.3
Discussion................................................................................................................................... 54
The initial list of questions............................................................................................. 54
Tools and methods .......................................................................................................... 55
Other issues ..................................................................................................................... 56
9
10
Conclusions ................................................................................................................................ 57
10.1
Background, tools and methods .................................................................................... 57
10.2
Performance of ACAS in European airspace .............................................................. 57
10.3
Phase 2 and extending ACAS equipage........................................................................ 58
10.4
Pilot response to ACAS .................................................................................................. 58
10.5
The effect on ACAS safety of imperfect surveillance ................................................. 58
10.6
Multiple encounters........................................................................................................ 59
10.7
Other factors influencing the safety of ACAS ............................................................. 59
11
Recommendations ..................................................................................................................... 60
11.1
General ............................................................................................................................ 60
11.2
Further work................................................................................................................... 60
11.3
Observation ..................................................................................................................... 60
12
References.................................................................................................................................. 61
13
Annexes ………………………………………………………………………………………. 65
ANNEX A : Comments on the ICAO encounter model
ANNEX B :Event probabilities
ANNEX C : ACASA WP-1 working papers
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page viii
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
Abstract
Context
ACAS II is an airborne avionics system designed to reduce the risk of mid-air collision. Its carriage by a
majority of aircraft within Europe is mandatory.
ACASA is a set of studies that has investigated several areas related to ACAS II operations in Europe.
Work Package 1 of the ACASA project investigated safety. The safety of ACAS was assessed purely in
terms of the degree to which it achieved its stated aim of preventing mid-air collisions.
Tools
A variety of tools and techniques have been used in the work. Notably two new tools have been
developed. These facilitated the assessment of the performance of the ACAS algorithms in isolation and
also their performance within the total ATM system. These new tools are applicable to studies going
beyond the work reported here.
Conclusions
ACAS works: the deployment of ACAS reduces the risk of collision by a factor of about three in
European airspace.
Every aircraft that equips with ACAS acquires a benefit in terms of a reduced risk of collision. The
extension of the mandatory carriage to more aircraft will directly benefit those aircraft that are required
to equip and will reduce further the risk of a collision within European airspace.
Pilots do not always respond to the alerts that ACAS generates. They should: pilots who ignore alerts are
almost twice as likely to be involved in a mid-air collision than pilots who do not.
When pilots do respond to the alerts that ACAS generates they do not always respond accurately. They
should: a pilot who always responds to alerts slowly could reduce the risk of collision they face by more
than a factor of two-and-a-half by responding to the alerts accurately.
ACAS can induce collisions that would not have occurred had it not been deployed. Although ACAS
reduces the overall risk of collision, induced collisions account for a significant fraction of the risk of
collision that remains. This level of induced risk is an issue requiring serious consideration.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page ix
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
Editorial notes
Blank pages
Pages that are blank, but for the header and the footer, are deliberately so.
Gender
The convention that all pilots are female and all controllers are male is adopted throughout this
document. This helps to clarify text that refers to both pilots and controllers, and to ensure a genderneutral text without resorting to ugly constructs such as he/she.
References
References to other work are made in two styles: [nn]; and [WP-1 nnn]. References in the style [nn] are
listed in chapter 12, under the title ‘References’. References in the style [WP-1 nnn] are listed in
Appendix C under the title ‘ACASA WP-1 Working Papers’.
Appendix C is a list of the significant papers written during WP-1.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page x
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
Executive summary
E1
Introduction
E1.1
Background
E1.1.1
The objective of the work reported here was to investigate a variety of issues relating to the
safety of ACAS II, the Airborne Collision Avoidance System. The work is in support of the
mandates for the carriage of ACAS II in Europe. More precisely, it concerns the effect that
ACAS II can be expected to have on the risk of collision, under a variety of circumstances,
including the value of extending the carriage of ACAS II.
E1.1.2
In the context of this work, the term ‘safety’ is equivalent to the term ‘efficacy’: the extent to
which ACAS achieves its purpose of reducing the risk of mid-air collision. Many other factors
that must be addressed before any system can be considered airworthy (e.g. compatibility with
other avionics systems, electrical safety) are not within the scope of the studies reported here.
E1.1.3
It is a feature of ACAS II, which was well known prior to this work and which these studies
have reinforced, that its performance is very sensitive to the characteristics of the airspace in
which it is deployed. In particular the results of a study carried out in one airspace cannot be
assumed to be applicable to an airspace with different characteristics.
E1.1.4
The development of ACAS has generally been spurred by the occurrence of actual mid-air
collisions. A notable exception was the decision in 1995 to mandate ACAS II within the ECAC
States, which was taken after due consideration of the merits of the system.
E1.2
European ACAS mandate
E1.2.1
The European ACAS II mandate is being introduced in two phases.
· Phase 1 – from 1st January 2000, all civil fixed-wing turbine-engined aircraft having a
maximum take-off mass exceeding 15,000kg or a maximum approved seating configuration
of more than 30 were required to be equipped with ACAS II.
· Phase 2 – from 1st January 2005, all civil fixed-wing turbine-engined aircraft having a
maximum take-off mass exceeding 5700kg or a maximum approved seating configuration
of more than 19 will be required to be equipped with ACAS II.
E1.2.2
Aircraft covered by Phase 1 are referred to as ‘large aircraft’ in this report. The further
population of aircraft covered by Phase 2 are referred to as ‘small aircraft’. Other civil aircraft
are referred to as ‘light aircraft’.
E1.3
Previous ACAS II Safety Studies
E1.3.1
The first comprehensive studies of the safety of ACAS II were carried out by MITRE in 1983
and 1985. They were based on a systematic analysis of all the factors and events that could
result in collision when ACAS II is fitted. Although conservative, these studies were
theoretical, and the results suggested a significantly better performance than would have been
achieved by ACAS II as designed at that time.
E1.3.2
The first examination of the effect of ACAS II on a very large number of real encounters,
captured from saturation radar recordings, was by the UK NATS during 1987–1989. The
results, obtained simply by modelling the operation of the logic, were very different to those
reported by MITRE. Results from this work were available for SICASP/4 and, despite the
limitations of the approach, resulted in a considerable revision of the earlier estimates. From
that time, most subsequent studies concentrated on the performance of the logic.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page xi
Studies on the safety of ACAS II in Europe
Executive Summary
E1.3.3
SICASP/5 reported to ICAO on the safety of ACAS II in 1993. They concluded that, in ideal
circumstances, the use of ACAS II could reduce the risk of collision to 11% of that without
ACAS II. They also reported on practical steps that could be taken to improve the chances that
ACAS would achieve this ideal performance.
E1.3.4
In 1997, SICASP/7 standardised the ACAS logic by specifying performance measures and the
values to be achieved for those measures. They also specified the method of calculation, which
involved using a standardised stochastic encounter model to generate very many encounters, in
which the operation of the ACAS logic can be simulated. This approach has been used in
ACASA.
E1.3.5
Even though the key issue seemed to be the operation of the logic, NATS commissioned a
study into the total effect of ACAS II on the risk of collision in 1995, carried out by EDS. This
study was the first attempt to consider the whole ACAS II system since the early MITRE
studies. The conclusion was that the risk of collision would be reduced to about one third of
the risk without ACAS II, a less favourable conclusion than the results for the logic alone. It
seemed that considering the effect of the traffic display had not led to improved performance,
because pilots often fail to take the avoiding action recommended by ACAS II and this is the
larger factor.
E1.3.6
The EDS study was limited to encounters between ACAS and unequipped aircraft. The event
tree approach has been developed as part of ACASA, to encompass ACAS-ACAS encounters,
developing a rather leaner and simpler tool in the process.
E1.4
Objectives
E1.4.1
The principal objective was to support the European mandate of ACAS II with quantitative
evidence of the benefits of equipping with ACAS II.
E1.4.2
Specific evidence was sought to support Phase 2.
E1.4.3
The evidence is also to support the continuing enforcement of Phase 1.
E1.4.4
A secondary objective was the development of tools that, in addition to providing the
quantitative evidence, could be readily used to study the interaction of ACAS with future ATM
changes in Europe.
E1.4.5
Finally, several questions relating to effect of imperfect surveillance on the collision avoidance
logic, the effect of slow (or actual) pilot response and the importance of multiple encounters
seemed to require resolution.
E2
Summary of Work Package 1
E2.1
The WP-1 work packages
E2.1.1
WP-1.1: Development of tools required to calculate the ‘logic risk ratio’ for Europe. QinetiQ
implemented the ICAO standard ACAS encounter model in order to validate the
implementation by CENA and Sofréavia. The ICAO standard ACAS encounter model was
developed into a European ACAS safety encounter model, culminating in the calculation of a
logic risk ratio specific to European airspace.
E2.1.2
WP-1.2: Performance limitations of small aircraft. WP-1.2 was to quantify the benefits of
Phase 2 of the ACAS mandate, the extension to small aircraft. A potential factor limiting these
benefits was felt to be the limited performance (i.e. climb rates) of small aircraft. WP-1.2
assessed these limitations, and evaluated the benefits to the small aircraft of Phase 2. The
benefit of Phase 2 to large aircraft, already equipped with ACAS II, and to light aircraft was
also evaluated.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page xii
Studies on the safety of ACAS II in Europe
Executive Summary
E2.1.3
WP-1.3: Effect of surveillance performance on the ACAS II logic. A model of ACAS II
surveillance was developed so that the sensitivity of ACAS II performance to imperfect
surveillance could be estimated.
E2.1.4
WP-1.4: Quality of the ACAS II surveillance performance. It had been hoped to use data from
ACAS II flight recorders to study actual ACAS II surveillance performance. In practice the
quantity and quality of data was limited, and this task was subsumed into WP-1.3.
E2.1.5
WP-1.5: Multiple encounters. The significance of multiple encounters to the overall safety of
ACAS II was studied. The frequency of multiple encounters was estimated and the
performance of ACAS II in those encounters assessed, as far as this was possible.
E2.1.6
WP-1.6: ACAS II collision risk ‘event tree’. An event tree was developed that enabled an
estimation of the overall value of ACAS II, taking into account all quantitatively significant
factors.
E2.1.7
WP-1.7: Human reaction to ACAS II alerts and displays. A review of the manner in which
pilots respond to ACAS II advice to take avoiding action was conducted. Flight recorder data
were examined to determine actual pilot responses and the effect of the alternative pilot
responses on the performance of the logic evaluated.
E2.1.8
WP-1.8: The final report.
E3
Tools and methods
E3.1
General
E3.1.1
ACASA developed many techniques and tools during Work Package 1 but two are of particular
interest. The European ACAS safety encounter model and the event tree are powerful tools,
which can be adapted for work that goes beyond the ACASA project.
E3.2
European ACAS safety encounter model
E3.2.1
Sofréavia and CENA had jointly implemented the ICAO standard encounter model before the
ACASA project began. QinetiQ implemented the ICAO model as part of the ACASA project.
Successive versions of each implementation were iteratively used to calculate the ACAS risk
ratio, with corrections being made on both sides until acceptable agreement was reached.
Comments on the ICAO standard encounter model arising from this experience were reported
back to SICASP.
E3.2.2
The ICAO encounter model is standardised solely to evaluate the ACAS logic and represents
no particular airspace. In order to provide specific advice on the operation of ACAS in
European airspace, the standard model was developed into a European ACAS safety encounter
model. The European model is richer than the ICAO model (many more characteristics of
encounters and their interdependencies are considered) and is tuned to European airspace (the
probabilities of various encounter characteristics have been determined from an analysis of
encounters collected during 1998 and 2000 from European radar data).
E3.3
Event tree
E3.3.1
The logic risk ratio, which is calculated using the encounter model, is an important
performance indicator for ACAS. However, the operation of the logic is just one of many
factors that determine the efficacy of ACAS in operational use. These additional factors can be
considered by using an ‘event tree’: a logical diagram that combines the relevant factors to
calculate a risk of collision for the whole system.
E3.3.2
Such an event tree was developed and used in the ACASA project. It was realised as an Excel
spreadsheet and probabilities for the base level events were estimated. The tree was then
exercised to produce various estimates of the risk of collision in European airspace.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page xiii
Studies on the safety of ACAS II in Europe
Executive Summary
E3.3.3
The event tree, and the system risk ratios calculated using it, are not substitutes for the
encounter model and logic risk ratio. There are many questions for which an event tree is not
suitable, and the logic risk ratio calculations are essential for the correct formation of the tree.
The logic risk ratio is more accurate in its own terms than the event tree is in its terms. For the
logic risk ratio one makes many assumptions and then derives an answer in which one can be
confident, but which is subject to those assumptions. For the event tree, a best judgement is
made for everything that matters, and the result is no better than the ability to judge.
E4
Performance of ACAS II in European airspace
E4.1
Logic risk ratios
E4.1.1
The logic risk ratios reported here are computed assuming that all other aspects of the system
operate as intended: the surveillance of intruders is perfect, and pilots react to all resolution
advisories (RAs) and with an ideal response. Altimeter error is an inherent feature of aircraft
navigation and is taken into account in the calculations.
E4.1.2
The calculations have been performed using two different, but equally valid, implementations.
The results have been combined to produce the values reported in E4.2. In specific tasks, an
initial value might be quoted that is based on only one implementation. Additionally, the logic
risk ratios are valid for a precise set of scenario assumptions, which will vary according to
context. Thus, for example, the values quoted in E4.2.2 and E5.2.1 differ, both because the
scenario assumptions differ, and because the results reported in E5.2 have employed only one
of the implementations.
E4.1.3
The logic risk ratio was calculated for two cases: the case where each encounter is between an
ACAS equipped aircraft and an unequipped aircraft; and the case where each encounter is
between two ACAS equipped aircraft (with full co-ordination of RAs). The former can be
thought of as the risk ratio faced by the first aircraft ever to equip with ACAS, when no others
have ACAS. The latter can be thought of as the logic risk ratio corresponding to a universal
mandate in which every aircraft is ACAS equipped.
E4.2
ACAS II logic performance
E4.2.1
Whilst ACAS resolves the majority of mid-air collisions, it also has the potential to induce
mid-air collisions that would not have otherwise occurred. This induced risk is a significant
proportion of the risk of collision that will remain when ACAS is fully deployed.
E4.2.2
The risk ratio in encounters between an ACAS equipped aircraft and an unequipped intruder is
22.9%, of which 13.7% is due to induced risk.
E4.2.3
The risk ratio in co-ordinated encounters between two ACAS equipped aircraft is 3.3%, of
which 2.2% is due to induced risk.
E4.3
Total system risk ratios
E4.3.1
The logic risk ratios quoted above (and others calculated for various non-standard pilot
responses) were combined with the probabilities of other system events, using the event tree, to
obtain risk ratios relevant to the operation of the total ACAS system.
E4.3.2
The risk ratios can be presented from two different perspectives:
· An overall airspace perspective – the reduction in the risk of collision within the airspace
resulting from ACAS equipage. This measure is termed here the ‘airspace risk ratio’.
· An individual aircraft perspective – the reduction in the risk of collision to an individual
aircraft that equips with ACAS, or some other nominated aircraft, which gets some measure
of protection, by proxy, because other aircraft are ACAS equipped.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page xiv
Studies on the safety of ACAS II in Europe
Executive Summary
E4.4
ACAS II total system performance
E4.4.1
62.9% of aircraft operating in European airspace were assumed to be ACAS equipped as a
consequence of Phase 1. With this level of equipage, the use of ACAS was found to reduce the
risk of collision in the airspace to 29.9% of the risk that would exist in the absence of ACAS.
The risk of an induced collision is 4.5% of the risk that would exist in the absence of ACAS.
E4.4.2
For Phase 2 an ACAS equipage level of 70.1% was assumed. The airspace risk ratio was found
to be 24.2%, of which the induced risk constituted 4.6%.
E4.4.3
The risk ratio1 for an aircraft that equips was found to be 26.7% before Phase 1, 27.8% after
Phase 1 but before Phase 2, and 27.2% after Phase 2. The advantage of fitting ACAS is
remarkably constant when expressed in these terms.
E4.4.4
At any time, regardless of the level of ACAS equipage by other aircraft, the risk of
collision for a specific aircraft can be reduced by a factor greater than three by fitting
ACAS.
E4.5
Use of 25ft format altitude data
E4.5.1
Mode S transponders are able to report altitude in increments of 25ft (vice altitude reported in
increments of 100ft by Mode C transponders). The ACAS mandated in Europe is able to
exploit data reported in this more precise format.
E4.5.2
The use of altitude data reported in the 25ft-format improves the performance of the logic and
as such is to be encouraged. However, the improvement is not so great as to warrant a mandate
for that reason alone.
E5
The effect on ACAS safety of imperfect surveillance
E5.1
Surveillance error model
E5.1.1
A surveillance error model was developed in order to estimate the effect of imperfect
surveillance upon the operation of the ACAS logic.
E5.1.2
The model assumed that, on each one second cycle, ACAS either received a reply from the
intruder or it did not (i.e. neither range nor altitude available).
E5.1.3
When a reply was received it was assumed that:
· the range measurement was correct or the range measurement was wrong; and
· the altitude was available or the altitude was not available.
E5.1.4
In each case there was a degree of correlation with state on the previous cycle: there was a
tendency for the state to persist.
E5.1.5
In this particular study, it was assumed that both aircraft were ACAS equipped (and therefore
Mode S equipped) and that pilots responded correctly to ACAS RAs.
E5.2
Imperfect surveillance – results and conclusions
E5.2.1
For the basic scenario, perfect surveillance, the logic risk ratio was 2.3%. Imperfect
surveillance had the following effect:
· altitude data not available on average 11% of the time – a risk ratio of 2.4% resulted.
· range data in error by up to 350ft 11% of the time – a risk ratio of 2.4% resulted.
1
In this case, the ratio of the risk of collision after fitting ACAS to the risk of collision, for that
aircraft, immediately before it fits with ACAS.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page xv
Studies on the safety of ACAS II in Europe
Executive Summary
· replies not available (i.e. no range and no altitude) 22% of the time – a risk ratio of 2.9%
resulted.
· finally the previous three effects were considered simultaneously – a risk ratio of 2.9%
resulted.
E5.2.2
The scenarios considered above are believed to be extreme – they effectively represent the
worst cases of imperfect surveillance. This considered, the results indicate the ACAS logic to
be robust.
E5.2.3
It is apparent that the most significant factor in imperfect surveillance is the failure to get
replies.
E5.2.4
This study was limited to an optimistic situation in which ACAS performs well, as described in
E5.1.5. The opposite extremes, where only one aircraft is equipped and pilots do not respond
correctly, have not been studied.
E5.2.5
The effect of imperfect surveillance on the operation of the ACAS logic has not been
considered when using the event tree and calculating the total system performance of ACAS,
but the possibility that an intruder is not tracked at all is included. In principle, imperfect
surveillance should be included in the event tree, but to do so requires the additional studies
indicated in E5.2.4.
E6
The effect on ACAS safety of pilot behaviour
E6.1
Actual pilot response and imperfect pilot response models
E6.1.1
Data from on-board recorders were collected and examined to determine the actual response of
pilots to operational RAs. A total of 61 RAs were collected.
E6.1.2
In 56 cases there was a corrective RA. The pilots reacted to these corrective RAs in only 31 of
these cases.
E6.1.3
When pilots did react to corrective RAs, an analysis of the reactions indicated that none of the
pilot responses were close to the standard response. Actual pilot responses fell into two distinct
groups:
· ‘aggressive response’ in which pilots achieved a vertical rate in excess of that required by
the RA; and
· ‘slow response’ in which the delay before a response was initiated was longer than
standard, the acceleration was lower than standard, and the vertical rate attained was less
than that required by the RA.
E6.1.4
These two observed responses were modelled in ACAS simulations to assess their effect on the
efficacy of ACAS.
E6.1.5
These data are now somewhat dated. More recent studies suggest that pilots are now more
likely to respond to an RA, and this later evidence has been taken into account in the event
tree. However, the manner of response, when pilots do react, seems (regrettably) to be much
the same.
E6.2
Effect of imperfect pilot response on the ACAS II logic performance
E6.2.1
The logic risk ratios were calculated for ACAS equipage corresponding to Phase 2. It was
assumed that 80% of the aircraft not equipped with ACAS reported altitude in 25ft increments.
E6.2.2
When the pilots of all ACAS aircraft respond with the standard response the risk ratio is found
to be 9.0% (of which 3.7% is induced).
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page xvi
Studies on the safety of ACAS II in Europe
Executive Summary
E6.2.3
When the pilots of all ACAS aircraft respond with the aggressive response the risk ratio is
reduced to 6.3% (of which 1.1% is induced). Although a lower risk ratio results from the
aggressive response, this does not imply that such a response from pilots is desirable: the
disruption to ATC resulting from larger deviations is discussed at greater length in the body of
this report.
E6.2.4
When the pilots of all ACAS aircraft respond with the slow response the risk ratio increases to
63.3% (of which 33.7% is induced). This increase in the risk of collision, by a factor of ten,
indicates the importance of an accurate response from pilots if the full benefits of ACAS
equipage are to be achieved.
E6.2.5
These logic risk ratio results relate to the manner of the pilot’s response. They do not concern
the significant probability that the pilot will fail to act at all on RAs.
E6.3
Effect of imperfect pilot response on the ACAS II total system performance
E6.3.1
The possibility of imperfect pilot response to RAs has been taken into account when evaluating
the full system performance: typically a proportion of pilots do not respond to RAs, and a
proportion of those pilots that do respond respond slowly or even with the wrong sense. The
results in this section indicate the consequences for an individual pilot of various pilot
responses: both the reduced risk that she can attain by responding better than her peers; and the
increased risk that she will face should her response be worse than that of typical pilots. The
precise values reported here are relevant to the current level of ACAS equipage (i.e. between
Phase 1 and Phase 2).
E6.3.2
A pilot that switches ACAS on and then responds in the same way as her peers reduces her risk
of collision to 27.8% of the risk immediately before turning ACAS on – the reduction in
collision risk reported in E4.4.3.
E6.3.3
A pilot that always responds to an RA further reduces her risk of collision to 23.7% of the risk
faced by the pilots of unequipped aircraft.
E6.3.4
Better is not merely to follow RAs, always, but also to do so accurately. Should a pilot who
always responds to RAs further ensure that her response is always that required by ACAS, she
will reduce her risk of collision further still to 13.2% of that faced by the pilots of unequipped
aircraft. Such an ideal pilot faces less than half the risk of collision faced by typical pilots of
ACAS equipped aircraft.
E6.3.5
A pilot who always follows RAs but always does so slowly will face a risk of collision that is
36.8% of that faced by the pilots of unequipped aircraft. Thus, although always responding,
she does not achieve the same level of risk reduction as her peers who respond promptly some
of the time, even though they sometimes fail to respond at all. It is critical to follow RAs
accurately.
E6.3.6
A pilot who never follows ACAS RAs faces a risk of collision that is 45.8%, rather than
27.8%, of that faced by the pilots of unequipped aircraft. Thus she faces more than one-anda-half times the risk faced by typical pilots, and more than three times the risk she would
face if she always followed RAs and followed them accurately.
E6.4
Benefit of situational awareness and alerting functions of ACAS
E6.4.1
If the benefits of the alerting functions are removed (neither visual acquisition nor contact with
the controller are prompted by ACAS alerts) the airspace risk ratio worsens to 56.9%. Thus the
effect of the ACAS alerting functions is to almost halve the risk of collision that would exist
were RAs the sole function of ACAS (56.9% is reduced to 29.9%).
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page xvii
Studies on the safety of ACAS II in Europe
Executive Summary
E7
The benefits of Phase 2 of the ACAS mandate
E7.1
Phase 2
E7.1.1
Phase 1 of the ACAS mandate has resulted in the equipage of the majority of aircraft operating
in European airspace. ACASA has studied the safety improvement that will result from
Phase 2
E7.1.2
The benefit of Phase 2 can be assessed from the point of view of the small aircraft that are
directly required to be equipped. It will also benefit the large aircraft already equipped under
Phase 1 and the light aircraft not subject to the mandate.
E7.2
Phase 2 – logic modelling and results
E7.2.1
A matter of potential concern when considering small aircraft is the degree to which they
exhibit limited performance, which might restrict their ability to comply with ACAS RAs.
E7.2.2
This concern has been addressed by modelling seven aircraft performance classes. Operational
ceilings and climb capability have been reviewed for each class. It transpired that the
performance limitations of small aircraft were no more significant than those of large aircraft
from an ACAS perspective. Nevertheless, these performance limitations were modelled for
each of the classes.
E7.2.3
The logic risk ratios were calculated assuming ideal pilot response.
E7.2.4
For Phase 1, an airspace risk ratio of 21.8% when no performance limitations were modelled
became 22.0% when performance limitations were modelled.
E7.2.5
For Phase 2, an airspace risk ratio of 12.8% when no performance limitations were modelled
became 13.0% when performance limitations were modelled.
E7.2.6
When considered from an aircraft perspective, it is found that aircraft not directly affected by
Phase 2 see a moderate reduction in the aircraft risk ratio2. For large aircraft a risk ratio of
9.2% after Phase 1 decreases to 8.4% after Phase 2; for light aircraft a risk ratio of 38.1% after
Phase 1 decreases to 29.0%. Light aircraft get a greater benefit than large aircraft because large
aircraft tend to be segregated from small and light aircraft, and the use of aircraft class in the
logic calculations allows this to be modelled.
E7.2.7
The most dramatic effect is, as expected, for small aircraft which are required to equip with
ACAS by Phase 2. For these, an aircraft risk ratio of 57.4% after Phase 1 decreases to 6.1%
when they equip with ACAS in accordance with Phase 2.
E7.3
Phase 2 – total system performance results
E7.3.1
The benefits of Phase 2 have also been quantified for the whole system using the event tree.
The event tree does not take aircraft class into account. However, it can allow for different
levels of ACAS equipage at each phase of the mandate, and can therefore be used to assess the
incremental benefit of Phase 2. Furthermore, large, small and light aircraft can be identified by
whether or not they equip in Phase 1 or Phase 2.
E7.3.2
From an airspace perspective Phase 2 is found to reduce the risk ratio from 29.9% to 24.2% of
the risk of collision in the absence of ACAS.
E7.3.3
Again it is found that aircraft not directly affected by Phase 2 see a moderate reduction in their
aircraft risk ratio. Small aircraft that equip in compliance with Phase 2 see a larger reduction in
the aircraft risk ratio from 54.8% after Phase 1 to 13.6%.
2
For the aircraft, the ratio of the risk of collision following Phase 1 (first figure) or Phase 2
(second figure) to the risk of collision before Phase 1. The aircraft itself might not be ACAS
equipped.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page xviii
Studies on the safety of ACAS II in Europe
Executive Summary
E7.3.4
The effect of Phase 2 will be to reduce the risk of collision by a factor of four for the
aircraft required to fit ACAS.
E8
Multiple encounters
E8.1
Multiple encounters – data sources and methods
E8.1.1
The performance of ACAS can differ when faced with two or more simultaneous threats. Such
situations are referred to as multiple encounters and the ACAS logic includes specific
algorithms to handle such cases. These algorithms underwent extensive revision between
version 6.04A and version 7 of TCAS.
E8.1.2
The importance of multiple encounters to the overall benefit of ACAS was investigated using
two approaches. In each approach the performance of version 6.04A and version 7 was
evaluated.
E8.1.3
In one approach, actual multiple encounters reported during the operational evaluation of
ACAS II were collected. The performance of ACAS in these encounters was simulated and its
effectiveness judged by expert staff. A total of eleven events were reported, but the simulations
revealed that in only two would the multiple-aircraft logic be invoked.
E8.1.4
The other approach examined a large body of radar data (collected over six months and
corresponding to 3.4´105 flying-hours) which was objectively searched for close encounters in
which the multiple-aircraft logic might be invoked. Seven encounters were found, but in only
two would there have been an RA. These encounters were then modified (by reducing the
horizontal miss distance), so that they became encounters in which there existed a risk of
collision. The effect of ACAS was then simulated in these modified encounters and the
performance evaluated in terms of the vertical miss distances both with and without ACAS.
E8.2
Frequency of multiple encounters
E8.2.1
The data imply that truly multiple encounters (i.e. encounters in which the ACAS multipleaircraft logic is invoked) are rare events, occurring perhaps once in every 1.7´105 flying-hours.
E8.2.2
Multiple encounters in which the horizontal miss distance is small and there is a risk that
ACAS will induce a near mid-air collision (NMAC) are even less frequent.
E8.3
Efficacy of the ACAS II logic in multiple encounters
E8.3.1
Version 7 of the TCAS logic is generally found to be more effective than version 6.04A in
multiple encounters. However, much of this better performance is a consequence of features of
the version 7 logic that are not specific to multiple encounters, i.e. version 7 is generally found
to give fewer RAs, RAs of shorter duration and RAs that, when followed, result in smaller
deviations.
E8.3.2
The simulations of modified encounters suggest that the mean time between NMACs induced
by ACAS in multiple encounters exceeds 6.8´106 flying-hours. However this result must be
considered only an indicative lower bound. The precise value cannot be trusted due to the
paucity of data upon which it is based; it arises from a single encounter and particular
treatment that involves extremely conservative assumptions.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page xix
Studies on the safety of ACAS II in Europe
Executive Summary
E9
Conclusions and future work
E9.1
Conclusions
E9.1.1
ACAS has been demonstrated to reduce the risk of mid-air collision.
E9.1.2
The extension of the mandate to small aircraft will reduce this risk still further, bringing
particular benefits to those extra aircraft that equip.
E9.1.3
The response of pilots to the RAs that ACAS generates is of crucial importance in achieving
the full potential benefit of the mandate.
· A pilot who does better than most by always responding and responding accurately faces
half the risk of collision of her peers.
· Conversely, a pilot who fails to respond to RAs is a danger to others as well as her own
aircraft.
E9.1.4
The use of altitude data reported in the 25ft-format improves the performance of the logic and
as such is to be encouraged. However, the improvement is not so great as to warrant a mandate
for that reason alone.
E9.1.5
The ACAS logic has been demonstrated to be robust when surveillance is imperfect for the
ideal case of co-ordinated encounters in which pilots respond as expected.
E9.1.6
Limitations on aircraft performance are not generally an issue with regard to the performance
of ACAS.
E9.1.7
Multiple encounters are rare events. However, there is insufficient data, at this stage, to
confirm that their significance in terms of the overall risk of collision with ACAS is negligible.
E9.2
Future work
E9.2.1
The work package has developed and used two important and powerful tools, viz. the European
ACAS safety encounter model and the ACAS safety event tree. These merit further
development and it is recommended that they be used in other work investigating the
interaction of ACAS and ATM practices.
E9.2.2
The evaluation of the principal performance measure of ACAS, viz. airspace risk ratio, is found
to be very sensitive to the assumed NMAC rate in the airspace under consideration. It would
therefore be worthwhile expending some effort to achieve a better estimate of this fundamental
quantity.
E9.2.3
A knowledge of the precise response of pilots to the RAs that ACAS generates is a crucial
factor in estimating the efficacy of the full system. It is worth monitoring the nature of pilot
response in order to obtain a better estimate of the current performance of ACAS, and also to
ensure that improved training and greater familiarity with the system leads to more accurate
pilot response as expected.
E9.2.4
Further studies could usefully be conducted to confirm that the performance of the ACAS logic
remains robust when considering imperfect surveillance even in encounters in which the pilot
response is not ideal or in which the intruder is not ACAS equipped. In addition, the effect of
Gilham encoding errors in altitude reports could also be considered.
E9.2.5
More extensive data needs to be gathered, and further simulations performed, if the anticipated
efficacy of ACAS in multiple encounters and the assumption that they are not significant in
terms of the overall performance of ACAS are to be demonstrated.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page xx
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
List of tables
Table 3.1
Various expressions of the assumed collision risk without ACAS......................................... 18
Table 4.1
System risk ratios .................................................................................................................... 25
Table 5.1
Risk ratios vs. scenarios .......................................................................................................... 32
Table 6.1
Number of RAs and of pilot reactions .................................................................................... 34
Table 6.2
Statistical results on the pilot manoeuvres in reaction to the RAs.......................................... 35
Table 6.3
Standard pilot reaction to RAs ................................................................................................ 36
Table 6.4
Pilot models............................................................................................................................. 39
Table 6.5
Risk ratio vs. pilot models....................................................................................................... 40
Table 6.6
Risk ratios vs. pilot behaviour................................................................................................. 42
Table 7.1
Vertical rate limitations per layer and per aircraft performance class,
weighted mean values ............................................................................................................. 44
Table 7.2
Risk ratio vs. scenarios............................................................................................................ 46
List of Figures
Figure 5.1
Monte Carlo simulations ......................................................................................................... 29
Figure 5.2
Simplified surveillance error model........................................................................................ 31
Figure 6.1
Delay (s) vs. vertical speed (fpm) ........................................................................................... 37
Figure 6.2
Vertical acceleration (g) vs. vertical speed (fpm) ................................................................... 38
Figure 7.1
Aircraft class distribution........................................................................................................ 45
Figure 7.2
Aircraft risk ratio..................................................................................................................... 47
ACASA/WP-1.8/210D
Version 1.3
Page xxi
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
Acronyms
ACAS
Airborne Collision Avoidance System
ACASA
ACAS Analysis
ATC
Air Traffic Control
ATS
Air Traffic Services
CENA
Centre d’Études de la Navigation Aerienne
DERA
Defence Evaluation and Research Agency, succeeded by QinetiQ on 1st July 2001
ECAC
European Civil Aviation Conference
EDS
Electronic Data Systems Corporation
EEC
EUROCONTROL Experimental Centre
FAA
USA Federal Aviation Administration
FL
Flight Level
FLnnn
The Flight Level at nnn00ft
fpm
feet per minute: a unit of vertical speed
hmd
horizontal miss distance
ICAO
International Civil Aviation Organization
JAA
Joint Aviation Authorities
MASPS
Minimum Aviation System Performance Standards
MITRE
The correctly spelt name of a not-for-profit research and development corporation
MOPS
Minimum Operational Performance Standards
NATS
National Air Traffic Services Ltd
NM
International Nautical Mile
NMAC
Near Mid-Air Collision
ppt
percentage points – see ‘Definitions’
QinetiQ
The correctly spelt name of a private limited company
RA
Resolution Advisory
RTCA
The correctly spelt name of a not-for-profit corporation
RVSM
Reduced Vertical Separation Minimum
SARPs
Standards and Recommended Practices – an ICAO document
SCRSP
Surveillance and Conflict Resolution Systems Panel
SICASP
SSR Improvements and Collision Avoidance Systems Panel
Sofréavia
Societé Francaise d’Études et Realisations d’Equipements Aeronautiques
SSR
Secondary Surveillance Radar
TA
Traffic Advisory
TCAS
Traffic alert and Collision Avoidance System, an ACAS
vmd
vertical miss distance
WP-1
Work Package 1 of ACASA
ACASA/WP-1.8/210D
Version 1.3
Page xxii
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
Definitions
ACAS II
An ACAS that generates vertical, but not horizontal, avoidance manoeuvre advice.
aircraft risk ratio
The risk of collision for a particular aircraft, or class of aircraft, for a given ACAS equipage
scenario divided by the risk of collision for the same aircraft before any aircraft were
equipped with ACAS. (The aircraft itself may not be equipped, or may be specified by the
fact that it is the only aircraft that is equipped.)
airspace risk ratio
The risk of collision in an airspace for a given ACAS equipage scenario divided by the risk
of collision in that airspace in the absence of ACAS.
crossing
An encounter is crossing when the aircraft cross in altitude in the course of the encounter.
encounter
Usually a pair of aircraft trajectories that are, in some sense that is defined by the context,
close.
Flight Level
Barometric altitude, measured in units of 100ft, referenced to a standard pressure of
1013.25hPa.
Gilham encoding
The convention used to encode Mode C altitude reports for broadcast via a transponder.
Each possible Mode C altitude is encoded into a unique sequence of ‘bits’ in such a way
that a change from one value to the next numerical value is always achieved by changing a
single bit.
horizontal miss distance
The horizontal separation at the time of closest approach between the two aircraft in an
encounter.
large aircraft
In this report, a ‘large aircraft’ is an aircraft covered by Phase 1 of the European ACAS II
mandate, i.e. one having a maximum take-off mass exceeding 15,000kg or a maximum
approved seating configuration of more than 30.
light aircraft
In this report, all aircraft except ‘large aircraft’, ‘small aircraft’ and military fighters.
logic risk ratio
The proportional reduction in the risk of collision calculated for stated assumptions
concerning the performance of ACAS. Typical assumptions are that certain aircraft are
equipped, that all aircraft report altitude and a certain proportion do so in 25ft increments
(as opposed to 100ft increments), that ACAS surveillance is perfect and that pilots response
exactly as intended. Logic risk ratio is a measure of the performance of the collision
avoidance logic.
Mode A
A mode of secondary surveillance radar replies that reports a four-digit aircraft identifier
allocated to the aircraft by ATC.
Mode C
A mode of secondary surveillance radar replies that reports altitude in whole units of 100ft.
Mode S
A mode of secondary surveillance radar that enables interrogators to address individual
aircraft using their unique aircraft addresses. The replies contain aircraft address, altitude
and other information. Aircraft fitted with ACAS II must be fitted with a Mode S
transponder. Mode S transponders can report altitude either in whole units of 25ft or in
whole units of 100ft.
ACASA/WP-1.8/210D
Version 1.3
Page xxiii
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
Near Mid-Air Collision
An encounter in which horizontal separation is less than 500ft and vertical separation less
than 100ft simultaneously. In this report, it is generally taken to be an encounter in which
hmd < 500ft and vmd < 100ft (i.e. at closest approach).
percentage points
The numerical difference between two percentages. In this report, the abbreviation ppt is
sometimes preferred to % when discussing a statistic that is already expressed as a
percentage. The purpose is to make it clear that the quantity is an absolute change in the
value of the statistic, rather than a proportional change. Thus, if the statistic has a value of
X%, a reduction of Yppt results in a value of (X – Y)%
progressional risk ratio
A calculation made for specific aircraft; the risk of collision after a specific equipment
change (e.g., but not always, equipage with ACAS) divided by the risk of collision
immediately before that equipment change. For example, it is possible to calculate the
progressional risk ratio for an ACAS equipped aircraft arising because all aircraft, as
opposed to just most aircraft, report altitude.
Resolution Advisory
An alert issued by ACAS II advising the pilot to take avoiding action in the vertical plane.
RAs may be ‘corrective’ (requiring the pilot to manoeuvre vertically) or ‘preventive’
(requiring no immediate manoeuvre but proscribing a range of vertical speeds).
Reduced Vertical Separation Minimum
The regime in which the standard cruising levels are separated by 1000ft between FL290 to
FL410 inclusive.
risk ratio
The ratio of the risk of collision with ACAS to the risk of collision without ACAS.
small aircraft
In this report, a ‘small aircraft’ is an aircraft that is not large, but is covered by Phase 2 of
the European ACAS II mandate, i.e. having a maximum take-off mass exceeding 5700kg or
a maximum approved seating configuration of more than 19.
system risk ratio
A calculation of risk ratio for the ACAS system as a whole, with estimates made for the
probable performance of all parts of ACAS, pilots and equipment on other aircraft.
TCAS II
The ACAS II mandated by the FAA and standardised by RTCA. In this report, the term
TCAS is used when referring to USA legislation or rules, and when referring to alternative
versions of the collision avoidance logic. Otherwise, the term ACAS is preferred.
Traffic Advisory
A preliminary alert issued by ACAS II in order to cue a visual search for the potential
threat, and to prepare the pilot for a possible Resolution Advisory.
version 6.04A
The release of the TCAS collision avoidance logic immediately preceding the latest
(version 7). There are no ACAS II installations still using earlier releases.
version 7
The current release of the TCAS collision avoidance logic. A requirement to fit ACAS II
(conforming to the ICAO SARPs) implies a requirement to fit version 7 as opposed to
version 6.04A.
vertical miss distance
The vertical separation at the time of closest approach between the two aircraft in an
encounter.
ACASA/WP-1.8/210D
Version 1.3
Page xxiv
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
1
Introduction
1.1
Objectives
1.1.1
The objective of the work reported here was to investigate a variety of issues relating to the
safety of ACAS II, the Airborne Collision Avoidance System. The work is in support of the
mandates for the carriage of ACAS II in Europe. More precisely, it concerns the effect that
ACAS II can be expected to have on the risk of collision, under a variety of circumstances,
including the value of extending the carriage of ACAS II.
1.1.2
The purpose of ACAS is to reduce the risk of mid-air collision. In this study and report, there
is no distinction between safety and efficacy. If ACAS reduces the risk of collision, it is both
safe and effective. The many safety issues that need to be considered before any equipment can
be installed on an aircraft (interactions with other systems, risk of causing injury, potential for
fire) were not the subject of this study.
1.1.3
The particular issues covered include the following.
· The reduction in collision risk that can be expected from ACAS II.
· The effect of imperfect surveillance on the efficacy of ACAS II.
· The effect of imperfect pilot response on the efficacy of ACAS II.
· The safety benefit of extending of the present mandate to small aircraft3.
· The safety of ACAS II in encounters involving more than two aircraft.
1.1.4
Some tools and capabilities required for ACAS II safety studies existed before the work
started, but these have been substantially overhauled, adapted to European airspace and
extended. These improved facilities are available for future work.
1.2
Background and context
1.2.1
The possibility of an airborne collision avoidance system based on Secondary Surveillance
Radar (SSR) was first mooted as early as 1958 [1] [2], following the mid-air collision between
two airliners over the Grand Canyon in 1956 (128 perished). However, it did not become a
viable prospect until equipage with Mode C SSR transponders was widespread. The US
Beacon Collision Avoidance System (BCAS) was developed during the late 1970s [3]. A
collision between a Boeing-727 and a private aircraft at San Diego in 1978 (150 perished)
spurred renewed efforts, and in 1981 BCAS became TCAS – Traffic alert and Collision
Avoidance System. Production in four years was promised, and, to develop international
standards for ACAS, ICAO formed SICASP, the SSR Improvements and Collision Avoidance
Systems Panel.
1.2.2
In 1986 a DC-9 collided with a private aircraft at Cerritos, California (85 perished). In 1987
the US Congress enacted Public Law 100-223, mandating the carriage of TCAS II in the USA
on all jet aircraft capable of carrying more than 30 passengers by the end of 1991. In the event,
full equipage in the USA was achieved by the end of 1993.
1.2.3
In view of the large number of international aircraft that would soon be carrying ACAS II, the
agreement of international standards seemed urgent at SICASP/4 in 1989. However, there were
two problems. Although two Limited Installation Programs, by United Airlines and NorthWest
Airlines, had demonstrated the potential and acceptability of ACAS II, it was none the less
expected to have an (unknown) effect on ATS. And, for several reasons, it had recently
become necessary to revise the collision avoidance logic. Thus SICASP/4, [4], recommended
3
The phrases ‘large aircraft’, ‘small aircraft’ and ‘light aircraft’ have precise meanings in this
report, related to the terms of the ACAS mandate. The meanings are in the Definitions.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 1
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
that the draft SARPs, current at the time, should serve as guidance (effectively, as interim
SARPs) for the purpose of an international evaluation of ACAS II while it was introduced in
the USA. However, they made firm recommendations on a number of operational and policy
matters relating to ACAS II, standards were changed as a result, and ACAS was born as an
internationally accepted concept [5].
1.2.4
The results of the operational evaluation were reported at SICASP/5 in 1993, [6], and SARPs
for ACAS II were recommended. The operational evaluation had resulted in some changes in
the collision avoidance logic, and version 6.04A of the logic was finalised in 1993 [7]. All US
operators were required by Airworthiness Directives to convert their TCAS II to version
6.04A4, and this version is still carried by many American aircraft. The SARPs were approved
in November 1995.
1.2.5
In 1995, the EUROCONTROL Committee of Management and the European Civil Aviation
Conference (ECAC) agreed a mandatory implementation schedule for the carriage and
operation of ACAS II in the airspace of the ECAC states. This was confirmed at the fifth
meeting of the Ministers for Air Transport in Europe in 1997. The EUROCONTROL ACAS
Programme acts as agents for ECAC and member States. Carriage by commercial transport
aircraft, as opposed to within ECAC airspace, is required by the JAA or by the airworthiness
authorities of the individual states, in co-ordination with the EUROCONTROL ACAS
Programme.
1.2.6
The European mandate is for version 7 of the collision avoidance logic. The MOPS were
finalised by RTCA in 1997 [8] and incorporated in SARPs at SICASP/6 in the same year.
Version 7 was modified by an FAA Technical Service Order in 1999 [9]. The European
mandate requires equipage with ACAS II in two phases.
· From 1st January 2000, all civil fixed-wing turbine-engined aircraft having a maximum
take-off mass exceeding 15,000kg or a maximum approved seating configuration of more
than 30 were required to be equipped with ACAS II.
· From 1st January 2005, all civil fixed-wing turbine-engined aircraft having a maximum
take-off mass exceeding 5700kg or a maximum approved seating configuration of more
than 19 will be required to be equipped with ACAS II.
1.2.7
The mid-air collision between a Saudi Boeing-747 and a Kazakh Ilyushin-76 at Delhi in 1996
(349 perished) initiated an ICAO proposal for a world-wide mandate. This is based on the
same rules of applicability as the European policy, and will require equipage by 2003 and 2005
for large and small aircraft respectively.
1.3
Previous ACAS II safety studies
1.3.1
The first comprehensive studies of the safety, or efficacy, of ACAS II were carried out by
MITRE in 1983 and 1985, some years before the US mandate [10] [11]. They were based on a
systematic analysis of all the factors and events that could result in collision when ACAS II is
fitted. Although, like most safety analyses, the analyses were conservative, they were
theoretical, and the results suggested a significantly better performance than would have been
achieved by ACAS II as designed at that time. The aspect of the studies that led to these
optimistic results was the absence of sufficient analysis of real encounters, which turned out to
be much more complex than was anticipated at that time. Nevertheless, these first studies
provided a framework for subsequent studies, and some of the detailed results are still used.
4
The term ‘version’, as in version 6.04A and version 7, is used for successive versions of the
TCAS collision avoidance logic. Version 7 complies with the present ACAS SARPs, and
version 6.04A does not.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 2
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
1.3.2
The first examination of the effect of ACAS II on a very large number of real encounters was
by the UK NATS, during 1987–1989. The results were reported at SICASP/4 [12] [13]. The
encounters were captured from saturation radar recordings, and enabled estimates of the
efficacy of ACAS II in real encounters, assuming that the surveillance systems were perfect
and pilots behaved as intended. Preliminary results from this work were available well before
SICASP/4, and resulted in a considerable revision of the earlier estimates [14] [15]. However,
the problem with this approach was that the real encounters studied were not sufficiently close;
that is, there was not sufficient loss of separation to create a significant risk of collision. The
ACAS II logic is designed for encounters in which there will be a collision, and it performs
better in those encounters.
1.3.3
The results that emerged at SICASP/4 focused attention on the ACAS collision avoidance
logic. It also happened that the operational evaluation launched by SICASP/4 suggested a need
for revisions to the logic, and for these reasons most subsequent studies concentrated on the
performance of the logic.
1.3.4
In the UK, DERA continued to use the encounters collected in 1987–1989 as the basis for
successive safety studies, for successive versions of the logic [16] [17] [18]. In this work, the
fallacy of using real encounters was addressed by modifying close encounters so that the
horizontal miss distance was zero, thus creating a set of fictitious encounters that could have
been regarded as a de facto model for collision risk encounters in the UK. MITRE preferred to
develop a stochastic5 model for collision risk encounters, and then to tune the model to the
characteristics of real encounters [19] [20] [21] [22] [23]. The most obvious advantage of the
MITRE approach is that it enables the generation of very large numbers of encounters,
removing the unknown effect of random variability from the results.
1.3.5
The interaction between the two approaches described above led directly to a demonstration
that ACAS II is not equally effective in all airspaces [24]. Air traffic consists of varying
mixtures of aircraft types, and is managed in different ways in different regions. This leads to
the encounters between aircraft having different characteristics in different regions. An
encounter model fit for the USA differs from an encounter model fit for Europe. In
consequence, ACAS II cannot be equally safe in all regions.
1.3.6
For ICAO, the definitive report on the safety of ACAS II was that presented in 1993 at
SICASP/5 [25]. On the basis of this report, SICASP/5 reported that the use of ACAS II would
reduce the risk of collision to 11% of that without ACAS II in ideal circumstances. This figure,
11%, was a compromise between the results obtained for various States in various ways, but all
the results were of comparable magnitude. The report went on to outline measures that are
required to achieve this ideal performance.
1.3.7
The MITRE approach of using a stochastic encounter model was eventually standardised in the
ICAO SARPs at SICASP/6 in 1997 [26]. In the meantime, CENA had also implemented what
was now considered the ICAO method. Both MITRE and CENA published the calculations for
the latest version of the ACAS II logic (i.e. version 7, the current version) later in 1997, but
CENA observed that they did not obtain identical results [27]. In both cases, version 7 was
judged safer than the previous version of the logic (version 6.04A) [28], which was the
practically important issue. However, the actual value of the safety measure had been made
part of the SARPs – so there seemed to be an issue to resolve at the start of the ACASA
project.
1.3.8
As already noted, nearly all this work concentrated on the performance of the collision
avoidance logic. This was reasonable because the original MITRE studies [10] [11] had
considered the whole system, and there was good reason to believe that the key issue was
5
A stochastic process is one characterised by a sequence of random variables, so that its
behaviour may be analysed statistically but not predicted precisely. In this work, the statistical
distributions of the random variables are specified, and thus known precisely.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 3
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
whether or not the logic was capable of producing a significant reduction in the risk of
collision. Furthermore, by the time of SICASP/5 in 1993, the operational experience of pilots
confirmed the value of the ACAS II traffic display and Traffic Advisories6. Nevertheless,
NATS commissioned a study into the total effect of ACAS II on the risk of collision, from
EDS in 1995 [29] [30].
1.3.9
Unlike the earlier MITRE work, the EDS study did not include the risks of hardware or
software failure in the calculations, because, generally7, they are negligible. A number of other
simplifications were necessary so that the study remained tractable. These included: the effects
of proximity to the ground were ignored; only one aircraft was ACAS equipped; and pilots that
responded to RAs were assumed to do so promptly or totally wrongly. In spite of these
limitations, the study was the first comprehensive attempt to calculate the overall value of
ACAS II since the early MITRE studies. The conclusion was that the risk of collision would be
reduced to about one third of the risk without ACAS II. This result was less favourable than
the simple, bald calculations of the performance of the logic, and some might have found it
surprising that considering the effect of the traffic display in improving visual acquisition had
not led to improved performance. However, pilots often fail to take the avoiding action
recommended by ACAS II (i.e. fail to follow the RAs), and the EDS calculations indicated that
this is a larger factor than improved visual acquisition.
1.4
The need for further studies, for Europe
1.4.1
Since the European mandate for ACAS II is clearly a fait accompli, it might be asked what
purpose could be served by further studies of the safety of ACAS II.
1.4.2
The most obvious answer is that the extension of the mandate to small aircraft (i.e. those
carrying 20–30 passengers, or with a maximum take-off mass between 5700kg and 15,000kg)
does not take effect until 2005. There is plainly a need to quantify the benefit of fitting
ACAS II to these aircraft, from the point of view of the operators of those aircraft and from the
point of view of the airspace regulators.
1.4.3
A further motive is to support the enforcement of the mandate with quantitative evidence.
Particular operators, or aircraft types, might plead their special cases, in which case it would be
useful to evaluate the special points made. Alternatively, there might be particular airspace
regimes, or operational procedures, for which it is important to know the effect of ACAS II
specifically for that regime or procedure. In general, this is an argument for having the tools,
adapted for European airspace, to carry out the studies when they arise. However, the specific
case of RVSM airspace has already been studied, in ACASA WP-3 [31] [32], and this study
used the tools developed specifically for European airspace in WP-1.
1.4.4
Even though the mandate is a fait accompli, and it is correctly recognised that ACAS II is a
valuable safety device that reduces the risk of collision, there are aspects of the question that
had still not yet been completely evaluated. Examples are;
· the effect of imperfect surveillance on the performance of the logic;
· the effect of slow pilot response; and
· multiple encounters.
6
Traffic Advisories are preliminary alerts issued by ACAS in order to cue a visual search for
the potential threat, and to prepare the pilot for a possible Resolution Advisory (advice to take
specified avoiding action).
7
Gilham encoding errors in altitude reports are a notable exception, and it is unfortunate that it
has not been possible, due to lack of time and resources, to consider them during this study.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 4
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
It was not considered that investigating any of these issues would call the value of the ACAS II
mandate into question, but rather that they merited quantitative investigation so that any
problem could be addressed. For example, a conclusion that pilots do indeed often respond
slowly to ACAS RAs, and that this has a significant adverse effect on the performance of
ACAS, could be addressed through pilot training.
1.4.5
Finally, prior to ACASA, the tools that had been developed for the study of ACAS II safety
were specific to the USA, France [33] or the UK, or were intended solely to evaluate the logic
on the basis of a model that did not represent any region or airspace. Yet, as has been noted in
paragraph 1.3.5, it is known that the performance of ACAS II depends on the airspace in which
it is used. Thus, it seemed necessary to develop operationally realistic advice that applies to
Europe.
1.5
ACASA8 project
1.5.1
ACASA, a EUROCONTROL ACAS Programme project supported by European Commission
Trans-European Network funds, has investigated several areas related to ACAS II operations in
Europe [34]. This report concerns Work Package 1 (WP-1). Other Work Packages cover
· the interaction between ACAS II and RVSM,
· the value for ACAS II of extending the requirement to report altitude via SSR,
· the effect of ACAS II on the SSR electromagnetic environment,
· the development of training material on ACAS II,
· a package of studies of ACAS II Mode S signals and their data content, and
· the potential value of ACAS horizontal resolution advisories (in addition to the present
ACAS II vertical resolution advisories).
1.5.2
WP-1 concerns the safety of ACAS II. The objectives have been described in section 1.1, and
the work package is discussed in greater detail in chapter 2.
1.5.3
The ACASA partners involved in WP-1 are EEC, QinetiQ (formerly DERA), Sofréavia and
CENA.
1.6
Scope of the document
1.6.1
This paper is the formal report on WP-1, and its delivery completes WP-1. It reports the work
carried out in the various sub-packages discussed in chapter 2. However, apart from the
minimum necessary discussion of WP-1 per se, it is intended as a non-technical discussion of
safety issues relating to ACAS II.
1.6.2
Technical material is presented in other working papers written during WP-1, or in the
Appendices where these are required for completeness.
1.6.3
As noted above, the document is mainly concerned with whether or not ACAS II is effective
when operating as designed. Safety issues that it shares with any other avionics system, which
would normally be considered by the safety authorities before the equipment is certified for
use, are not all covered in this report; there is no attempt to view ACAS II from that
perspective. If ACAS II reduces the risk of collision, the safety of flight is increased.
8
Airborne Collision Avoidance Systems Analysis
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 5
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
1.7
Structure of the remainder of the report
1.7.1
This report is structured to try to distinguish between the facts of WP-1 on one hand and a
report on, and discussion of, the safety of ACAS II on the other. The need to discuss the tools
and methods used, albeit in general terms rather than technical detail, has also been separated
from the discussion of the results found using those tools.
1.7.2
Chapter 2 discusses WP-1: what was planned, and why, and what was actually carried out. In
effect, chapter 2 reports that WP-1 was carried out, as planned.
1.7.3
Chapter 3 contains the discussion of the tools and methods used. It describes how calculations
are made of the performance of the logic, and the tool used to examine the whole system. The
work relating to the ICAO standards is also discussed in this chapter, because of its technical
nature and the fact that it does not relate specifically to Europe.
1.7.4
The results relating to matters that are likely to be of operational interest are in chapters 4–8:
· the performance of ACAS II in Europe – chapter 4;
· the effect of imperfect surveillance – chapter 5;
· the effect of imperfect pilot response – chapter 6;
· the benefits of Phase 2 of the ACAS II mandate – chapter 7; and
· multiple encounters – chapter 8.
1.7.5
The remaining chapters synthesise chapters 4 to 8, and draw out the main conclusions and
recommendations from the work.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 6
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
2
Summary of WP-1
2.1
Introduction to WP-1
2.1.1
The plans and effort estimates for the project are presented in ‘European TEN study: ACAS
Analysis – Work Plan’ [34].
2.1.2
Previous ACAS II safety studies have been reviewed in section 1.3. As noted there, the
operation of the collision avoidance logic has received considerable attention, and ACASA
builds on the technical consensus for evaluating the logic outlined in the SARPs for ACAS II.
However, the approach described in the SARPs was developed for technical purposes and, as
stressed in the SARPs themselves, it does not yield operationally realistic advice. Thus, the
methods described in the SARPs require adaptation in order to deliver realistic advice for
Europe. Also, it was noted in paragraph 1.3.7 that the two implementations of the SARPs
approach prior to ACASA did not yield the same results. For these reasons, the SARPs
approach was to be implemented a third time, and then the tools used were to be adapted
specifically for European airspace.
2.1.3
Additionally, several aspects of the system as a whole were to be studied; and a new tool
(based on established methods) to estimate the performance of the whole ACAS II system,
capable of incorporating the estimates of the performance of the logic, was to be developed.
2.1.4
The earlier calculations of the performance of the logic have made precise assumptions about
the other aspects of the system, usually that they are working as intended. For examples: the
surveillance system produces the required report at every attempt; the pilot responds to advice
exactly as anticipated; and the aircraft is fully capable of performing all the manoeuvres that
the system recommends. These limitations could be significant to the total system
performance. In particular, there were concerns about the potential consequences of extending
the ACAS II mandate to small aircraft, whose supposed limited performance characteristics
were not considered when the logic was designed. These issues were to be addressed.
2.1.5
At the beginning of the work, the Work Plan posed the following questions for WP-1.
· What is the overall reduction in the risk of collision in European airspace obtained by
extending the ACAS II mandate to small aircraft?
· What benefit do small aircraft gain in terms of reduced risk of collision from fitting
ACAS II in an environment where large aircraft are already equipped (and thus some
measure of protection is already available, by proxy)?
· Is the performance of small aircraft so limited that ACAS II is inherently of less value to
them than to large aircraft, and how does this affect the answers to the previous questions?
· What additional benefit do large aircraft, which are already ACAS II equipped, gain when
small aircraft are also equipped?
· What is the effect on the risk of collision of non-response and slow response to ACAS II
avoidance manoeuvres?
· What effect on the risk of collision do the situational awareness and alerting functions of
ACAS II provide?
· Can a useful reduction in the risk of collision be achieved by mandating the use of 25ft
altitude data by suitably equipped aircraft?
· Is the frequency of multiple encounters (close encounters between more than two aircraft)
high enough to warrant the inclusion of this factor in calculations of the risk of collision
with or without ACAS II?
· What effect does version 7, vice 6.04A, have on the risk of collision in multiple encounters?
These questions will be addressed in the Conclusions to the report.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 7
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
2.2
The WP-1 work packages
2.2.1
WP-1.1: Development of tools required to calculate ‘logic risk ratio’9 for Europe
2.2.1.1
This work package falls into two natural halves.
2.2.1.2
The inconsistencies between the calculations of the efficacy of the ACAS II (performed
separately by MITRE and by CENA) were to be resolved by having QinetiQ repeat the
calculations. The ICAO standard encounter model was to be implemented by QinetiQ. QinetiQ
and CENA would then be able to compare their calculations, both the methods and the results,
and to repeat the calculations until acceptable agreement was achieved.
2.2.1.3
The ICAO standard ACAS encounter model was to be developed into a European ACAS safety
encounter model, designed to provide a basis for operational advice concerning the effect of
ACAS II on the risk of collision in Europe. Accordingly, a new model designed for European
airspace was to be specified, jointly by CENA and QinetiQ, and implemented independently by
both. Building the numerical details of the model would require analysis of a large number of
actual encounters, observed by ATC radars. When completed, QinetiQ and CENA were to
compare their calculations again.
2.2.1.4
This summary of the work that was to be carried out in WP-1.1 is deliberately terse. The tools
are of little interest in themselves, and the interested reader is referred to the working papers.
However, in terms of effort, WP-1.1 was huge: larger than all the remainder of WP-1, and
comparable on its own to any other ACASA Work Package.
2.2.2
WP-1.2: Performance limitations of small aircraft
2.2.2.1
This task was based in part on a premise that small aircraft suffer more stringent performance
limitations than large aircraft. The objectives were to assess these limitations, and then to
evaluate the benefit to the small aircraft of equipping with ACAS II. The benefit to the large
aircraft, already equipped with ACAS II, of equipping the small aircraft was also to be evaluated.
2.2.2.2
The first step was to review the performance limitations of small aircraft, and the implications for
the suitability of equipping these aircraft with ACAS II. To anticipate, the review of performance
limitations covered all aircraft. The aim was to determine whether there are such limitations in
any of the aircraft likely to be affected by a mandate, and which aircraft are limited.
2.2.2.3
The effect of performance limitations was to be quantified, and the benefit of extending the
mandate quantified. At this stage, it is again necessary to anticipate and make it clear that these
two estimates are distinct, not least because it was found that small aircraft are no more
performance limited than the large aircraft already equipped with ACAS II.
2.2.2.4
An estimate of the proportion of aircraft likely to report altitude in 25ft increments, rather than
the 100ft increments to which Mode C is limited, was also required. The benefits of the finer
quantisation were to be quantified.
2.2.3
WP-1.3: Effect of surveillance performance on the ACAS II logic
2.2.3.1
CENA were to develop a model of ACAS II surveillance so that the sensitivity of ACAS II
performance to imperfect surveillance could be estimated. This model was also to be used in
WP-1.4 as a framework for analysing actual ACAS II surveillance performance on the basis of
data recorded in flight. In the event, a limited analysis of the small quantity of data available
was also carried out in WP-1.3.
2.2.3.2
The effect of imperfect surveillance on the performance of ACAS II was then to be calculated.
9
The effect of ACAS on the risk of collision is expressed through a measure, referred to as
‘risk ratio’, which expresses the risk of collision with ACAS as a ratio to the risk of collision
without ACAS. Risk ratio is discussed further in section 3.1.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 8
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
2.2.4
WP-1.4: Quality of the ACAS II surveillance performance
2.2.4.1
Data from ACAS II flight recorders was to be used to evaluate the performance of ACAS II
surveillance, using the model developed in WP-1.3 as a framework. In practice the quantity of
data was limited, and this task was carried out as part of WP-1.3. WP-1.4 will not be discussed
further.
2.2.5
WP-1.5: Multiple encounters
2.2.5.1
Multiple encounters are rare but they occur. Furthermore, concerns about the risk that ACAS II
will cause the equipped aircraft to move into the path of a third party are frequently expressed,
though usually by people meeting ACAS for the first time. The multiple encounter logic has
been dramatically improved in version 7 by comparison with version 6.04A [35].
2.2.5.2
The purpose of this Work Package was to evaluate the significance of multiple encounters to
the overall safety of ACAS II. Part of the aim was to justify the simplification in the remainder
of the ACAS II safety study that multiple encounters can be neglected. The frequency of
multiple encounters had to be estimated, and the performance of ACAS II in those encounters
assessed. Two approaches were planned.
2.2.5.3
Radar data recordings for actual multiple encounter events observed during the operational
evaluation of ACAS II were to be used to assess the performance of ACAS II.
2.2.5.4
Saturation radar data recordings were to be examined to collect multiple encounter events. The
performance of ACAS II in these encounters was simulated. In this case, estimating the
frequency of multiple encounters was expected to be relatively straightforward. Obviously, the
result would apply to the particular airspace studied – one of the busiest airspaces in Europe,
with many holding patterns.
2.2.6
WP-1.6: ACAS II collision risk ‘event tree’10
2.2.6.1
The objective of this Work Package was to construct an event tree that enables an estimation of the
overall value of ACAS II, taking all factors that have a material effect on the result into account.
The study would start with earlier work in this area [29]. The aim was to construct a relatively
simple tree that would elucidate the main processes. For example, software coding errors and
equipment failure were to be excluded, as was consideration of the proximity of the ground.
2.2.6.2
WP-1.7 was to provide the human factors information required for WP-1.6.
2.2.6.3
The event tree was to be used to estimate the overall efficacy of ACAS II, including its manner
of use, from the perspectives of the large aircraft already equipped with ACAS II and small
aircraft covered by the Phase 2 mandate (see paragraph 1.2.6). Two estimates were required,
once when the small aircraft are not equipped with ACAS II, and again when the small aircraft
are equipped with ACAS II.
2.2.7
WP-1.7: Human reaction to ACAS II alerts and displays
2.2.7.1
This Work Package was to provide a review of the way pilots respond to ACAS II advice to
take avoiding action, and of the extent to which pilots obtain other benefits (situational
awareness and preparedness) from ACAS II.
2.2.7.2
Flight recorder data were to be examined for evidence on the nature of actual pilot responses. The
effect of the alternative pilot responses on the performance of the logic was to be evaluated.
10
This was originally referred to as a ‘fault tree’, but the term ‘event tree’ is now preferred,
because it seems more accurate. Many of the events described in the tree and potentially
leading to collision cannot be properly characterised as ‘faults’. Nevertheless, readers who are
familiar with fault trees and event trees might consider the term fault tree more appropriate,
because it might fits better with their technical experience.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 9
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
2.2.7.3
EDS had carried out a theoretical study of the use of ACAS II from a human factors
perspective [36], and this was to be reviewed to provide revised advice suitable for us with the
event tree developed in WP-1.6.
2.2.8
WP-1.8: The final report
2.2.8.1
This final report was to be written.
2.3
The work carried out
2.3.1
WP-1.1: Development of tools required to calculate ‘logic risk ratio’ for Europe
2.3.1.1
QinetiQ implemented the ICAO standard encounter model; QinetiQ’s calculations were
already compliant with the ICAO standards in other respects. Sofréavia and CENA had already
implemented the ICAO methods. The two approaches (QinetiQ and Sofréavia/CENA) were not
identical; nor did they need to be, and neither was the same as MITRE’s approach (the original
for the ICAO standards). Unsurprisingly, the first results did not agree, and the QinetiQ and
Sofréavia/CENA approaches had to be compared in detail (both the implementations of the
ICAO encounter model and the methods of calculation) in order to achieve reasonable
agreement [WP-1 035] [WP-1 036] [WP-1 041] [WP-1 062] [WP-1 065]. There were
corrections and amendments on both sides, until it was felt that the agreement was acceptable
and that further effort would not be cost effective [WP-1 073]. The Sofréavia/CENA
calculations had converged with those of MITRE during this process, and all three risk ratio
calculations were reported to SICASP, for information, in November 1999 [37].
2.3.1.2
ACASA reported to SICASP/7 in September 2000 on the experiences of Sofréavia/CENA and
QinetiQ in the use of the ACAS encounter model [37]. These conclusions are discussed, in
very brief terms, in section 2.4.
2.3.1.3
The next task was to specify and implement the European ACAS safety encounter model,
which is discussed in section 3.4. QinetiQ and Sofréavia/CENA developed the specification
jointly [WP-1 186], and a précis of the specification is available [WP-1 101]. QinetiQ and
Sofréavia/CENA each implemented the European model, independently.
2.3.1.4
To complete the encounter model, by tuning it to the real, European, world, it was necessary to
process radar data and collect a large, representative, sample of real encounters. The
specifications of the criteria for selecting these encounters were developed jointly [WP-1 075]
[WP-1 078] [WP-1 085]. Both organisations analysed about six months’ worth of ground radar
data. Sofréavia reported on the French encounter collection exercise, which used data from
five radars across France and yielded 1243 encounters [WP-1 116]. QinetiQ collected another
1144 encounters using data from a single radar in South-East England, so that a total of 2387
encounters were available for further analysis. These are all encounters in which the hmd was
small (less than 2NM at high altitude; less than 0.5NM below FL50), and in which there was
also some prospect of an ACAS RA.
2.3.1.5
CENA developed and delivered software to analyse these encounters, and construct the
encounter model tables [WP-1 086]. Through the application of this software to the 2387
encounters captured from radar data, the two implementations of the European ACAS safety
encounter model could be completed.
2.3.1.6
There are two significant differences between the two implementations, and a third related
point.
· Both QinetiQ and Sofréavia/CENA adjust the statistical distribution of vmd in order to
imply a pre-determined underlying NMAC rate in the absence of ACAS; this is discussed in
section 3.6. They carry out this adjustment differently, and resources were not available to
adopt a common method. Calculations by both organisations show that ACAS is judged
more effective when the Sofréavia/CENA method is used.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 10
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
· The specifications for the European ACAS safety encounter model include the concept of
‘aircraft class’. Sofréavia/CENA, who required it in order to carry out WP-1.2,
implemented this feature. QinetiQ did not implement this feature, because aircraft class
does not appear necessary for studies other than WP-1.2, and resource limitations would
have prevented the work. The merits of the feature are discussed in section 3.9.
Sofréavia/CENA have shown that ACAS is judged more effective when aircraft class is
included in the model [WP-1 182].
· In addition to these known differences, the two organisations obtain different results when
these known differences of method are removed. Again, ACAS tends to be judged more
effective by the Sofréavia/CENA calculations.
2.3.1.7
Altogether, these three factors lead to results differing by about 50%11; each of the factors
contributes to the comparison. The objective was to install two mutually validated facilities for
assessing the safety of ACAS. This has been achieved in considerable measure. The initial
comparison and the joint development of the model specifications have improved both
facilities significantly. The models are compatible when questions such as ‘What is the effect
of using 25ft data?’ and ‘What is the effect of a slow pilot response?’ are addressed. When, as
happens, one of the models seems to yield surprising results, the question ‘Why?’ is posed and
itself leads to valuable conclusions: for example, the identification (or rediscovery) of precise
problems in the collision avoidance logic [WP-1 187].
2.3.1.8
A European ACAS safety encounter model has been developed and implemented. The
existence of two implementations means that it is possible to verify any conclusions that
warrant such attention.
2.3.2
WP-1.2: Performance limitations of small aircraft
2.3.2.1
Sofréavia drafted detailed plans for WP-1.2, and subsequently carried out the work [WP-1 145]
[WP-1 172].
2.3.2.2
This task required the inclusion of distinct aircraft classes in the encounter model, as envisaged
in an ACASA working paper [WP-1 005], and discussed in section 3.9.
2.3.2.3
This work addressed the value of equipping with ACAS from an airspace point of view – what
was the reduction in the risk of collision in the airspace in question? After completion, interest
turned to the value of equipping with ACAS from the point of view of individual aircraft, i.e.
operators. Sofréavia drafted plans for an extension, and carried out further work to examine
this issue [WP-1 184] [WP-1 204].
2.3.2.4
The results of WP-1.2 are presented and discussed in chapter 7.
2.3.2.5
WP-1.2 concentrated on the question of equipping small aircraft with ACAS. The proportion
of aircraft that report altitude in 25ft increments, as opposed to 100ft increments, was
determined as part of WP-1.2, and used to define the scenarios for which results are presented.
2.3.2.6
Paragraph 2.1.5 lists a further question: ‘Can a useful reduction in the risk of collision be
achieved by mandating the use of 25ft altitude data by suitable equipped aircraft?’ WP-1.2 did
not address this question, because there was no need at that stage. It is well known that the
availability of 25ft altitude data from the intruder increases the efficacy of ACAS [18] [28]
[38] [39], and straightforward to confirm this with calculations using the tools developed in
WP-1.1. The value of increasing the reporting of 25ft altitude data is discussed in section 4.3.
11
The larger is 1.5 times the smaller; the factor is not constant. On the other hand, the
differences between the performance of ACAS in ACAS/ACAS and ACAS/unequipped
encounters, or between the performance of ACAS above FL295 and in the whole of the
airspace, are very much greater than this.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 11
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
2.3.3
WP-1.3: Effect of surveillance performance on the ACAS II logic
2.3.3.1
CENA specified a model for imperfect ACAS surveillance, which was discussed and agreed by
EEC and QinetiQ [WP-1 155]. CENA and Sofréavia used this as the basis for evaluating the
effect of imperfect surveillance on ACAS performance [WP-1 173]. This work is reported in
chapter 5.
2.3.4
WP-1.4: Quality of the ACAS II surveillance performance
2.3.4.1
See paragraph 2.2.4.
2.3.5
WP-1.5: Multiple encounters
2.3.5.1
CENA and Sofréavia investigated 11 actual ACAS events in which a third aircraft was
proximate [WP-1 146]. In particular, they examined the interaction with ATC.
2.3.5.2
QinetiQ investigated 8 similar events found during the processing of six months’ worth of
radar data. In this case, they examined the effect of making the hmd zero in order to find the
performance of the logic should there have been an actual risk of collision [WP-1 212].
2.3.5.3
The 11 actual ACAS events were all the ACAS events that have been reported by a pilot or a
controller during several years of evaluation, throughout Europe, for which there is radar data.
The 8 events found in the radar data were the only multiple encounters in the six month’s
worth of radar data. It seems that pilots and controllers do not complain of multiple encounters
very frequently, and that they happen rarely.
2.3.5.4
The results of these studies are presented in chapter 8.
2.3.6
WP-1.6: ACAS II collision risk ‘event tree’
2.3.6.1
QinetiQ developed a logical diagram that enables calculations of the efficacy of ACAS in
reducing collision risk [WP-1 197]. The diagram has been implemented as an Excel
spreadsheet calculation, which has been delivered [WP-1 217].
2.3.6.2
The calculations require statements of the probabilities of many other events, some faults (the
probability that a pilot will fail to respond to an RA) and others not faults (the pilot contacts
the controller following an ACAS alert). Default values for these probabilities, and reasons for
the values that have been selected, have also been delivered [WP-1 213].
2.3.6.3
The sensitivity of the calculations to variations in the assumed probabilities within their
realistic range has also been examined [WP-1 216].
2.3.6.4
The event tree is discussed in section 3.10, part of the chapter concerning tools and methods.
Thereafter, it has been used throughout the report to provide advice on the likely performance
of the system as a whole, rather than just the performance of the logic.
2.3.7
WP-1.7: Human reaction to ACAS II alerts and displays
2.3.7.1
Sofréavia examined flight-recorded data provided by EEC to determine how pilots responded
to RAs in practice [WP-1 161]. They formulated alternative pilot response models and used
these to investigate how the alternatives affected the performance of the collision avoidance
logic [WP-1 171].
2.3.7.2
The importance of the pilot’s response in determining the effectiveness of ACAS is discussed
in chapter 6.
2.3.7.3
QinetiQ considered these results obtained by Sofréavia and the earlier work of EDS on the
effect of human factors on the performance of ACAS [36] in order to determine the best values
to use for various human factors events in the event tree discussed in section 2.3.6 [WP-1 213].
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 12
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
2.4
Results concerning the ICAO standards
2.4.1
It has been reported above that the ICAO standards for evaluating the performance of the
ACAS collision avoidance logic [26] were the starting point for the development of tools
designed specifically for Europe. The main innovation was the need to implement the ICAO
standard encounter model – the model of the idealised, non-existent airspace to which the
performance standards apply. ACASA learnt a great deal about the standards themselves in the
process of using them. These experiences were reported back to SICASP/7 [40], and have been
recorded in a more complete ACASA paper [WP-1 109]. This section presents the main points
from that report.
2.4.2
The report to SICASP/7 included the performance figures calculated by ACASA, but they are
not repeated here. The ICAO standards include statements of the performance to be achieved
by the ACAS logic. They also make clear that these standards are not to be interpreted as
guidance to the actual performance of ACAS in the real world. The values of the performance
measures calculated using the standard ICAO encounter model relate to that model, and not to
Europe. As reported above, the Sofréavia/CENA values, the QinetiQ values and the values
calculated by MITRE, on which the standards are based, are in reasonable agreement.
2.4.3
There were two sorts of comments on the ICAO model: those of a technical nature, concerning
the way in which the aircraft trajectories are constructed; and those bearing more directly on
the operational realism of the model. Appendix A contains an abbreviated version of the points
made in the report to SICASP [WP-1 109].
2.4.4
In the main, the technical points concern unresolved ambiguities. Resolving these ambiguities
requires changes in definitions, or a slightly richer structure for the conceptual model of an
encounter. These problems have been resolved in the European ACAS safety encounter model.
2.4.5
The points concerning lack of operational realism arise either from some structural feature of
the specification, or from the data used to tune the model to reality. An example of the former
is that, when one aircraft is climbing, the ICAO standard encounter model specifies that other
aircraft is equally likely to be climbing or descending; this arises because of the way the
encounter classes are constructed. An example of the latter is that the altitude distribution of
encounters is not realistic for Europe: it contains too high a proportion of low-level encounters.
2.4.6
Most importantly, the calculation of the efficacy of the ACAS collision avoidance logic was
shown to be critically dependent on the airspace model used and its implementation. This is
effectively a reprise of the point made in paragraph 1.3.5 that the result is dependent on the
airspace for which it is calculated. It also indicates that future changes to ATM operations
may have an impact on current ACAS performance.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 13
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
3
Tools and methods
3.1
Risk ratio
3.1.1
The effect of ACAS on the risk of collision is usually expressed through a measure referred to as
‘risk ratio’, which expresses the risk of collision with ACAS as a ratio to the risk of collision without
ACAS. This is a relative measure. ACAS does not make flight safe; it makes it safer, and the extent
to which it makes it safer is expressed as a fraction of the risk of collision in the absence of
ACAS. A risk ratio of 0.2, or 20%, means that ACAS removes 80% of the risk of collision.
3.1.2
The fact that it is a ratio with the pre-existing risk of collision means that risk ratio is extremely
sensitive to the underlying safety of the airspace. This has been discovered repeatedly, and is
discussed in section 3.6. A high risk ratio means that ACAS is not effective at reducing the risk
of collision, but this can be due simply to the fact that there is little risk of collision to reduce.
Since, unavoidably, ACAS has the potential to induce some collisions, and this tends to happen
in particular circumstances, risk ratio should be expected to be extremely sensitive to the
airspace for which it is calculated, as has been found and has been noted above.
3.1.3
Calculations of risk ratio are made for the collision avoidance logic considered in isolation (the
‘logic risk ratio’) and for the system considered as a whole (the ‘full system risk ratio’).
Questions such as ‘What is the effect of pilots responding slowly to RAs?’ and ‘What is the
effect of reporting altitude in 25ft increments?’ can be considered in the context of the logic
alone. They could also be considered for the whole system, but this tends not to happen for two
reasons. Firstly, until now, the tools have not been available to consider the whole system
reasonably conveniently, at reasonable cost. Secondly, even now, the methods require a large
number of ‘best guesses’, and thus the results are much more vague. When a risk ratio result is
stated, it is important to note whether it relates to the logic, operating under some specific
assumptions (possibly ideal circumstances), or to the whole system.
3.1.4
Risk ratio is calculated for NMACs rather than collisions. An NMAC is an encounter in which
horizontal separation is less than 500ft and vertical separation less than 100ft simultaneously.
In this report, it is generally taken to be an encounter in which hmd < 500ft and vmd < 100ft.
The reasons for discussing NMACs rather than collisions are matters of history, convenience
and politics. (People do not have an immediate emotional response to the word ‘NMAC’.)
However, the practice tends to over-estimate risk ratio; in other words, it errs appropriately, on
the conservative side [41].
3.2
Risk ratio for the airspace, or for the aircraft
3.2.1
The benefits of equipping with ACAS can be expressed from two alternative (symbiotic)
points of view:
· that of the ATS airspace regulator – ‘What is the effect on the risk of collision within an
airspace?’; and
· that of the airline operator – ‘What is the effect on the risk that one of my aircraft will be
involved in a collision?’
3.2.2
The concept of risk ratio is the same in the two cases: it expresses the risk of collision with
ACAS to the risk of collision without ACAS. However, in general, the numerical result is not
the same (and nor is the practical advantage gained by the two people posing the question). To
understand this, consider the effect of a universal mandate. The airspace regulator will
compare the risk of collision between two equipped aircraft with the risk of collision between
two unequipped aircraft. The airline operator will appreciate this perspective, but might want
to know when it is of most value to equip. If he equips before anyone else, he will compare the
risk of collision between an equipped aircraft (his own) and an unequipped aircraft with the
risk of collision between two unequipped aircraft. If he equips last, he will compare the risk of
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 14
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
collision between two equipped aircraft with the risk of collision between an equipped aircraft
(the other aircraft) and an unequipped aircraft (his own). In both cases, the answer for the
airline operator is different to that for the airspace regulator. (Incidentally, even this limited
test will usually suggest it is best to equip first.)
3.2.3
The term ‘risk ratio’ usually refers to the airspace perspective, and this will not be spelt out
each time. However, the analyses reported in this paper have also been carried out from the
aircraft point of view. The results will be reported, and it will be made clear that they relate to
the aircraft point of view.
3.2.4
Consistent use of terms is required to avoid an ambiguity in the concept of ‘aircraft risk ratio’.
The concept of ‘airspace risk ratio’ is relatively straightforward, and always means the ratio of
the risk of collision in the airspace for a known ACAS equipage scenario to the risk of
collision were ACAS totally absent. ‘Aircraft risk ratio’ could refer to the base state in which
ACAS is totally absent on all aircraft, or it could refer to a base state immediately before a
change in equipment is made on a particular aircraft (as suggested in paragraph 3.2.2). For
clarity, two distinct terms are used for these two alternatives. The term ‘aircraft risk ratio’ will
be used when referring to the base state in which ACAS is totally absent (as does airspace risk
ratio). The term ‘progressional risk ratio’ will be used when discussing the immediate
advantage to an aircraft of an equipment change, judged using the environment at the time of
the change as the base state.12
3.3
Estimating the performance of the logic
3.3.1
The logic risk ratio is calculated by simulating the operation of the collision avoidance logic in
very many encounters. These encounters are hypothetical, not real, because aircraft have a
fortunate tendency not to come sufficiently close sufficiently frequently to provide a statistically
realistic test of ACAS. The construction of these encounters is discussed in the next section.
3.3.2
In these encounters, the trajectories of both aircraft without the intervention of ACAS are
known. ACAS range measurements and the altitude reports available to ACAS are calculated
from the trajectories and supplied to the ACAS logic. If there is an ACAS RA, the trajectory of
the aircraft is modified in response to the RA, using whatever assumptions have been made
about how the pilot will behave.
3.3.3
These calculations typically assume that other aspects of the system are working perfectly. For
example, it is usually assumed that the surveillance system produces the required report on
every one-second cycle. Less often, but still typically, it is assumed that the pilot responds to
advice to climb or descend exactly as anticipated. It is almost always assumed that the aircraft
is fully capable of performing all the manoeuvres that the system can recommend. (There are
good reasons for that last assumption.) These assumptions can all be varied.
3.3.4
Knowing the trajectories of the two aircraft with and without ACAS, it is possible to count the
numbers of NMACs with and without ACAS. Altimetry error is also considered at this stage.
3.3.5
The method of allowing for altimetry error supposes that the known trajectories of the aircraft
are based on altimeter measurements, and are thus in error – they are only approximate. The
altimeter errors are added after the simulations are complete, in order to recover the ‘true’
altitudes of the two aircraft. The numbers of NMACs are calculated, or counted13, after
altimeter error has been added. The model used for altimeter error is that specified in the ICAO
SARPs for ACAS, except where otherwise stated [26].
12
Readers should be aware that this term is not standard and that in some of the references this
quantity is referred to as ‘aircraft risk ratio’.
13
QinetiQ finds this result by calculating, for each encounter, the probability that altimeter
error results in an NMAC. Sofréavia/CENA randomly samples a pair of altimeter errors for
each encounter, and add it. The two methods are equivalent for large samples.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 15
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
3.4
A European ACAS safety encounter model
3.4.1
The ICAO SARPs specify a standard encounter model for the calculation of logic risk ratio
[26]. This model consists of a framework within which encounters can be constructed given
the values for a range of parameters, and probability tables and distributions from which to
sample values for those parameters randomly. Any number of encounters can be generated.
The ACASA experience with this model was reviewed in section 2.4.
3.4.2
The results reported here are based on a European ACAS safety encounter model, which is a
substantial revision of the ICAO model. The most significant improvements include the
following.
· The European model distinguishes between climbing and descending aircraft, so that
encounters in which one aircraft is climbing and the other descending can be more frequent
than encounters in which both aircraft are climbing (or both are descending).
· The European model enables the probability of a vertical manoeuvre to be related to the
vmd.
· The European model has many more encounter types.
· Both encounter models are based on an abstract encounter structure, defining a collection of
properties that are required to construct individual encounters. The European model’s
encounter structure is richer and more coherent than that of the ICAO model.
3.4.3
The specification of the European model was a living document during the ACASA project,
the final version not appearing until December 2001 [WP-1 186]. A précis of the specification
was prepared in February 2000, and provides more detail than is provided here [WP-1 101].
3.4.4
The encounter model is a critical component in the calculation of logic risk ratio. Changes that
might seem small can have a large effect on the results and this, once again, indicates that
ACAS performance is dependent on the airspace.
3.5
Data for the European ACAS safety encounter model
3.5.1
The encounter model has two components: a structure; and the probability values for the
various options that the structure defines. The probability values are given in the probability
tables mentioned in 3.3.1. The ability of the model to reflect any airspace accurately (for risk
ratio calculations) depends on the structure of the model, but the tables have to be filled with
appropriate probability values to produce an accurate model. The probability values are found
by analysing a large quantity of ATC radar data for the airspace that is to be modelled. The
trajectories of individual aircraft observed in the radar data are reconstructed, and tested to find
whenever two trajectories are sufficiently close to constitute an ‘encounter’. The
characteristics of each encounter found in this way are compared to the structure of the
encounter model, and all the relevant properties of the encounter are classified. It is then
possible to increment counters, which correspond to the appropriate entries in the probability
tables, in respect of each encounter. At the end of the exercise, the probability tables have been
populated with counts, which can be converted to the probabilities required.
3.5.2
Two aspects of this process are important:
3.5.3
·
the airspace in which the radar data are gathered; and
·
the criteria used to identify the encounters that are to be analysed.
French and UK radar data were used, amounting altogether to about a year of coverage from a
single ATC SSR radar. In the UK, data from a radar in South-East England were collected
between June and November 1998. These data represent 154 days of observation (many
incomplete), correspond to approximately 3.4´105 flying-hours, and yielded 1144 encounters.
In France, several radars were used between January and May 2000, giving good coverage of
France as detailed in an ACASA working paper [WP-1 116]. The French data represent 5585
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 16
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
hours of observation, corresponded to approximately 6.9´105 flying-hours, and yielded 1243
encounters.
3.5.4
The characteristics of the UK and French radar data have been compared [WP-1 148], and
there are significant differences, no doubt reflecting differences in operational practices.14 It is
valuable that both contain features widening the relevance of the model, i.e. making it more
relevant to Europe as a whole, but the drawback is that the combined model represents an
amalgam that does not exist in either country. Furthermore, the relative weights of the parts of
the amalgam are an unconsidered consequence of simply pooling the two sets of data.
Ultimately, there is no resolution to this problem: it is not possible to collect sufficient data for
any one location to form a model; and collecting data from several locations produces a result
that represents none faithfully. Details of the encounter model (specifically, the boundaries
between altitude layers) were varied so as to minimise the differences between the UK and
French data.
3.5.5
The criteria used to select encounters have as great an effect on the quality of the model as the
radar data used [WP-1 075]. The precise criteria used are specified in a working paper [WP-1
085]. The encounters collected were those where:
· the aircraft were sufficiently close, in terms of a range closure rate test and a vertical
proximity or closure rate test, to be candidates for an ACAS RA; and
· the horizontal miss distance was sufficiently small to be regarded as random, i.e. only by
chance not zero.
These encounters were then processed and treated as though the hmd had been effectively zero.
The basis for these tests is that the only encounters that are relevant to ACAS risk ratio are
those in which there is a real risk of collision or a real risk that an ACAS RA will induce a
collision.
3.5.6
The encounters captured during this process could be inappropriate for many reasons.
Examples of potential problems are:
· both aircraft are military jets;
· neither aircraft would fit ACAS for other reasons (two GA);
· horizontal separation is preserved even though the hmd is small (e.g. approach to CDG
airport, Paris);
· altitude reporting errors (e.g. swapping of the Mode C reports from the two aircraft).
It is hoped that every unrealistic encounter has been removed. This was achieved by a
combination of objective tests, for example based on the aircraft’s Mode A code, and
subjective examination.
3.6
NMAC rates
3.6.1
The ACAS risk ratio is extremely sensitive to the NMAC rate in the airspace for which it is
calculated [WP-1 074]. This is readily understood. The risk ratio is the sum of two
components: one corresponding to the ability of ACAS to resolve collision risk in otherwise
dangerous encounters; and a second corresponding to the risk that ACAS will cause an NMAC
by generating an RA in a safe encounter. All other things being equal, a higher NMAC rate
leads to a lower (better) risk ratio, and a lower rate of RAs (a property of the airspace as much
as of ACAS) leads to a lower risk ratio. Because of this sensitivity to the underlying NMAC
rate, it is important that the encounter model is consistent with a realistic NMAC rate.
14
More precisely, they were used separately to produce two distinct models, solely for the
purposes of this comparison.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 17
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
3.6.2
Capturing close encounters and processing them as described in the previous section tends to
lead to models that imply unrealistically high NMAC rates. That this is true was easily verified
by observing the actual record of mid-air collisions. There are far fewer mid-air collisions than
one would expect on the grounds of random chance from the losses of separation, as observed
by processing samples of radar data.
3.6.3
For these reasons, the encounter model is adjusted so that it implies a realistic NMAC rate.
This implies that a realistic NMAC rate is known, but in practice this information is extremely
difficult to obtain. The NMAC rate imposed on the encounter model is given in Table 3.1,
where it is compared with hypothetical frequencies for related incidents.
vmd
hmd
rate
–5
serious loss of separation
500ft
2NM
3´10 per flying-hour
critical airmiss
400ft
1500ft
3´10–6 per flying-hour
NMAC
100ft
500ft
3´10–7 per flying-hour
collision
3´10–8 per flying-hour
Table 3.1: Various expressions of the assumed collision risk without ACAS
3.6.4
The NMAC rate of the encounter model is adjusted by varying the probability that a random
encounter generated by the model will have vmd < 100ft. The way in which this adjustment
can be made is described in a working paper [WP-1 115]; in practice, CENA and QinetiQ use
slightly different methods, but based on the same objectives and the same general approach.
(The consequence of using different methods was discussed in section 2.3.) In order to relate
the probability that a random encounter will have vmd < 100ft to the desired NMAC rate, it is
necessary to have some notion of the frequency with which the encounters represented by the
model occur in the real world. This can come from either of two sources:
· noting the rate at which RAs occur in the real world, which can be inferred either from
general experience or from an appropriate sample of radar data; or
· noting the time over which the encounters that formed the basis of the statistical analysis
leading to the model were collected.
Comparing the results implied by the two sources has been a valuable check on the soundness
of the estimations.
3.6.5
The sensitivity of risk ratio to a number of factors will be discussed later in this report. Here
we note that the imposed NMAC rate is the single most important factor. Not only does risk
ratio vary greatly as NMAC rate is varied, but the uncertainty in the true value for NMAC rate
is great. The numbers presented in Table 3.1 are not defended here: they are reported. The best
efforts of the consortium have been devoted to using a realistic value, but the consortium
cannot guarantee that the figure used is uniquely correct and would not wish to defend it.
3.7
ACAS II surveillance
3.7.1
Most calculations of ACAS risk ratio are for the logic rather than the system as a whole, and
assume that the ACAS surveillance system works perfectly. In other words, the range of the
intruder from the ACAS aircraft is calculated, and the altitude of the intruder (which is
simulated to arbitrary precision) is converted to a 100ft or 25ft quantised report, on each 1s
cycle and both are provided to the ACAS logic.
3.7.2
The consortium is not aware of any work on the effect of imperfect surveillance on the
performance of the logic prior to ACASA. This issue is addressed in chapter 5.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 18
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
3.7.3
Apart from the work reported in chapter 5, all the logic risk ratio results reported here assume
perfect surveillance. This is justified by the results reported in chapter 5, but only to a limited
extent, because chapter 5 concerns circumstances in which the logic performs well.
3.7.4
Turning to the whole system, ACAS itself can experience imperfect surveillance at two
different levels: the logic can receive an imperfect record of ranges and altitudes for an
intruder that is detected; and ACAS can fail to form a track at all, i.e. an intruder can pass
undetected. The work in chapter 5 concerns the former problem. Section 3.10 describes
calculations of the performance of the whole system, and these should anticipate both aspects
of imperfect surveillance.
3.7.5
The original MITRE safety studies [10] [11] used 0.03 as the estimate of the probability that
ACAS would not form a track for an intruder, meaning (at that time) a Mode C equipped
intruder. This figure has been carried forward into later work, e.g. the report of SICASP/5 on
the safety of ACAS [25]. The SARPs require that tracks be established ‘with at least 0.90
probability that the track is established 30s before closest approach’ [42].
3.7.6
In the event tree work described in section 3.10, the probability of tracking a Mode C intruder
is taken to be 0.97, with 0.95 as a worst case value, and the probability of tracking a Mode S
intruder is taken to be 0.995, with 0.97 as a worst case value. (The tracking of Mode S
intruders is much more reliable than that of Mode C intruders.) The effect of imperfect
surveillance on the operation of the logic is not modelled.
3.8
Pilot response
3.8.1
The ‘standard’ pilot response to a corrective RA is that she reacts within 5s and applies an
acceleration of 0.25g to achieve the required vertical velocity. (The standard response to other,
often more urgent, RAs is given in chapter 6.) The logic has been designed for this response,
and most logic risk ratios assume this response. The manner of modelling the pilot response is
simply to calculate the deviation that should be achieved and to apply that to the original
aircraft trajectory.
3.8.2
Until recently, there has been little compelling evidence concerning how real pilots respond to
RAs. They are trained to follow RAs accurately, and to follow the more urgent RAs more
urgently (but not with greater vertical velocities than those requested). Section 6.2 is a
discussion of actual pilot behaviour.
3.8.3
It has been known for a long time that it is important that pilots respond accurately [43].
Usually, deviant pilot responses are modelled simply by varying the parameters of the pilot
response, but it is possible to model more complex behaviour than simply a slow or a rapid
response. The nature of the RA is known at the point where the pilot response is being
modelled so that, for example, it is possible to calculate the consequences of failing to respond
to particular RAs.
3.8.4
When calculating logic risk ratios consideration of the pilot’s behaviour is limited to her
reaction to RAs. However, pilots can react to ACAS in other ways: by being alerted to a
potential problem; and by being aided in their visual acquisition of an intruder. These
processes are taken into account when considering the performance of the total system, as
discussed in section 3.10.
3.9
Aircraft performance and type
3.9.1
The specifications for the European ACAS safety encounter model include the concept of
‘aircraft class’. This feature was essential for the work of WP-1.2 and was successfully
implemented by Sofréavia/CENA. There are seven aircraft classes defined for the European
ACAS safety encounter model [WP-1 186]:
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 19
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
· Class A: piston aircraft with a maximum take-off mass below 5700kg;
· Class B: turboprop aircraft with a maximum take-off mass below 5700kg;
· Class C: turboprop aircraft with a maximum take-off mass between 5700kg and 15,000kg
(class subject to Phase 2);
· Class D: jet aircraft with a maximum take-off mass between 5700kg and 15,000kg (class
subject to Phase 2);
· Class E: turboprop with a maximum take-off mass in excess of 15,000kg (class subject to
Phase 1);
· Class F: jet aircraft with a maximum take-off mass in excess of 15,000kg (class subject to
Phase 1);
· Class G: high performance (military) jets.
3.9.2
Using these aircraft classes, it is possible to model the varying performance of typical aircraft
in different classes. In particular, it is possible to model the responses to RAs, taking account
of the more limited climb capabilities of turboprop aircraft. However, it should not be
imagined that the ACAS performance calculations reported here apply to all the individual
aircraft types in each class; they are not intended to, and do not. For example, the Belfast is in
Class E, but has much more restricted ability to climb than is modelled for Class E.
3.9.3
The modelling of aircraft class also enables some account to be taken of the airspace
segregation that occurs between the different classes. Different aircraft classes, particularly
military aircraft, are involved in different encounter types. In general, in this work, it is
desirable to include encounters between military aircraft and the aircraft of interest, but
encounters between two military aircraft need to be excluded.
3.9.4
The encounter model contains an extremely large number of degrees of freedom and it is
impracticable to obtain sufficient data to populate the full model well, even without aircraft
classes. Additionally, aircraft class does not appear essential for studies other than WP-1.2, so
QinetiQ has not implemented the aircraft class feature. Sofréavia/CENA have been able to test
the sensitivity of the results obtained using the model by temporarily removing aircraft classes
and found it to be small (as discussed in paragraph 2.3.1.6) [WP-1 182].
3.10
Estimating the performance of the whole system
3.10.1
There are many factors that influence the efficacy of ACAS that are not included, or not
always included, in the logic risk calculations. Examples are:
· whether or not ACAS tracks the intruder;
· whether ACAS prompts visual acquisition;
· whether the pilot sees the intruder (IMC, line of sight, and the chance of actually seeing the
intruder);
· whether or not a controller seeks to ‘help’;
· whether the pilot prefers the controller’s instruction or the RA;
· how the pilot responds to the RA; and
· whether the RA works, given that response.
3.10.2
A collision risk event tree, or a fault tree, is used to calculate the efficacy of taking these
factors and others into account [WP-1 197]. As discussed in section 1.3, the approach is not
new, but it is hoped that this tree is simpler and more accurate than earlier attempts. Having
said that, the tool developed for ACASA is not intended to be comprehensive: it is intended to
provide an approximate calculation of total system risk ratio. These issues are discussed in the
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 20
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
working paper. Safety experts who are used to the methods of fault trees and events trees
should examine the tree as it is (i.e. just a logical diagram for the sake of a calculation) rather
than following their expectations.
3.10.3
The event tree is a very large logical diagram, comprising 535 logic gates. It shows how to
calculate the risk of collision taking into account the interdependence between the events and
circumstances (conditions) affecting the calculation. The tree has been implemented as an
Excel spreadsheet [WP-1 217], which enables instantaneous calculation of the results when the
probabilities of the underlying events are varied.
3.10.4
The tree is garlanded with the probabilities of the events constituting the leaves of the tree. A
complete list of these events and their probabilities is presented in Appendix B. The reasons
for these probability values are given in a working paper [WP-1 213].
3.10.5
The event tree is not a substitute for the logic risk ratio. There are many questions for which an
event tree is not suitable, and the logic risk ratio calculations are essential for the correct
formation of the tree. The logic risk ratio should be more accurate in its own terms than the
event tree is in its terms. For the logic risk ratio one makes many assumptions and then derives
an answer in which one can be confident, but which is subject to those assumptions. For the
event tree, a best judgement is made for everything that matters, and the result is no better than
the ability to judge.
3.10.6
In addition to the probability of every event, bounds on that probability have been estimated.
By making changes in the probability of each event (or related set of events) in turn, it has
been possible to find in what areas our lack of definite knowledge is most serious. The
probabilities that most require further work to improve the quality of the estimates are, in
order:
· the probability that the pilot responds promptly given that she does respond to the RA;
· knowledge of the ACAS equipage level;
· the probability that the pilot ignores the RA.
· the probability of IMC;
· the probability that there is a controller;
· the probability that the controller is already involved in the incident – i.e. the situation has
arisen in part because of a controller instruction, when it is known that the aircraft are on
collision course and that there is a controller;
· the probability that the logic works.
This is not a list of the items having the greatest affect on risk ratio. The factors having most
effect on risk ratio are reported in section 4.5.
3.10.7
The first item on the list produces an uncertainty of about 7ppt in a risk ratio of (say) 25%; in
other words, the risk ratio lies between 18% and 32%, and the reason we cannot be more
certain is that we do not know how reliably the pilot makes an accurate response. This issue is
discussed in chapter 6. The probability of more accurate information concerning the
probability of a prompt response is small. In particular, replies in pilot questionnaires are
clearly of no value in this context. In view of this, it is concluded that the current uncertain
knowledge of the efficacy of ACAS is as good as it gets.
3.10.8
The last item on the list – the probability that the logic works, which is directly related to the
logic risk ratio – produces an uncertainty of about 1.6ppt.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 21
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
4
Performance of ACAS II in European airspace
4.1
The performance of the ACAS II logic
4.1.1
Scenarios
4.1.1.1
The performance of the ACAS II logic is given for two idealised scenarios:
· an encounter between an ACAS equipped aircraft and an unequipped intruder; and
· an encounter between two ACAS equipped aircraft.
4.1.1.2
It has been assumed that the pilots of ACAS equipped aircraft respond to RAs and do so
correctly, with the standard delay and achieving the vertical rates indicated by the RAs. The
effects on the performance of the logic for the case of imperfect response, and for the case of
one pilot ignoring the RA, are discussed in chapter 6.
4.1.1.3
It has been assumed that both aircraft report altitude in 100ft increments. The effect of
reporting altitude in 25ft increments is reported in section 4.3.
4.1.1.4
As explained in chapter 3, the logic risk ratio depends on the model used for altimeter error. In
each equipage scenario, the altimetry error model relevant to that scenario has been used.
4.1.1.5
As explained in paragraphs 2.3.1.6 and 2.3.1.7, there remain further options in the calculation
of logic risk ratio and, consequently, the Sofréavia/CENA calculations and the QinetiQ
calculations are not identical. The results presented below are a best estimate based on both
calculations15.
4.1.2
ACAS – unequipped
4.1.2.1
In encounters between an ACAS equipped aircraft and an unequipped intruder the logic
risk ratio is found to be 22.9%.
4.1.2.2
This expresses the advantage gained by the first aircraft to equip with ACAS, assuming perfect
operation and perfect compliance with RAs. The figure is not meaningful for an airspace. As
more aircraft become equipped, the advantage gained by further aircraft from the fact of
equipping, compared with the risk immediately before equipping increases. (The logic risk
ratio16 improves to 14.6% for the last aircraft to equip.) However, the absolute value of the
reduction in collision risk reduces because the equipage of other aircraft means there is less
risk to remove.
4.1.3
ACAS – ACAS
4.1.3.1
In encounters between two ACAS equipped aircraft, both of which respond as intended,
the logic risk ratio is found to be 3.3%.
4.1.3.2
This expresses the reduction in the risk of collision in an airspace when all aircraft are
equipped with ACAS, assuming perfect operation and perfect compliance with RAs. It is also
the reduction in the risk of collision for each individual aircraft achieved by this hypothetical
global equipage.
15
To be precise, they are estimates made using the event tree spreadsheet. This requires
probabilities for a large number of logic related events. When appropriate probabilities are
supplied to the spreadsheet, and appropriate assumptions made in the form of probabilities for
other events, the spreadsheet reproduces the logic risk ratio. The probabilities that have been used
for the logic events are intermediate between those of Sofréavia/CENA and those of QinetiQ.
16
i.e. the progressional logic risk ratio – risk after that aircraft equips divided by risk
immediately before that aircraft equips.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 22
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
4.2
ACAS II induced risk
4.2.1
Introduction
4.2.1.1
ACAS is designed to reduce the risk of mid-air collision. In many cases it does this by advising
the pilot to manoeuvre vertically (corrective RAs). It is an inevitable feature of any ACAS (as
of any predictive system) that there is a finite probability that responding to the corrective RAs
will cause a collision with the threat that would not have occurred in the absence of ACAS.
This possibility is expressed through the induced component of the risk ratio.
4.2.1.2
As an example, imagine that under certain circumstances the risk ratio is quoted as being 12%.
This means that for every 100 collisions that would occur in the absence of ACAS, the use of
ACAS will result in only 12 collisions occurring. However, if the induced component of the
risk ratio is 2% this means that ACAS has resolved 90 of the original collisions, but has in
addition caused 2 that would not have occurred otherwise.
4.2.1.3
Whilst clear-cut in concept, the evaluation of the induced risk has inherent complications.
Firstly, the induced risk is particularly sensitive (more so than the total risk) to the details of
the calculation, notably: the vertical separation that is considered to constitute a collision
(100ft in the case of an NMAC); and the precise altimetry error model that is used. Secondly,
the term ‘induced collision’ is fraught with unpalatable associations. This has led to
complicated distinctions between the induced risk due to the operation of the ACAS algorithms
(‘induced by the logic’) and the induced risk due to imperfect altimetry (‘induced by altimetry
error’), with the implication that ACAS cannot be held accountable for the latter. No such
distinction is employed here.
4.2.2
ACAS – unequipped
4.2.2.1
It was reported above that, in encounters between an ACAS equipped aircraft and an
unequipped intruder, the logic risk ratio is 22.9%. Of this, the induced component accounts for
13.7ppt and the remaining 9.2ppt is due to failure to resolve pre-existing collision risk.
4.2.3
ACAS – ACAS
4.2.3.1
It was reported above that, in encounters between two ACAS equipped aircraft, the logic risk
ratio is 3.3%. Of this, the induced component accounts for 2.2ppt and the remaining 1.1ppt is
due to failure to resolve pre-existing collision risk.
4.2.4
Discussion
4.2.4.1
ACASA has already discussed the significance of ACAS induced risk, in the context of the
introduction of RVSM [32]. It has also been observed repeatedly that the risk ratio is highly
dependent on the airspace in which ACAS is deployed. A naïve but nevertheless potentially
useful model would be to regard the unresolved component of risk ratio as relatively constant
and the induced part as much more variable. In this model, the induced part would be
proportional to the rate of RAs and inversely proportional to the pre-existing risk of collision,
as discussed in [32] and [WP-1 074].
4.2.4.2
Prior to ACASA, there has been very little discussion of the absolute rate at which ACAS
might induce collision. The reason for this is that ACAS reduces the risk of collision.
However, the results presented above indicate that once aircraft are equipped with ACAS, the
combined effects of the ACAS logic and altimetry error become a significant source of
collision risk. For any other system, one would calculate the frequency with which it might
cause the loss of aircraft and require that the frequency be acceptably low.
4.2.4.3
If we consider the first aircraft to equip with ACAS and assume that all components of the
system work as specified (perfect surveillance, perfect pilot response), the result reported in
4.2.2.1 for the logic would indicate a rate of induced NMACs approximating 4 in 108 flyinghours, i.e. a rate of induced collisions approximating 4 in 109 flying-hours. Alternatively, if we
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 23
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
consider a hypothetical universal equipage, the result reported 4.2.3.1 indicates a rate of
induced NMACs approximating 7 in 109 flying-hours.
4.2.4.4
These results confirm that ACAS induced collisions should be taken seriously and all practical
steps should be taken to reduce their frequency. Anticipating later results, it is critical that as
high a proportion of aircraft as possible are equipped with ACAS, and that pilots
respond accurately and promptly to RAs, because the ACAS induced rate is least in ACASACAS encounters where both pilots respond correctly.
4.2.4.5
It is also important that any change in the collision avoidance logic to reduce the frequency
of ACAS induced NMACs must be encouraged and promoted. The expected rate of
induced NMACs (even for perfect ACAS operation) might prove to be tolerable only because
ACAS reduces the overall risk of collision.
4.3
25ft altitude data
4.3.1
Altitude reporting format
4.3.1.1
Mode C transponder equipped aircraft report their altitude in increments of 100ft. Mode S
transponder equipped aircraft, including by definition ACAS equipped aircraft, have the
potential to report their altitude more precisely in increments of 25ft.
4.3.1.2
Use of altitude data reported in the 25ft-format can result in more accurate estimation of the
altitude and (more importantly) vertical rate of intruders. This in turn can lead to the
generation of more effective and yet less disruptive RAs. This consideration motivated the
inclusion in version 7 of specific tracking algorithms to exploit altitude data in the 25ft-format
when available. Version 6.04A is unable to exploit 25ft-format altitude data – indeed, it must
degrade 25ft-format data into the corresponding 100ft-format before they can be used.
4.3.2
ACAS – unequipped
4.3.2.1
It was reported above that, in encounters between an ACAS equipped aircraft and an
unequipped intruder, the risk ratio is 22.9%. In that scenario it was assumed that the intruder
was reporting altitude in 100ft increments.
4.3.2.2
If, instead, the intruder reports altitude in 25ft increments, the risk ratio is reduced to 19.2%.
4.3.3
ACAS – ACAS
4.3.3.1
It was reported above that, in encounters between two ACAS equipped aircraft, the risk ratio is
3.3%. In that scenario both aircraft were assumed to be reporting altitude in 100ft increments.
4.3.3.2
If, instead, one of the ACAS equipped aircraft reports altitude in 100ft increments (as before)
whilst the other reports altitude in 25ft increments, the risk ratio is reduced to 3.0%.
4.3.3.3
If both ACAS equipped aircraft report altitude in 25ft increments, a further reduction in the
risk ratio to 2.8% is calculated.
4.3.4
Conclusion
4.3.4.1
The availability of altitude data measured precisely and reported in 25ft increments, as
opposed to 100ft increments, significantly improves the performance of the logic. On this
basis, aircraft equipped with Mode S transponders should be encouraged to use a digital
data source and report altitude in 25ft increments. (However, see 4.5.4.3 below.)
4.3.4.2
A warning has to accompany advice that it is desirable to report 25ft altitude data, even though
the warning does not arise from the studies reported here. Altitude data originally measured in
100ft increments must be reported using the format for 100ft data, with the Q bit17 set to
17
The Mode S data formats allow altitude data to be reported in two different formats, one for
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 24
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
indicate 100ft data. There have been instances where 100ft data have been digitised and the
format for 25ft data has been used to report the data (so that the reported altitude value always
ends in two decimal zeroes, and never with a decimal 5); this causes ACAS to use an
unsuitable altitude tracker.
4.4
The total effect of ACAS II on the risk of collision
4.4.1
Risk ratios
4.4.1.1
The preceding sections of this chapter report logic risk ratios for specific equipage scenarios.
In this section risk ratios for the full system are reported, and these have been calculated using
the event tree developed as part of the WP-1. They take into account the level of ACAS
equipage, realistic rather than idealised pilot response, the benefits of ACAS prompting visual
acquisition and the potential effect of the intervention of controllers.
4.4.1.2
The risk calculated with the event tree is reported from two different perspectives:
· the airspace risk ratio, giving the perspective of an ATS airspace regulator who decides to
mandate ACAS; and
· the aircraft risk ratio and the progressional risk ratio, giving the perspective of an airline
operator who decides to equip his fleet, or a pilot who decides to switch her ACAS on.
4.4.1.3
The risk ratios calculated with the event tree are reported for both phases of the mandate.
airspace
risk ratio
aircraft
risk ratio
progressional
risk ratio
Before Phase 1
100%
26.7%
26.7%
Between Phase 1 and Phase 2
29.9%
15.3%
27.8%
After Phase 2
24.2%
13.6%
27.7%
Table 4.1: System risk ratios
4.4.2
Discussion
4.4.2.1
Under Phase 1, the airspace risk ratio is 29.9%. Phase 2 reduces airspace risk ratio by a
further 5.7ppt to 24.2%.
4.4.2.2
From an aircraft perspective, the benefit of ACAS (aircraft risk ratio) is greater because own
aircraft has ACAS, and thus ACAS is present in all encounters.
4.4.2.3
The proportional benefit of fitting ACAS (progressional risk ratio) is notably constant – it
varies between 26.7% to 27.8% as the mandate progresses. An aircraft that equips with
ACAS at the end of Phase 2 still reduces its risk of collision by a factor of more than
three in spite of the protection afforded by ACAS on other aircraft.
4.4.2.4
The incremental benefit of Phase 2 is discussed further in chapter 7, taking into account the
effect of distinct aircraft classes as discussed in section 3.9.
25ft data and the other for 100ft data. There is a bit called the Q bit that is set to indicate which
format has been used.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 25
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
4.5
Factors affecting the efficacy of ACAS II
4.5.1
Factors investigated
4.5.1.1
It is reasonable to enquire which factors, beyond the operation of the ACAS logic, have the
greatest effect upon the total risk of collision when ACAS is deployed. Many questions of this
nature can be answered using the event tree and changing the probabilities of relevant base
level events. The factors investigated in this section fall into three categories.
· Human factors.
· Changes to the ACAS system itself (specifically, improvements in the tracking and display
of intruders).
· Changes to the equipment of second-party aircraft (which may or may not be ACAS
equipped themselves) that indirectly affect the performance of ACAS through the
information they provide to ACAS equipped aircraft they encounter.
4.5.1.2
The first of these categories includes the effect of variant pilot responses. This has been found
to be the most important single factor, other than equipage with ACAS, and thus warrants a
chapter of its own – the effect of variant pilot response is considered in chapter 6.
4.5.1.3
The second and third categories have been investigated by changing the probabilities of the
“equipment events” in the event tree – factors relating to levels of equipage with certain
equipment and its performance. The equipment changes have been investigated in terms of
‘upgrades’, i.e. changes to equipment that improve the ability of ACAS to reduce the risk of
collision.
4.5.1.4
These factors were investigated for the situation after Phase 1 but before Phase 2. Recall that
the airspace risk ratio, under Phase 1, is 29.9% and from the perspective of an ACAS equipped
aircraft is 15.3% (section 4.4.1).
4.5.2
Human factors
4.5.2.1
The most important single factor affecting the performance of ACAS is the response of pilots
to ACAS RAs. This is considered in chapter 6. In this section we consider the benefit of the
alerting functions of ACAS II.18 In an encounter, ACAS is capable of alerting the pilot to an
impending collision. Thus alerted, the pilot may be prompted to contact the controller, or be
prompted to conduct a visual search for the intruder that may or may not be aided by the traffic
display. These factors mean that ACAS may avert a collision, by virtue of its alerting
functions, before a pilot receives an RA.
4.5.2.2
If the benefits of the alerting functions are removed (neither visual acquisition nor contact with
the controller are prompted by ACAS alerts) the airspace risk ratio is 56.9%. Thus the effect
of the ACAS alerting functions is to almost halve the risk of collision that would exist
were ACAS the sole function of ACAS (56.9% is reduced to 29.9%).
18
In paragraph 2.1.5, it was reported that the one of the questions to be addressed was “What
effect on the risk of collision do the situational awareness and alerting functions of ACAS II
provide?” The situational awareness aspect of the ACAS has not been investigated: it is a
nebulous concept that is difficult to quantify and it would be unwise to claim any benefit from
ACAS purely because of improved situational awareness. Indeed, it could be argued that
situational awareness might cause pilots to take risks where they would otherwise exercise
caution.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 26
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
4.5.3
Equipment upgrade directly benefiting ACAS equipped aircraft
4.5.3.1
The effect of improving ACAS so that it could better track intruders and better display them to
the pilot was investigated. It was assumed that ACAS was always able to track transponder
equipped intruders, that bearing data was always available for these intruders and that the
traffic display was always available to the pilot.
4.5.3.2
It was found that these improvements to the ACAS system reduced the airspace risk ratio by
1.2ppt from 29.9% to 28.7%.
4.5.3.3
From the perspective of the ACAS aircraft, the aircraft risk ratio is reduced by 1.1ppt from
15.3% to 14.2%. The progressional risk ratio arising from these equipment changes (not the
decision to fit ACAS) is 93.2%.
4.5.4
Equipment upgrades indirectly benefiting ACAS equipped aircraft
Discussion
4.5.4.1
The effect of four indirect factors was investigated. In each case, equipment other than ACAS
was assumed to be upgraded (either a change to the equipage of an aircraft or an improvement
in the performance of the equipment carried). For each factor, it is possible to examine the
benefit to the aircraft that makes the change as well as the effect on the airspace risk ratio
(which might be small if only a small proportion of airspace users are involved). The benefit to
the aircraft that makes the changes results from the improved ability of ACAS equipped
aircraft to avoid it.
· If aircraft that do not currently carry a transponder equip with a Mode C transponder, the
airspace risk ratio is reduced by 0.3ppt to 29.6%. A much more dramatic benefit is evident
if we consider the situation from the perspective of the aircraft that equips: a nontransponder aircraft that equips with a Mode C transponder will reduce its risk of collision
with another aircraft to 57.0% of the risk it faced without a transponder.
· If aircraft that are currently equipped with only a Mode C transponder upgrade to Mode S,
the airspace risk ratio would be reduced by 1.3ppt to 28.6%. An aircraft making the change
would reduce its risk of collision with another aircraft to 97.1% of the risk it faced with a
Mode C transponder. This benefit arises because ACAS tracks Mode S intruders more
reliably than Mode C intruders, because Mode S aircraft are assumed to report altitude
more reliably, and because Mode S aircraft are capable of reporting altitude in 25ft
increments.
· An altitude reporting mandate would require all aircraft that are not transponder equipped
to equip with a transponder (an upgrade to Mode C rather than Mode S is assumed here),
and would further require all aircraft to report altitude. The structure of the event tree, as it
stands, is unsuited to determining the effect of such a mandate on the airspace risk ratio, but
these changes can be investigated from the perspective of the aircraft making the change.
- An aircraft that equips with a Mode C transponder and ensures that it reports altitude
will reduce its risk of collision with another aircraft to 56.6% of the risk it faced when
not transponder equipped. This benefit is greater than that reported above for a nontransponder aircraft upgrading to Mode C equipage, since in that case there was a small
probability that the Mode C equipped aircraft might not report altitude.
- A Mode C equipped aircraft that reports altitude will be exposed to only 87.3% of the
risk of collision it would face if it was Mode C equipped but did not report altitude.
- A Mode S equipped aircraft that reports altitude will be exposed to only 86.1% of the
risk of collision it would face if it was Mode S equipped but did not report altitude.
· Requiring all Mode S equipped aircraft and ACAS equipped aircraft to report altitude in
25ft increments results in a negligible improvement to the airspace risk ratio. Part of the
reason that the benefit is so slight is that the majority of Mode S equipped aircraft (be they
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 27
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
equipped with ACAS or not) already report altitude in the more precise 25ft-format. When
considered from the perspective of an aircraft that changes to reporting altitude in 25ft
increments, the greatest benefit is to Mode S aircraft that are not equipped with ACAS. A
Mode S aircraft that changes to reporting altitude in 25ft increments will reduce its risk of
collision to 99.2% of the risk it faced when reporting altitude in 100ft increments.
Conclusion
4.5.4.2
The greatest advantage to be gained by ACAS equipped aircraft from any of these changes in
absolute terms is a reduction in the risk of collision amounting to less than one collision in
109 flying-hours. In themselves, these individual advantages would seem unlikely to justify
regulatory measures to achieve the change.
4.5.4.3
This conclusion needs to be reconciled with that concerning the value of 25ft altitude data in
paragraph 4.3.4.1. The point is that most aircraft are already reporting altitude data in 25ft
increments, so that the advantage to be gained by the ACAS equipped aircraft in making it
compulsory is likely to be small.
4.5.4.4
The true beneficiary of equipping with a transponder, of replacing a Mode C transponder with
a Mode S transponder, of reporting altitude, or of reporting altitude in 25ft increments rather
than 100ft increments is the aircraft making the change. An aircraft that is not equipped
with a transponder approximately halves its risk of mid-air collision when it fits a
transponder.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 28
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
5
The effect on ACAS safety of imperfect surveillance
5.1
Introduction
5.1.1
This chapter reports the effect of imperfect surveillance on ACAS II logic. To perform this
assessment, a two-step approach was adopted. The first step consisted in specifying and
developing an ACAS II surveillance error model [WP-1 155]. The second step consisted in
calculating the impact of imperfect surveillance on the ACAS II logic risk ratio [WP-1 163].
The main factors affecting the performance of the logic were also investigated during the
second step.
5.2
Tools and methods
5.2.1
Monte Carlo approach
5.2.1.1
As presented in Figure 5.1, the approach [WP-1 142] relies on using ‘Monte Carlo’
simulations. It consists in conducting a very large number of simulations and modifying for
each simulation the initial encounters in order to take into account the uncertainties of
surveillance data.
Set of N initial
encounters
initial
encounter 1
Sample 1
ACAS
logic
Sample 2
ACAS
logic
.
.
.
Sample M
ACAS
logic
Variations into
surveillance data
N.M simulations
initial
encounter 2
.
.
.
initial
encounter N
Figure 5.1: Monte Carlo simulations
5.2.1.2
Due to computer constraints, the number of initial encounters is restricted to N = 10,732. For
each initial encounter, the number of variations into surveillance data is M = 50.
5.2.1.3
When no surveillance error model is introduced, this set of 10,732 encounters includes the
encounters (with vmd < 500ft) that have most effect on the risk ratio calculation. It has been
demonstrated that the risk ratio computed on this basis is similar to the risk ratio computed on
a larger set of encounters (e.g. 100,000 encounters).
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 29
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
5.2.2
Risk ratio comparison
5.2.2.1
The effect of imperfect surveillance is assessed by comparing the risk ratio computed without
surveillance error against the risk ratio computed when the surveillance error model is
introduced into the simulations.
5.2.2.2
The sensitivity of ACAS II performance to the main surveillance parameters is also assessed
by introducing the elements of imperfect surveillance separately.
5.2.3
Simulation hypotheses
Equipment model
5.2.3.1
One equipage scenario is studied, consisting of 100% aircraft equipped with version 7 (and
therefore with Mode S transponders). In this scenario the aircraft report their altitude in 25ft
increments. They provide the ACAS II logic with own altitude data in 1ft increments.
5.2.3.2
This choice comes from the indications that more than 90% of the European fleet is Mode S
equipped, with 90% reporting their altitude in 25ft increments [44] and because of the need to
contain the problem.
Encounter model
5.2.3.3
The study is based on the use of the European ACAS safety encounter model [WP-1 186]
developed within the framework of WP-1.1.
Pilot model
5.2.3.4
The simulations assume a standard pilot reaction. This means that the pilots follow correctly
their RAs in accordance with the ACAS SARPs [42].
5.2.4
Surveillance error model
5.2.4.1
The surveillance error model considers two states for a Mode S reply:
·
an available reply; and
·
a missing reply.
5.2.4.2
When ACAS II does not get a Mode S reply, range and altitude data are not available. When
ACAS II gets a Mode S reply, a range measurement is possible, but the measure can be
inaccurate. In addition, the altitude can be missing despite the availability of the Mode S reply.
5.2.4.3
When a Mode S reply is available, two states are considered for the range measurement:
·
inaccurate range measure (i.e. range gross error); and
·
range measure with no error.
5.2.4.4
As defined in [WP-1 142], the range gross error is composed of a small error x(n) and a bias b.
The bias is assumed constant during all the time of the gross error state. This bias is drawn
randomly according to a uniform distribution between plus and minus the current intruder
range limited to a maximum value. The purpose of the small error x(n), which is simulated by
an auto-regressive process normally distributed with zero mean and a standard deviation of
114ft, is to introduce an auto-correlation within the range gross error model.
5.2.4.5
When a Mode S reply is available, two states are considered for the altitude data:
·
altitude data not available; and
·
altitude data with no error.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 30
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
5.2.4.6
The surveillance error model focuses on the availability of altitude reports because constant
and gross errors for altitude data are already included in the risk ratio calculation through the
use of the ICAO standard altimetry error model. In addition, the surveillance error model does
not encompass bearing aspects because of the low significance of the bearing data from a risk
ratio perspective [WP-1 081]. Indeed, the bearing data are only used by the ACAS II logic for a
miss distance filtering purpose and then only as a crosscheck for the acceleration test (based on
range measurement).
5.2.4.7
Figure 5.2 summarises this simplified surveillance error model.
Mode S reply?
Yes
Range error?
Yes
No
Missing range
Missing altitude
No
Altitude availability?
No range error
Range with gross error
No
Yes
No altitude error
Missing altitude
Figure 5.2: Simplified surveillance error model
5.2.5
Scenarios
5.2.5.1
Six different scenarios are used in order to, firstly, assess the impact of imperfect surveillance
performance on the efficacy of ACAS II and, secondly, in order to separately evaluate the
influence of different factors:
· the availability of Mode S replies;
· the availability of altitude data; and
· the availability of accurate range data, and the effect of varying the magnitude of the range
errors.
5.2.5.2
The six scenarios are defined as follows:
· Scenario Z: no surveillance errors are considered;
· Scenario A: the only failure is unavailability of altitude data; altitude data are available at a
rate of 89%;
· Scenario B1: the only failure is the occurrence of range errors; range data without error are
available at a rate of 89%; the error extent is limited to small range errors (i.e. ±352ft for
99.8% of the time);
· Scenario B2: the only failure is the occurrence of range errors; range data without error are
available at a rate of 89%; the error extent is significantly increased with a maximum bias
equal to ±1NM;
· Scenario C: the only failure is unavailability of Mode S replies; Mode S replies are
available at a rate of 78%; and
· Scenario D: surveillance errors are considered; the parameters included in scenarios A, B1
and C are simultaneously considered.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 31
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
5.2.5.3
The comparison between scenario Z and scenario D shows the impact of imperfect surveillance
performance on the efficacy of ACAS II. The comparisons between scenario Z and scenarios
A, B1 and C show the influence of each single factor. The comparison between scenario B1
and scenario B2 shows the impact of large errors in the range data.
5.3
Robustness of the ACAS II logic to imperfect surveillance
5.3.1
General
5.3.1.1
Table 5.1 presents the risk ratio computed for each proposed scenario. In scenario D, the
indicated probability of the availability of altitude data (or range data without error) is when a
Mode S reply is available. The actual overall availability is 69% (i.e. 78%´89%) for both the
altitude data and the range data without error.
5.3.1.2
The proportion of split RAs19 is also provided. This indicator enables the credibility of certain
scenarios to be verified.
Type of error simulated
No errors Missing
altitude
Small
range
errors
Large
range
errors
Missing Combined
Mode S
errors
replies
Scenario
Scen. Z
Scen. A
Scen. B1
Scen. B2
Scen. C
Scen. D
Avail. of Mode S reply
100%
100%
100%
100%
78%
78%
Avail. of altitude data
100%
89%
100%
100%
100%
89%
Avail. of range without error
Maximum bias
Small errors (in 99.8%)
100%
0ft
0ft
100%
0ft
0ft
89%
0ft
±352ft
89%
±6077ft
±352ft
100%
0ft
0ft
89%
0ft
±352ft
Risk ratios
2.3%
2.4%
2.4%
4.8%
2.9%
2.9%
Split RAs
0.3%
0.3%
0.4%
23.0%
0.3%
0.2%
Table 5.1: Risk ratios vs. scenarios
5.3.2
Influence of missing surveillance data (scenario Z vs. scenarios A, B1 and C)
5.3.2.1
The results indicate that neither the availability of altitude data nor the availability of range
data without error (when only considering small range errors) affects ACAS II performance
seriously (+0.1ppt). Also, the absence of Mode S replies does not induce a significant increase
in the risk ratios (+0.6ppt) despite a low probability of availability (78%). The availability of
Mode S reply seems to be the most significant factor when compared with the availability of
altitude data alone or the availability of range data alone.
5.3.3
Influence of large errors in the range measurements (scenario B1 vs. scenario B2)
5.3.3.1
The results show that the introduction of large errors (up to ±1NM) in the range measurements
degrades (+2.5ppt) the safety performance of ACAS II. However, the extent of the range error
required for observing such a result seems implausible when its effect is assessed from an
operational point of view, because of the proportion of split RAs (23.0%) induced by
scenario B2 is high. The frequency of split RAs should be extremely low for encounters
19
A ‘split RA’ is in fact two separate, sequential RAs against the same intruder in a single
encounter (i.e. a “Clear of conflict” indication is issued in the initial RA whilst the range is still
decreasing). The second alert is issued without regard to the sense nor strength of the first
alert. The split can occur as the result of imperfect surveillance or, in the case of version 7, as a
consequence of the operation of the miss distance filtering logic.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 32
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
represented by the European ACAS safety encounter model, because the main mechanism
causing split RAs (the ‘miss distance filter’, which suppresses RAs when hmd is large) should
not operate in these encounters, for which hmd < 500ft.
5.3.4
Influence of the global surveillance error model (scenario Z vs. scenario D)
5.3.4.1
The results clearly show that an imperfect surveillance involving various influencing factors is
not significantly degrading (+0.6ppt) the safety performance of ACAS II.
5.4
Conclusion
5.4.1
The various simulations conducted at CENA introducing different surveillance errors indicate
an encouraging robustness in the ACAS II logic. Except when improbably large errors in the
range measurements are introduced, the efficacy of ACAS II is not significantly altered.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 33
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
6
The effect on ACAS safety of imperfect pilot response
6.1
Methods and data sources
6.1.1
This chapter reports the implications of non-standard pilot reactions for ACAS II safety
performance.
6.1.2
A set of airborne TCAS II recorded data provided by EEC has been analysed to identify actual
pilot reactions to real RAs. These recordings come from dedicated TCAS recorders that EEC
provided to some airlines in the early 1990s, within the framework of the European ACAS II
operational evaluation. They include a total of 61 exploitable RAs, recorded from 4 airlines.
6.1.3
For the purpose of this analysis, the most relevant parameters provided by the recordings are
time, altitude (own and intruder), vertical speed (own and intruder), intruder ACAS status, and
Resolution Advisory. As some parameters are missing, which would have helped to
characterise pilot reactions more accurately (vertical acceleration and auto-pilot modes), an
approximate vertical acceleration is computed based on the vertical speed variations.
6.1.4
It should be noted that the altitude and vertical speed parameters are those tracked by ACAS.
They can differ from the real ones and therefore the computed vertical acceleration too. Indeed,
the analysis of pilot reaction based on ACAS tracked parameters may lead to an overestimation
of the time to react and to an underestimation of the vertical acceleration [WP-1 161].
6.2
Actual pilot behaviour
6.2.1
Preventive RAs
6.2.1.1
5 out of the 61 RAs are preventive and have not been followed by any corrective RAs. Pilots
did not react to any of these preventive RAs. In a sixth event, a preventive RA was generated
on board a level aircraft, and after 5 seconds it strengthened to a corrective RA. The pilot
reacted to the RA only when it became corrective. This event has been included in the set of
corrective RAs.
6.2.1.2
Even though based on the limited evidence of 6 cases, it appears that pilots do not deviate
from their vertical flight path in response to preventive RAs.
6.2.2
Corrective RAs without pilot reactions
6.2.2.1
The analysis deals with a set of 56 corrective RAs.
6.2.2.2
An initial analysis of the airborne data highlights the low number of pilots that respond to
corrective RAs. Table 6.1 provides the total number of reactions as well as the number for each
airline.
Airline 1
Airline 2
Airline 3
Airline 4
Total
Number of RAs
17
7
23
9
56
Reaction
11
0
7
7
25
No reaction
6
7
16
2
31
Rate of reaction
65%
0%
30%
72%
45%
Table 6.1: Number of RAs and of pilot reactions
6.2.2.3
Table 6.1 clearly shows some significant differences between airlines. Two of them provide
unacceptable figures, with very low (and even nil!) rates of reaction. This is a concern. These
differences may come from the ACAS II training that the airlines have provided to their pilots.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 34
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
6.2.2.4
As a whole, a reaction has been detected in only 45% of the RAs. In other words, more than
half of the corrective RAs have not been followed by pilots.
6.2.2.5
Two factors that might have been expected to increase the chances of a non-reaction have not
been confirmed by the analysis:
· the rate of reaction to RAs during level-off encounters is slightly greater than the average,
despite the fact that they are sometimes considered nuisance alerts;
· positive (i.e. “Climb” or “Descend”) and negative (i.e. reduction of vertical rate) RAs are
followed in the same proportion, despite the fact that positive RAs have a greater impact on
the aircraft trajectory.
6.2.2.6
Nevertheless, some possible influencing factors have been identified:
· the RAs generated below FL50 and above FL250 are followed less frequently than those
between FL50 and FL250 (32% vs. 57%);
· pilots of level aircraft seem to react less often to corrective RAs than those that are either
climbing or descending (35% vs. 49%);
· the lack of a TA before the RA contributes to a lower rate of reaction to the RAs as the
crew is unprepared.
6.2.2.7
Some other possible influencing factors could not be addressed in this study, which is based
solely on airborne recorded data. They include, for instance, visual acquisition of the intruder,
the provision of traffic information by the controller and the level of pilot training on ACAS II.
6.2.3
Pilot reactions to RAs
6.2.3.1
The detailed analysis of pilot reactions is based on the 25 RAs for which the pilots have
initiated a manoeuvre. In addition, three other RAs were processed in which the pilot did not
react to the initial corrective RAs, but reacted to a change in the RA.
6.2.3.2
15 out of the 25 initial RAs are positive RAs, the 10 other ones being “Reduce rate” RAs.
6.2.3.3
Among the 25 initial RAs that have been followed, only 2 subsequent corrective RAs have
been generated. In both cases, both the initial RA and the subsequent RA were “Reduce rate”,
and the change was only to strengthen the vertical speed restrictions.
6.2.3.4
All the 15 positive RAs followed by pilots turned into either a weakening RA (13 instances) or
Clear of Conflict (2 instances).
6.2.3.5
Table 6.2 provides statistical results on the pilot manoeuvres in reaction to the RAs.
Positive RAs
Negative RAs
All RAs
Delay
Vertical
acceleration
Vertical speed
7.1s
0.18g
1987fpm
2.9s
0.09g
1727fpm
Average
6.0s
0.12g
-
Standard deviation
2.6s
0.08g
-
Average
6.7s
0.16g
-
Standard deviation
2.8s
0.09g
-
Average
Standard
deviation20
Table 6.2: Statistical results on the pilot manoeuvres in reaction to the RAs
20
Standard ‘deviation’ refers to the mathematical statistic, and is not related to any vertical
deviation caused by the RAs.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 35
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
6.2.3.6
The figures computed on the 25 reactions show that pilots tend to react with a greater delay
than the standard (6.7s vice 5.0s) and with a lower vertical acceleration (0.16g vice 0.25g). The
standard deviations indicate that these figures are only averages and that there are some
discrepancies between the reactions.
6.2.3.7
The comparison between positive and negative RAs shows that the delay of reaction is slightly
longer for positive RAs (7.1s) than for negative ones (6.0s). Moreover, there are a few
important delays of reaction to positive RAs (from 10s to 14s), whereas the breakdown for
negative RAs is well distributed between 2s and 9s.
6.2.3.8
On the other hand, the acceleration for positive RAs is greater than for negative RAs (0.18g vs.
0.12g). The reactions to positive RAs have been performed with accelerations centred on the
range 0.10g–0.19g, with two accelerations greater than the standard (0.30g and 0.40g).
6.2.3.9
Based on the available data set, it can be concluded that pilots wait a little longer before
reacting to a positive RA, but they react with a greater vertical acceleration than to a
negative RA.
6.2.3.10 For the positive RAs, the average of the vertical speed adopted during the reaction is good
(1987fpm vs. 1500fpm–2000fpm). Nevertheless, a detailed study of the vertical speed
breakdowns clearly indicates that about half of the pilots have adopted a vertical speed lower
than 1000fpm whereas the other half have overreacted, with vertical speeds over 2500fpm. No
real correlation could be established between the vertical speed achieved by pilots and the
vertical tendency of the aircraft when the RA was generated.
6.2.3.11 For the negative RAs, 7 pilots out of 10 have complied with the vertical speed reduction. 2 out
of the 3 insufficient manoeuvres correspond to slight under-reaction. On the other hand some,
strong reactions have also been noticed. For instance, a pilot whose aircraft was climbing at
4500fpm received a “Reduce climb to 1000fpm” and initiated a descent manoeuvre at
1200fpm; he could have kept on climbing.
6.2.4
Specification of the pilot models based on the airborne data
Introduction
6.2.4.1
Having identified the actual pilot reactions to real RAs, the second step, presented in this
section, consists in the development of pilot models based on the results reported in the
previous section.
Description of the pilot models
6.2.4.2
The standard pilot reaction to RAs is defined in SICASP/6-WP/44 as follows:
Type of RA
Delay
Acceleration
Initial corrective RAs
5s
0.25g
Increase rate RAs
2.5s
0.35g
RA reversals
2.5s
0.35g
Weakening RAs
2.5s
0.25g
Strengthening RAs21
2.5s
0.25g
Table 6.3: Standard pilot reaction to RAs
21
“Strengthening RAs” means a subsequent RA, which revises the intensity of the preceding
RA by either increasing the vertical speed limitation of a negative RA (e.g. “Reduce descent to
500fpm” after a preliminary “Reduce descent to 1000fpm”) or by generating a positive RA
after a negative RA (e.g. “Climb” after a preliminary “Reduce descent to 1000fpm”).
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 36
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
6.2.4.3
In addition the vertical speeds adopted by the pilot in response to RAs are:
· positive RAs: ±1500fpm;
· increase rate RAs: ±2500fpm;
· vertical speed limit RAs: as appropriate (i.e. 0fpm, 500fpm, 1000fpm or 2000fpm).
Determination of the pilot model parameters
6.2.4.4
The various pilot models will be specified through three parameters:
· the delay to react;
· the vertical acceleration;
· the vertical speed.
6.2.4.5
The 15 positive RAs followed by the pilots have been analysed to determine whether a
correlation can be identified between the parameters related to the reactions.
6.2.4.6
Two groups of vertical speeds have been clearly identified (< 1000fpm or > 2500fpm).
Assuming that the vertical speed is a discriminating factor, two types of pilot models can be
envisaged: a slow one and an aggressive one.
6.2.4.7
Figure 6.1 and Figure 6.2 present consecutively the delay and the vertical acceleration
associated to these vertical speeds.
16
14
Slow pilot
12
10
8
Aggressive pilot
6
4
Standard pilot
2
0
0
1000
2000
3000
4000
5000
6000
Figure 6.1: Delay (s) vs. vertical speed (fpm)
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 37
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
0.45
0.4
0.35
Standard pilot
0.3
0.25
0.2
Aggressive pilot
0.15
0.1
Slow pilot
0.05
0
0
1000
2000
3000
4000
5000
6000
Figure 6.2: Vertical acceleration (g) vs. vertical speed (fpm)
6.2.4.8
These Figures clearly show that:
· the positive RAs that have resulted in a low vertical speed have been followed with a delay
greater than or equal to 7s, whereas the RAs with a high vertical speed (> 2500fpm) have
been followed with a delay less than or equal to 6s (except in one case);
· the positive RAs that have resulted in a low vertical speed have been followed with an
acceleration lesser than or equal to than 0.15g, whereas the RAs with a high vertical speed
have been followed with an acceleration greater or equal than 0.19g;
· None of the reactions matches a standard reaction.
6.2.4.9
Therefore, two pilot models can be defined (with the rounded average values in parentheses):
· a slow pilot model that follows RAs with a too low vertical speed (500fpm), vertical
acceleration (0.1g) and with a too high delay (9s);
· an aggressive pilot model that follows RAs with a good delay (5s) and vertical acceleration
(0.25g) but with a too high vertical speed (3700fpm).
6.2.5
Conclusion
6.2.5.1
This analysis highlights some interesting results on pilot responses to RAs. Some of them were
suspected and they have been either confirmed or amplified by the analysis of the airborne
recorded data.
6.2.5.2
The main issue is that only 45% of the pilots have reacted to the corrective RAs. Some
possible influencing factors (i.e. altitude, vertical tendency and TA generation) have been
identified.
6.2.5.3
Some additional issues have been identified:
· several pilots even ignored two successive corrective RAs;
· all the crossing, reverse and increase rate RAs were ignored by the pilots;
· a few inappropriate reactions were detected (e.g. reduction of descent rate in response to a
“Descend” RA).
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 38
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
6.2.5.4
When the RAs have been followed, statistically the pilots have reacted slower (6.7s) and less
strongly (0.16g) than expected by ACAS, with some little differences between positive and
negative RAs. In response to the “Climb” and “Descend” RAs, the adopted vertical speeds
were either too low (< 1000fpm) or too high (> 2500fpm).
6.2.5.5
It can also be noticed that the pilots did not follow the RAs with a constant vertical speed like
the ACAS II standard pilot. The acceleration is progressive and lasts longer.
6.2.5.6
Two new pilot models could be developed based on this analysis:
·
·
a slow pilot model (over-relaxed in terms of delay, vertical speed and vertical
acceleration);
an aggressive pilot model (aggressive in terms of vertical speed).
6.3
The effect of non-standard behaviour on the logic
6.3.1
ACAS II simulations
Simulation hypotheses
6.3.1.1
The calculation of risk ratios with these new pilot models is reported in this section.
6.3.1.2
The risk ratios are computed using the European ACAS safety encounter.
6.3.1.3
The hypothesis for aircraft to be ACAS equipped or not is based on Phase 2 of the European
ACAS mandate. Classes C, D, E and F of the European encounter model are fitted with
ACAS II.
6.3.1.4
ACAS equipped aircraft are also fitted with an Air Data Computer (i.e. own altitude data are
provided in 1ft increments to the logic) and report their altitude with 25ft increments.
6.3.1.5
Aircraft not equipped with ACAS report their altitude with either 25ft (80%) or 100ft (20%)
increments.
6.3.1.6
The risk ratios will be computed on the two new pilot models as well as on the standard pilot
model to compare the results.
6.3.1.7
The three pilot models used for the study are described in Table 6.4:
Slow
Standard Aggressive
Initial corrective RA delay
9s
5s
5s
Other RA delay22
2.5s
2.5s
2.5s
Standard RA acceleration23
0.10g
0.25g
0.25g
Increase/Reversal RA acceleration
0.10g
0.35g
0.25g
Positive RA rate
500fpm
1500fpm
3700fpm
Increase RA rate
500fpm
2500fpm
3700fpm
Vertical speed limit
as requested by RAs
Table 6.4: Pilot models
22
Other RAs include weakening, strengthening, increase and reverse RAs.
23
Standard RAs include initial, strengthening and weakening RAs.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 39
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
Risk ratio calculation
6.3.1.8
Table 6.5 presents the risk ratios computed with the three pilot models and the contribution of
unresolved and induced NMAC for the various risk ratios.
Pilot model
Slow
Standard Aggressive
Unresolved NMACs
29.6%
5.3%
5.2%
Induced NMACs
33.7%
3.7%
1.1%
Risk ratio
63.3%
9.0%
6.3%
Table 6.5: Risk ratio vs. pilot models
6.3.1.9
These figures clearly show that the slow pilot degrades ACAS safety performance drastically
(63.3% vice 9.0%). On the other hand, the aggressive pilot improves the safety performance
slightly (6.3% vice 9.0%).
6.3.1.10 The decrease of the ACAS safety performance by the slow pilot model is due to an increase of
both types of NMAC but particularly of induced NMACs.
6.3.1.11 On the other hand, the improvement noticed with the aggressive pilot model is due to a
reduction in induced NMACs, the number of unresolved NMACs being similar.
Deviations
6.3.1.12 If, on one hand, the aggressive pilot model decreases slightly the risk ratio, on the other hand it
induces much larger deviations (deviation average increases from 189ft to 453ft), and therefore
it decreases the ACAS compatibility with ATC.
6.3.1.13 There is a huge increase of the number of deviations greater than 600ft with the aggressive
pilot model compared with the standard pilot model (229 vs. 2799, i.e. +1122%!). With the
standard pilot model, only 0.5% of the RAs induce a deviation greater than 600ft whereas, with
the aggressive pilot model, this ratio is 6.5%.
6.3.1.14 These figures have been computed with the hypotheses that weakening RAs are correctly
followed by pilots. However, the actual deviation could be even greater with the aggressive
pilot because it is not likely that aggressive pilots, who overreact to the initial RAs, will follow
the weakening RAs.
6.3.2
Conclusion
6.3.2.1
The slow pilot model raises a major safety issue. The risk ratio is seven times greater
than with a standard pilot (63.3% vs. 9.0%). This result is due to a high increase of
unresolved NMACs and even more of induced NMACs.
6.3.2.2
The aggressive pilot model provides slightly greater safety performance (6.3% vs. 9.0%). The
main issue with this pilot model is the associated high vertical deviations. The two main
consequences are an incompatibility with ATC and a risk of induced RAs with a third aircraft.
6.3.2.3
The standard pilot model is the best compromise. The safety performance achieved with a
standard pilot reaction is good, and it minimises the disruption for ATC controllers and the risk
of conflict with a third aircraft. This result emphasises the need for pilots to react correctly to
RAs.
6.3.2.4
As a consequence, pilot training is a crucial factor in achieving a successful
implementation of ACAS II. This training should clearly explain why an appropriate reaction
to RAs is necessary. The ACASA working papers [WP-1 161] and [WP-1 171] could usefully
be disseminated in the pilot arena to this end.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 40
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
6.3.2.5
All these results clearly highlight the need to emphasise the pilot training on ACAS II. The
discrepancies between airlines in terms of reaction rate confirm that the training was not
addressed correctly by all of them.
6.3.2.6
The trends presented in this section were confirmed using recent US data [45].
6.4
Total system performance with real pilots
6.4.1
Pilot behaviour
6.4.1.1
One of the simplest pieces of advice to give to pilots, in order to improve the efficacy of
ACAS, is to remind them never to ignore an RA. Further improvements can be obtained by
training pilots to respond to RAs accurately. The benefits, within the context of the full system
performance, can be quantified by making suitable assumptions to the appropriate event
probabilities and exercising the event tree. The undesirable consequences of poor behaviour
can also be emphasised in a similar manner.
6.4.1.2
A number of particular pilot behaviours have been investigated (in this context ‘behaviour’ is
the combination of whether a pilot responds to an RA and the manner in which she responds
when she does).
· typical pilot – the pilot might not respond to an RA (either because she ignores it, because
she has visual acquisition, or because she elects to follow a controller instruction rather
than an RA) and when she follows an RA her response may be imperfect. This is the default
behaviour that is assumed when reporting full system performance figures in section 4.4.
· always notes RAs – the pilot never ignores an RA (though she might elect to follow a
controller instruction rather than the RA, and does not follow RAs when she has visual
acquisition) and responds to the RA as her peers do.
· always follows RAs – not only does the pilot always note RAs but she also always prefers
them to controller instructions. If there is an RA the pilot will respond to it, although her
response will be the same as that of her peers.
· ideal response – in addition to always responding to RAs the pilot always responds to them
accurately.
· always follows RAs, but slowly – the pilot always follows RAs but her response is slow.
· ignores RAs – the pilot never responds to RAs. The only benefit she can get from ACAS is
from situational awareness and the alerting function of ACAS.
6.4.2
Risk ratios
6.4.2.1
The consequence of each of the behaviours described above has been evaluated for two
different scenarios:
· from the airspace perspective when all pilots exhibit the particular behaviour;
· from an individual pilot’s perspective when she exhibits the particular behaviour but all
other pilots behave typically.
6.4.2.2
The results for the current stage of the mandate (i.e. after Phase 1 but before Phase 2) are
shown in Table 6.6 – qualitatively similar results are obtained for Phase 2. The progressional
risk ratio for a pilot exhibiting each particular behaviour is discussed below (in the same order
as they are described in 6.4.1.2).
6.4.2.3
A pilot that switches ACAS on and then responds in the same way as her peers reduces her risk
of collision to 27.8% of the risk immediately before turning ACAS on – the progressional risk
ratio reported in 4.4.1.3.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 41
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
6.4.2.4
A pilot can reduce her risk of collision by systematically responding to RAs. Even though she
might elect to follow a controller instruction rather than an RA, does not follow RAs when she
has visual acquisition, and responds to the RA imperfectly (as her peers do), she will face a
risk of collision that is 24.5% of the risk faced by the pilots of unequipped aircraft.
6.4.2.5
If the pilot further improves her behaviour by always preferring to follow an RA rather than a
controller instruction, she will further reduce her risk of collision to 23.7% of the risk faced by
the pilots of unequipped aircraft.
6.4.2.6
Best of all is not merely to follow RAs, always, but to do so accurately. Should a pilot who
always responds to RAs further ensure that her response is always that required by ACAS, she
will reduce her risk of collision further still to 13.2% of that faced by the pilots of unequipped
aircraft. Such an ideal pilot faces less than half the risk of collision faced by typical pilots of
ACAS equipped aircraft.
6.4.2.7
A pilot who always follows RAs but always does so slowly will face a risk of collision that is
36.8% of that faced by the pilots of unequipped aircraft. Thus, although always responding,
she does not achieve the same level of risk reduction as her peers who might not respond to
RAs, and might respond slowly, but who respond promptly some of the time. It is critical to
follow RAs accurately.
6.4.2.8
A pilot who never follows ACAS RAs faces a risk of collision that is 45.8% of that faced by
the pilots of unequipped aircraft. Thus she faces more than one-and-a-half times the risk faced
by typical pilots, and more than three times the risk she would face if she always followed RAs
and followed them accurately.
airspace risk ratio
– when all pilots exhibit the
particular behaviour
(induced component in
brackets)
progressional risk ratio
– for a pilot who
exhibits the particular
behaviour whilst other
pilots behave typically
38.0% (0%)
45.8%
always follows RAs, but slowly
34.1% (5.4%)
36.8%
typical
29.9% (4.5%)
27.8%
always notes RAs and if responding
responds typically
28.4% (4.6%)
24.5%
always follows RAs and typically
28.1% (4.6%)
23.7%
ideal
22.6% (3.8%)
13.2%
behaviour
ignores RAs
Table 6.6: Risk ratios vs. pilot behaviour
6.4.3
Conclusion
6.4.3.1
When all pilots note RAs, the airspace risk ratio is improved to from 29.9% to 28.4%.
6.4.3.2
The effect is more significant from the aircraft point of view. The aircraft risk ratio is
improved from 27.8% to 24.5%: i.e. a pilot who never ignores RAs is exposed to only 88% of
the risk that ‘average’ pilots experience.
6.4.3.3
The risk is reduced even further when a pilot ensures that she always follows RAs and follows
them accurately: by exhibiting such ideal behaviour she can reduce her risk to less than
half that faced by typical pilots of ACAS equipped aircraft.
6.4.3.4
The consequence of behaviour in the other extreme have also be evaluated. A pilot who
always ignores RAs exposes herself, and the unwitting pilots of the aircraft she encounters,
to a risk of collision more than one-and-a-half times that faced by average pilots.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 42
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
7
The benefits of Phase 2 of the ACAS mandate
7.1
Introduction
7.1.1
This chapter provides an assessment of the maximum benefit to be obtained by fitting ACAS II
to small aircraft, i.e. those aircraft covered by Phase 2 of the ACAS mandate. These aircraft
have a maximum take-off mass exceeding 5700kg or carry more than 19 passengers, but have a
maximum take-off mass less than 15,000kg and carry less than 31 passengers. The work
included a review of the performance limitations of aircraft and their implications for the
suitability of equipping these aircraft with ACAS II.
7.1.2
This assessment is achieved using the usual airspace risk ratios and the aircraft risk ratios. This
approach enables answers to the following questions to be given:
· What benefit do small aircraft gain in terms of reduced risk of collision from fitting ACAS
in an environment where large aircraft are already equipped?
· What additional benefit do large aircraft, which are already equipped with ACAS, gain
when small aircraft are also equipped?
7.1.3
Sections 7.2 and 7.3 examine the benefits of fitting ACAS to small aircraft by calculating logic
risk ratio, using the concept of aircraft type introduced in section 3.9. This work is described in
more detail in two reports [WP-1 172] [WP-1 204]. Section 7.4 addresses the issue by
considering the performance of ACAS as a whole.
7.2
Methods and data sources
7.2.1
Aircraft performance classes
7.2.1.1
Aircraft performance limitations were simulated. Following [WP-1 145], this was done by
limiting the verticals speed that could be achieved in response to climb-sense RAs to suitable
values less than the nominal values (which are 1500fpm for “Climb” RAs and 2500fpm for
“Increase climb” RAs).
7.2.1.2
These aircraft performance limits vary with the type of aircraft. The mandatory carriages that
were simulated (Phases 1 and 2) also apply on different types of aircraft. For these reasons,
classes of aircraft were used. These classes are the 7 classes defined for the European ACAS
safety encounter model [WP-1 186]:
· Class A: piston aircraft with a maximum take-off mass below 5700kg;
· Class B: turboprop aircraft with a maximum take-off mass below 5700kg;
· Class C: turboprop aircraft with a maximum take-off mass between 5700kg and 15,000kg
(class subject to Phase 2);
· Class D: jet aircraft with a maximum take-off mass between 5700kg and 15,000kg (class
subject to Phase 2);
· Class E: turboprop with a maximum take-off mass in excess of 15,000kg (class subject to
Phase 1);
· Class F: jet aircraft with a maximum take-off mass in excess of 15,000kg (class subject to
Phase 1);
· Class G: high performance (military) jets.
7.2.1.3
Only classes C, D, E and F have been fitted with ACAS II during the simulations as the other
classes are not subject to any mandatory carriage. Consequently, performance limitations were
defined for these classes.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 43
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
7.2.1.4
Table 7.1 shows the vertical rate limitations that were used for the classes subject to Phase 1
and Phase 2. They depend on altitude, and so were treated as functions of the altitude layers
defined for the European ACAS safety encounter model [WP-1 186].
Class
Layer
C
Turboprop
(Phase 2)
D
Jet aircraft
(Phase 2)
E
Turboprop
(Phase 1)
F
Jet aircraft
(Phase 1)
1:
1000ft-FL50
2046fpm
3204fpm
1934fpm
2794fpm
2:
FL50-FL135
1673fpm
3472fpm
1555fpm
3002fpm
3:
FL135-FL215
1301fpm
2894fpm
1191fpm
2606fpm
4:
FL215-FL295
907fpm
2117fpm
741fpm
1876fpm
5:
FL295-FL415
0
1140fpm
0
1208fpm
Table 7.1: Vertical rate limitations per layer and per aircraft performance class
weighted mean values
7.2.1.5
The vertical rates of classes C and E in layer 5 are set to zero as turboprops usually do not go
over FL295. These classes of aircraft are represented in very small proportion in layer 5 (2.5%
for class E, 0.0% for class C).
7.2.1.6
The performance limitations have no effect when the vertical rate limitation is greater than
2500fpm (i.e. bold values in Table 7.1).
7.2.1.7
A climbing aircraft that receives a “Maintain climb” RA does not decrease its vertical rate to
comply with the appropriate value of Table 7.1, but goes on with its current vertical rate.
7.2.1.8
It will be seen that the performance limitations of aircraft covered by Phase 2 are,
generally speaking, no worse (from an ACAS performance standpoint) than those of the
aircraft covered by Phase 1.
7.2.2
Scenarios
7.2.2.1
All the simulations were performed using the European ACAS safety encounter model.
7.2.2.2
The version of the collision avoidance logic used for the simulations is version 7.
7.2.2.3
Standard pilot reactions were simulated for pilots following their RAs.
7.2.2.4
Five scenarios were used in order to enable a fair assessment of the benefits to be obtained by
fitting ACAS II to small aircraft:
· Scenario 0: all aircraft equipped with ACAS II. No performance limitations were taken into
account;
· Scenario 1: only aircraft of classes E and F equipped with ACAS II (Phase 1). No
performance limitations were taken into account;
· Scenario 2: only aircraft of classes E and F equipped with ACAS II (Phase 1). Performance
limitations were taken into account;
· Scenario 3: only aircraft of classes C, D, E and F equipped with ACAS II (Phase 2). No
performance limitations were taken into account;
· Scenario 4: only aircraft of classes C, D, E and F equipped with ACAS II (Phase 2).
Performance limitations were taken into account.
7.2.2.5
All the aircraft potentially equipped with ACAS (i.e. classes C, D, E, and F) were simulated
reporting their altitude in 25ft increments. The proportion of aircraft reporting their altitude in
25ft increments within classes A, B and G (i.e. unequipped classes) was chosen to be 80%. The
remaining 20% were reporting their altitude in 100ft increments. The proportion of aircraft
reporting their altitude in 25ft increments was chosen on the basis of the results presented in [44].
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 44
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
7.2.2.6
The comparison between scenario 0 and scenario 1 shows the impact of including a proportion
of non-equipped aircraft. Scenario 0 provides information about a theoretical full equipage.
Scenario 1 is closer to the operational reality.
7.2.2.7
The comparisons between scenarios 1 and 2 and between scenarios 3 and 4 show the impact of
including performance limitations in the simulations.
7.2.2.8
The comparisons between scenarios 1 and 3 and scenarios 2 and 4 show the impact of fitting
ACAS to small aircraft.
7.2.2.9
Scenarios 2 and 4 were also simulated with a proportion of pilots not following their RAs
ranging from 0% to 100%, in steps of 20%. These scenarios were used to compute the aircraft
risk ratios, showing the benefit to the individual aircraft of fitting ACAS. These risk ratios are
computed for a given class of aircraft rather than for a given airspace.
7.2.3
Aircraft class distribution
7.2.3.1
Figure 7.1 shows the proportion of aircraft classes included in the European ACAS safety
encounter model [WP-1 186].
60
2005 mandatory carriage
2000 mandatory carriage
50
Proportion (%)
40
30
20
10
0
Class A
Class B
light
Class C
Class D
Class E
small
Class F
large
Class G
mil
Figure 7.1: Aircraft class distribution
7.2.3.2
The aircraft subject to Phase 1 (classes E and F) represent 66.3% of the fleet (scenario 1 and
2), while the aircraft subject to Phase 2 (classes C, D, E and F) represent 73.2% of the fleet
(scenarios 3 and 4). The aircraft subject only to Phase 2 comprise 6.9% of the overall fleet
included in the European ACAS safety encounter model.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 45
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
7.2.3.3
The encounters that involve small aircraft represent only 14% of the encounters of the
European ACAS safety encounter model. As a consequence, the proportion of encounters that
can be affected by Phase 2 is small. Moreover, these 14% comprise:
· 9% of encounters involving large aircraft versus small aircraft. For such encounters, one of
the two aircraft is already fitted with ACAS according to Phase 1;
· 5% of encounters that do not involve a large aircraft. In these encounters neither aircraft is
fitted with ACAS during Phase 1, but one or both of them is fitted with ACAS during
Phase 2.
7.3
Value to logic risk ratio of the extension of the ACAS II mandate
7.3.1
Ground point of view
7.3.1.1
Table 7.2 presents the logic risk ratios for each scenario. These risk ratios include the
contribution from encounters in which no aircraft is equipped with ACAS. This is shown as
“encounters without ACAS”. The contribution to risk ratio from encounters in which at least
one aircraft is equipped with ACAS is shown as “encounters with ACAS”.
scenario
0
1
2
3
4
mandatory carriage
None
Phase 1
Phase 1
Phase 2
Phase 2
ACAS II equipage
Class A
Class B
Class C
Class D
Class E
Class F
Class G
Class E
Class F
Class E
Class F
Class C
Class D
Class E
Class F
Class C
Class D
Class E
Class F
performance limitation
Without
Without
With
Without
With
encounters with ACAS
2.3%
9.4%
9.6%
9.0%
9.2%
encounters without ACAS
0.0%
12.4%
12.4%
3.8%
3.8%
risk ratio
2.3%
21.8%
22.0%
12.8%
13.0%
Table 7.2: Risk ratios vs. scenarios
Scenario 0 vs. scenario 1: Influence of unequipped aircraft
7.3.1.2
The results show that the main contribution to risk ratio during Phase 1 is from encounters in
which no aircraft are fitted with ACAS. Artificially equipping all aircraft reduces the risk ratio
dramatically, from 21.8% to 2.3%.
Scenario 1 (2) vs. scenario 3 (4): Influence of the equipage
7.3.1.3
As assessed by the logic risk ratio, fitting ACAS to small aircraft has a disproportionately
large effect on the risk of collision in the airspace. The logic risk ratio decreases from 21.8%
to 12.8%, i.e. a 41% decrease. In contrast, the proportion of aircraft that are not fitted with
ACAS decreases from 33.7% after Phase 1 to 26.8% after Phase 2, i.e. a 20.5% decrease. This
marked improvement is due to encounters in which the aircraft were not fitted with ACAS
under Phase 1, but at least one aircraft is equipped in Phase 2.
Scenario 1 (3) vs. scenario 2 (4): Influence of the performance limitations
7.3.1.4
The results show that the aircraft performances limitation have a limited influence on the risk
ratios. Indeed the risk ratio can be unchanged or increased by a small amount (+0.2ppt).
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 46
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
7.3.1.5
This can be explained by the fact that:
· the European ACAS safety encounter model has 70% of its encounters in the two first
altitude layers;
· the performance limitations in these layers only affect classes C and E.
7.3.2
Airborne point of view
7.3.2.1
Aircraft risk ratios were computed [WP-1 204]. These risk ratios show the benefits of Phase 2 in
terms of risk of collision for a given class of aircraft (i.e. large aircraft, small aircraft or light
aircraft). Figure 7.2 shows the aircraft logic risk ratios. The aircraft risk ratios related to Phase 1
are drawn with full lines. The aircraft risk ratios related to Phase 2 are drawn with dotted lines.
small aircraft (2000)
large aircraft (2000)
light aircraft (2000)
100
small aircraft (2005)
large aircraft (2005)
light aircraft (2005)
95
90
85
80
75
70
65
ratio
60
small (2000)
55
risk
50
45
40
light (2000)
35
30
light (2005)
25
20
15
large (2000)
large (2005)
small (2005)
10
5
0
0%
20%
40%
60%
proportion of RAs followed
80%
100%
Figure 7.2: Aircraft risk ratio
7.3.2.2
Fitting version 7 to small aircraft has a small influence on the aircraft risk ratio for large
aircraft. This was expected as the proportion of encounters involving large aircraft versus small
aircraft is only 9% of the total amount of encounters of the European ACAS safety encounter
model. So, for the majority of the encounters involving large aircraft, Phase 2 provides small
benefits. The main benefit is already brought by Phase 1.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 47
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
7.3.2.3
The improvements brought by Phase 2 for small aircraft is significant, as the aircraft risk ratio
is divided by up to 9.4 when all the pilots follow their RAs. The aircraft risk ratio is still
divided by 3.6 when a proportion of 20% of the pilots do not follow their RAs. Thus it can be
emphasised that the benefits brought by Phase 2 are significant for small aircraft, especially
when the pilots of other aircraft follow their RAs.
7.3.2.4
The improvement brought by Phase 2 for light aircraft are important, even if they are not as
significant as for small aircraft. The improvement is due to the addition of ACAS aircraft in
encounters between a small aircraft and a light aircraft.
7.3.2.5
All the risk ratios increase rapidly when the proportion of RAs followed decreases. It must be
highlighted that even a small proportion (i.e. 20%) of pilots who do not follow their RAs
debases seriously the safety benefits of the ACAS mandates, whatever the aircraft class. This
highlights once again the need to enhance the proportion of pilot following their RAs
through pilot training.
7.4
Consideration of the whole system
7.4.1
Introduction
7.4.1.1
The considerations in sections 7.2 and 7.3 were in terms of the logic risk ratio. These have the
advantage of being well tried and tested in the ACAS community, and (in this case) of being
able to include the effects of aircraft class. However, it is the effect of the Phase 2 mandate on
the ideal performance of the system that is judged. The event tree tool described in section 3.10
enables a global assessment of the effect on the risk of collision, taking account of all the
factors affecting the performance of ACAS. Amongst many other factors, it takes account of
the probable response of pilots rather than assuming the standard response.
7.4.1.2
The calculations reported in this section do not take aircraft type into consideration.
Nevertheless, it is possible to distinguish ‘large’, ‘small’ and ‘light’ aircraft.
7.4.1.3
This section reports the advantages of increasing the level of ACAS equipage from that
provided by Phase 1 to that provided by Phase 2. Large aircraft are those covered by Phase 1.
Small aircraft are those covered by Phase 2. Light aircraft are not covered by a mandate. Of
course, during Phase 1 there will be some small aircraft that have fitted ahead of schedule, and
some light aircraft might fit at any time. Nevertheless, the approximation is made that the
advantage of the Phase 2 mandate to large aircraft is the same as the advantage to aircraft that
are already fitted with ACAS.
7.4.1.4
It was reported in paragraph 7.3.1.4 that aircraft performance limitations have little effect on
risk ratio, and observed in paragraph 7.2.1.8 that there is little difference in performance
between large and small aircraft. This suggests that the inability of the event tree, in its current
form, to describe aircraft class is not a serious limitation in this context.
7.4.1.5
There is no subtlety of method; the numbers are simply inserted into and extracted from the
event tree tool.
7.4.2
Levels of ACAS equipage
7.4.2.1
The levels of ACAS equipage assumed for the event tree are 64.9% for Phase 1 and 71.2% for
Phase 2. These differ slightly from the values assumed in section 7.3 (66.3% and 73.2%) for
reasons explained in [WP-1 213].
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 48
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
7.4.3
Ground point of view
7.4.3.1
As reported in section 4.4, the airspace risk ratio for Phase 1 is 29.9%, and for Phase 2 it is
24.2%.
7.4.3.2
These results necessarily differ from those reported above (21.8% and 12.8%); it is a different
thing that is being reported.24 The stark conclusion, in paragraph 7.3.1.3, that a 20.5%
reduction in the aircraft not equipped leads to a 41% reduction in risk ratio is not shown here.
Nevertheless, it is still found that an 18.5% reduction in unequipped aircraft leads to a
significant, now approximately proportionate, reduction in risk ratio – 19.1% when the whole
system is considered.
7.4.4
Airborne point of view
7.4.4.1
In this section, we report the progressional risk ratios for Phase 2 of the ACAS mandate for
large and small aircraft. In other words, for both classes, we report
progressional risk ratio =
risk of collision for the aircraft at the end of Phase 2
,
risk of collision for the aircraft at the end of Phase 1
which expresses the immediate benefit of the change to the aircraft in question.
7.4.4.2
The results are as follows.
progressional risk ratio for large aircraft
86.2%
progressional risk ratio for small aircraft
24.0%
progressional risk ratio for light aircraft
86.6%
7.4.4.3
These results show that small aircraft will derive significant benefit from Phase 2 of the ACAS
mandate.
7.4.4.4
However, small aircraft appear to derive less benefit when assessed from the total system
perspective than is claimed in section 7.3. This is because the logic risk ratio overestimates the
efficacy of ACAS, as it does for all aircraft. (In this case, particularly because correct pilot
response was assumed for the logic risk ratios – as emphasised in section 7.3. More generally,
the event tree describes very many more failure modes than does the logic risk ratio.)
7.4.4.5
Again, large aircraft and light aircraft also derive a benefit from Phase 2, as discussed in
section 7.3.2. However, in the system results, large aircraft and light aircraft seem to derive
similar benefit, whilst from the logic risk ratio results discussed in section 7.3.2 it appeared
that large aircraft derived relatively less benefit than light aircraft. This is probably because the
total system perspective (as implemented here) does not include the concept of aircraft classes
(as implemented for the logic risk ratio calculations). There is a measure of segregation
between large aircraft and small aircraft in the encounter model (as used for these logic
calculations), which would lead to the relatively small benefit for large aircraft reported in
section 7.3. This point has been checked by repeating the logic and system calculations using
QinetiQ’s implementation of the encounter model, which does not include aircraft class. The
results for large and light aircraft were more consistent in these calculations, confirming this
understanding. In this respect (that the benefit for large aircraft is very small), the logic
calculations reported in section 7.3 are probably giving the better picture.
24
Less importantly, there are many detailed differences in the assumptions made. Paragraph
7.4.2.1 is the first. The second, which is that the probabilities for the logic events are not the
same in section 7.3 as those in the event tree, is discussed in paragraph 4.1.1.5. There are
others.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 49
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
7.5
Conclusions
7.5.1
Including aircraft performance limitations in the ACAS II simulations has nearly no impact on
the computed risk ratios.
7.5.2
The benefit to be obtained by fitting ACAS II to small aircraft is significant. Implementing
Phase 2 of the ACAS mandate is predicted to reduce the logic risk ratio (based on an
accurate pilot response) from 21.8% to 12.8%, and to reduce the overall system risk ratio
from 29.9% to 24.2%.
7.5.3
From an airborne point of view, Phase 2 brings:
· substantial safety benefits to small aircraft (i.e. those subject to Phase 2); and
· significant safety benefits to aircraft with a maximum take-off mass below 5700kg (i.e. light
aircraft, not subject to any mandatory carriage);
· small benefits to large aircraft (subject to Phase 1).
7.5.4
The CENA study confirms that the more the pilots comply with their RAs then the greater the
benefits brought by the both phases of the mandate.
7.6
Recommendations
7.6.1
It is proposed that the marked safety benefit of fitting ACAS II to small aircraft be
brought to the attention of European decision makers to support Phase 2.
7.6.2
As reported in chapter 6, the need to follow the RAs must be emphasised.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 50
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
8
Multiple encounters
8.1
Introduction
8.1.1
Multiple encounters are rare but they occur. Furthermore, concerns about the risk that ACAS II
will cause the equipped aircraft to move into the path of a third party are frequently expressed,
usually by people meeting ACAS for the first time. On the other hand, ACAS safety studies
have tended to avoid the question of the effect of multiple encounters on the overall ‘safety’ of
ACAS. The consideration has been that the normal ACAS logic will handle any encounter
induced with a third party, and that genuinely multiple encounters are extremely rare. The first
purpose of the studies of multiple encounters reported here was to collect objective evidence
on the frequency of multiple encounters, and to quantify their effect on the assessment of the
safety of ACAS.
8.1.2
The multiple encounter logic has been dramatically improved in version 7 by comparison with
version 6.04A [35]. The second purpose of the studies of multiple encounters reported here
was to demonstrate this improvement.
8.2
Methods and data sources
8.2.1
ACAS multiple encounter data were collected from two sources:
· radar data were collected for multiple encounter events reported by controllers and pilots;
· 154 days of radar data for South-East England were processed during the construction of
the European encounter model, as discussed in section 3.5. All multiple encounters were
collected at the same time.
8.2.2
The processing of the first set of events was reported in [WP-1 146], and is summarised in
section 8.4. Here the objective was to examine the performance of the collision avoidance
logic, particularly from an operational perspective.
8.2.3
The second events enabled a calculation of the frequency of multiple encounter events for the
airspace observed (a very busy airspace, with several holding patterns). This was reported in
[WP-1 212], and in section 8.3. The same data was used to attempt to estimate the significance
of multiple encounters for the safety of ACAS, which is discussed in section 8.5
8.3
The frequency of multiple encounters
8.3.1
The radar data examined in order to capture encounters amounted to 3.4´105 flying-hours.
8.3.2
The data were examined to find every encounter between two aircraft (A and B) in which there
could be an RA. For each such encounter, there was a further search for proximate aircraft
sufficiently close to qualify the original encounter as a ‘multiple-aircraft encounter’. This
included the idea that the deviation caused by the RA in aircraft A might cause it to move so
close to a third party, aircraft C, that there would be an RA against aircraft C.
8.3.3
Seven encounters were found, each involving only three aircraft. Each encounter was
examined, and in only two encounters would there have been an RA. The others are not
multiple encounters in the sense that an RA would cause an interaction with a third party, nor
in the sense that ACAS would have to resolve two collisions threats at the same time, or even
in rapid succession.
8.3.4
On this basis, it is concluded that the frequency of multiple ACAS encounters is of the
order of one in 1.7´105 flying-hours in the busy airspace observed.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 51
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
8.4
The efficacy of the ACAS II logic in multiple encounters
8.4.1
The purpose of this section is to report the operational performance of ACAS II during
multiple-aircraft encounters (i.e. encounters in which an aircraft of interest is close to a second
aircraft and proximate to at least one other).
8.4.2
With that aim, a subset of 11 operational multiple-aircraft encounters was simulated with
version 7. Version 6.04A was also simulated so as to enable an assessment of any
improvements brought by version 7. The encounters were reviewed with the help of an air
traffic controller in order to judge the operational performance of ACAS.
8.4.3
This assessment [WP-1 146] concludes that, since multiple-aircraft encounters are uncommon
events [WP-1 212], RAs against multiple-threats (i.e. RAs that invoke the multiple-aircraft
logic) can be considered rare, as only two encounters out of the total of eleven generate such
RAs. In addition, these RAs are considered efficient as they take into account all threats
simultaneously and do not induce any significant deviations that are disruptive from an ATC
standpoint.
8.4.4
The version 7 RAs in these multiple-aircraft encounters have shorter durations than the
corresponding version 6.04A RAs, and induce slightly smaller deviations. The shorter duration
of the RAs observed with version 7 is due partly to the earlier issuance of the “Clear of
conflict” indication (resulting from the implementation of a miss distance filter in version 7)
and partly due to the later issuance of the initial RAs.
8.4.5
Moreover, version 7 performs more efficiently as it prevents the generation of a significant
fraction (48%) of the RAs generated by version 6.04A, and those that remain are more
compatible (especially in encounters where the threat levels-off 1000ft away).
8.4.6
The RAs between two aircraft induced a conflict with a third aircraft in only one encounter out
of the eleven that were simulated. This behaviour was also observed in the real encounter as
recorded. This situation is mainly caused by an overreaction of the pilot, who is known to have
had version 6.04A onboard her aircraft. The simulations indicate that had this aircraft been
equipped with version 7 the induced conflict would most likely not have occurred, since the
RA which leads to the overreaction of the pilot with version 6.04A is not triggered with
version 7.
8.4.7
Thus, from an operational point of view, version 7 performs well in these eleven multipleaircraft encounters, and is considered to perform better than version 6.04A.
8.4.8
However, the extremely small number of cases in which ACAS had to resolve simultaneous
threats limits this conclusion. There were only two examples in all the multiple encounter data
studied, from both sources. The part of the version 7 logic that handles several simultaneous
threats has been extensively tested by the designers MITRE [35], so further validation might
not be required. However, if further validation is desired, it will have to be based on artificially
created encounters.
8.5
Multiple encounters and the safety of operations with ACAS II
8.5.1
None of the encounters collected was directly relevant to a calculation of ACAS risk ratio.
Either the vertical separation was so large that there was no ACAS RA, or the hmd was larger
than 0.1NM, so that there was no NMAC and no chance of an RA inducing an NMAC. Further
processing was required to examine the relevance of multiple encounters to NMAC rate with
or without ACAS.
8.5.2
In each encounter, the aircraft of central importance is aircraft A: aircraft A is certainly ACAS
equipped, aircraft B is close enough to consider the possibility of an RA, and aircraft C is close
enough to aircraft A to consider the interaction between aircraft C and the RA. The trajectories
of aircraft B and C were modified so that the hmd became zero in their encounters with
aircraft A.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 52
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
8.5.3
With this modification, it was hoped to address three questions:
· What is the risk that the proximity of aircraft C will lead to an NMAC between aircraft A
and B that would not occur in the absence of aircraft C?
· What is the risk of an NMAC between aircraft A and C that would not occur in the absence
of aircraft B?
· How do the answers to these two questions differ for version 6.04A and version 7?
8.5.4
In spite of modifying the aircraft trajectories, there were still only two encounters in which
there was an RA in aircraft A. Furthermore, unfortunately, one of the two encounters
dominates the results and needs special discussion, in paragraph 8.5.6. Nevertheless, the
calculation was completed. The rate at which NMACs are induced in multiple encounters was
found to be less than 1.5 in 107 flying-hours.
8.5.5
It is essential to appreciate that the result in the previous paragraph is based on a single
encounter, and also on a particular view of that encounter. It is considered that the risk of
collision, with ACAS, between aircraft A and aircraft C in the presence of aircraft B is as
though ACAS were not fitted, whereas ACAS significantly increases the separation between
the aircraft. There is no genuine indication that the rate at which NMACs are induced in
multiple encounters is remotely as large as 1.5 in 107 flying-hours. But the sample is so small
that a good estimate for this statistic is not possible, and it has not been proved that induced
collisions in multiple encounters are numerically negligible.
8.5.6
In the single encounter that dominated this result, the ACAS equipage would have led to an RA
in the original two-aircraft encounter between aircraft A and C. This would have been resolved
by a non-crossing RA: advice to increase the altitude separation by moving away from the
altitude of the other aircraft. However, in the three aircraft encounter, the RA in aircraft A
against aircraft B preceded the RA against aircraft C, and the RA against C was a crossing RA:
advice to cross through the altitude of the other aircraft. This happened to result in greater
separation than the non-crossing RA. Had this been one of many encounters that were being
considered, this result would have been used as it stood. However, it is plain that the presence
of a third party will in general degrade the performance of ACAS, and it was felt unacceptable
to rely on a single encounter and report the opposite finding! Crossing RAs are generally
considered a contra-indication. For these reasons, it was judged that the presence of aircraft B
caused ACAS not to resolve the original encounter between aircraft A and aircraft C, and the
result reported is the consequence of this extremely conservative approach.
8.5.7
The investigation of the effect of multiple encounters on the effect of ACAS was based on very
limited data, and a very conservative analysis of those data. It has not been possible to
demonstrate that multiple encounters can be ignored in assessing the overall safety of
ACAS, and further work would be required to investigate this issue.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 53
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
9
Discussion
9.1
The initial list of questions
9.1.1
At the beginning of the work, the Work Plan posed a number of questions for WP-1, as
detailed in paragraph 2.1.5. Here we reiterate the questions and indicate the answers revealed
by the study.
9.1.2
What is the overall reduction in the risk of collision in European airspace obtained by
extending the ACAS II mandate to small aircraft?
The current benefit of ACAS is estimated to be a reduction in the risk of collision to 29.9% of
the risk that would exist without ACAS equipage. Extending the mandate to include small
aircraft is expected to reduce this risk still further to 24.2% of the risk that would exist without
ACAS equipage.
9.1.3
What benefit do small aircraft gain in terms of reduced risk of collision from fitting ACAS II in
an environment where large aircraft are already equipped (and thus some measure of
protection is already available, by proxy)?
Small aircraft receive the same benefit from fitting ACAS II as does any other aircraft that
equips at the same time. Small aircraft that equip as a consequence of Phase 2 of the mandate
face a risk of collision that is 24.0% of that they faced after Phase 1 of the mandate.
9.1.4
Is the performance of small aircraft so limited that ACAS II is inherently of less value to them
than to large aircraft, and how does this affect the answers to the previous questions?
The study has shown that concerns regarding the performance limitations of small aircraft in
general are unfounded. Indeed, in many cases the performance of small aircraft is less limited
than that of large aircraft. The study has addressed the consequences of limited aircraft
performance only for classes of aircraft, and not for particular aircraft types.
9.1.5
What additional benefit do large aircraft, which are already ACAS II equipped, gain when
small aircraft are also equipped?
The level of ACAS equipage increases as a consequence of Phase 2 of the ACAS mandate.
The aircraft that equipped as a consequence of Phase 1 will experience a further reduction in
the risk of collision. The event tree indicates that it will reduce to 86.2% of the risk that existed
immediately after Phase 1, but does not take account of airspace segregation between large and
small aircraft. The logic calculations based on a model that distinguishes between aircraft
classes, and thus models this segregation, indicate that it will reduce to 91.3% of the risk that
existed immediately after Phase 1.
9.1.6
What is the effect on the risk of collision of non-response and slow response to ACAS II
avoidance manoeuvres?
The effect of imperfect response to RAs is critical to the performance of ACAS. Failing to
respond, or responding inaccurately, nullifies much of the safety benefit available from ACAS.
9.1.7
What effect on the risk of collision do the situational awareness and alerting functions of
ACAS II provide?
The supposed benefit of the enhanced situational awareness available from use of ACAS has
not been assessed. However, the alerting functions (which can prompt visual acquisition or
contact with the controller) are found to almost halve the risk of collision that would exist if
the sole function of ACAS was to generate RAs.
9.1.8
Can a useful reduction in the risk of collision be achieved by mandating the use of 25ft altitude
data by suitably equipped aircraft?
The use of 25ft altitude data is found to reduce the risk of collision. However, the benefit is
small and the mandatory use of 25ft altitude data is hard to justify on this basis alone.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 54
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
9.1.9
Is the frequency of multiple encounters (close encounters between more than two aircraft) high
enough to warrant the inclusion of this factor in calculations of the risk of collision with or
without ACAS II?
Multiple encounters are rare events. However, it has not been possible to demonstrate that
multiple encounters can be ignored is assessing the overall safety of ACAS. Further work
would be required to investigate this issue.
9.1.10
What effect does version 7, vice version 6.04A, have on the risk of collision in multiple
encounters?
The short answer is that we cannot tell: there is insufficient data to form a meaningful answer.
Version 7 is expected to be significantly better than version 6.04A, and there is nothing to
contradict this. From an operational point of view, version 7 was generally found to perform
better than version 6.04A in multiple encounters, but this was largely a consequence of
features in version 7 unrelated to the multiple-aircraft logic.
9.2
Tools and methods
9.2.1
The tools and methods used to carry out this work are not the point of these studies, and it is
acknowledged that they will be of little general interest. Nevertheless, many lessons have been
learnt in the course of the work, and the opportunity needs to be taken to record some of them.
9.2.2
The European ACAS encounter model is a considerable improvement on the ICAO model, and
more importantly it is tuned to Europe. However, even when large quantities of radar data were
collected, it was difficult to populate all corners of the model with statistically useful numbers.
It is worth seeking methods to reduce the numbers of options in the model, without harming its
richness and flexibility.
9.2.3
There are related problems in the collection of encounters with which to tune the model and
the process of adjusting the NMAC rate. Encounters in which the vmd is small are amongst
those most likely to be unrepresentative, either because aircraft are deliberately flying close
together with small horizontal separation or because of radar processing error. Moreover, the
values of vmd < 100ft in the encounter model are those most influenced by the process of
imposing the desired NMAC rate. It would be worth considering eliminating all the observed
radar encounters from the tuning process and determining the probability of small vmd from
the imposed NMAC rate alone.
9.2.4
The surviving differences between the Sofréavia/CENA implementation and the QinetiQ
implementation have been noted. They are frustrating, and it would be pleasing to resolve
them. However, such an effort is likely to prove costly and the work with the event tree
indicates that the differences in the evaluation of the logic are not the most serious issue should
we wish to improve our confidence in our estimates of the efficacy of ACAS. It might also be
very useful to build a second event tree to validate that developed by ACASA.
9.2.5
The effect of imperfect surveillance on the logic has been only partially examined, and we do
not know how significant this is. If further logic risk ratio calculations can be made for less
optimistic scenarios than that examined in chapter 5, it would be relatively straightforward to
modify the event tree so as to take them into account.
9.2.6
We have very little knowledge of how frequently ACAS fails to track an intruder altogether.
The estimates used here are believed realistic.
9.2.7
Several changes would increase the utility of the event tree, which has in any case proved a
remarkably useful tool:
· the ability to describe ACAS on, but in TA only mode;
· undue credit is given to the ability of the traffic display to aid visual search when there is no
altitude data; and
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 55
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
· recrafting the tree so that it makes calculations oriented to a particular aircraft with a
precise configuration, while the population as a whole is described probabilistically as at
present.
9.3
Other issues
9.3.1
The single most important factor for the results from WP-1 is the level of collision risk without
ACAS. For this, ACASA had to make a judgement. It would have been most useful to have
open and widely accepted data concerning the actual level of collision risk in Europe.
9.3.2
The risk that the operation of the ACAS logic will induce a collision is probably at the margin
of acceptability. This is justified by the fact that ACAS reduces the risk of collision. However,
it indicates that it is essential to take any opportunity to reduce this risk. In this context, we
would remark that the problem is theoretically unavoidable (it can only be minimised), and is it
not a matter of software engineering. The remedy lies in the collision avoidance algorithms
themselves rather than their implementation as code.
9.3.3
ACASA has failed to demonstrate that the risk of ACAS inducing collisions in multiple
encounters is sufficiently small to be neglected in the context of estimating ACAS safety. None
the less, still the assertion is probably true. We now know the frequency of multiple
encounters, at least for the very busy airspace observed. What is required now is a very large
number of artificial simulations to quantify the risk of induced collisions in multiple
encounters with small hmd, when they occur. Extending the European ACAS safety encounter
model to include a third aircraft would be a significant undertaking.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 56
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
10
Conclusions
10.1
Background, tools and methods
10.1.1
The need for these ACAS safety studies, to support the ACAS mandate in Europe, was
outlined in chapter 1, and the work described in chapter 2 has been carried out.
10.1.2
The tasks outlined for ACASA WP-1 required fresh work on the development of tools for the
evaluation of ACAS. A European ACAS safety encounter model and a new ACAS collision
risk event tree are the most notable. (Chapter 3)
10.1.3
It was known before the work started that assessments of ACAS performance were extremely
sensitive to the characteristics of the airspace for which the assessment was made. This was
confirmed during WP-1. Future changes to ATM operations can be expected to have an effect
on ACAS performance. (Paragraph 2.4.6)
10.1.4
The central and most important assumption in all the work is the figure used for the NMAC
rate before ACAS is introduced. The rate assumed was three NMACs in 107 flying-hours. This
assumption was considered reasonable and realistic, but the ACASA consortium cannot and
does not assert that it is correct. It simply reports that this was the value used. (Section 3.6)
10.1.5
The encounter model developed by ACASA arises directly from the encounter model specified
in the ACAS SARPs for the evaluation of ACAS logic. This ICAO model has been
implemented by both CENA and Sofréavia, and by QinetiQ. Comments on the ICAO standards
are in Appendix A.
10.1.6
The tools developed by ACASA merit further consideration. Three areas are particularly
noteworthy. (Paragraph 9.2.7)
· There is considerable potential of the European ACAS safety encounter model to be used in
other work investigating the interaction of ACAS and ATM practices.
· The structure of the event tree and the probabilities it receives as input could be refined to
produce an improved estimate of the benefit of ACAS.
· The techniques developed to study the effects of imperfect surveillance should be extended
to further scenarios relevant to the performance of ACAS in European airspace, in
particular encounters where one aircraft is not equipped and the pilot does not respond
perfectly.
10.2
Performance of ACAS in European airspace
10.2.1
The study has demonstrated that ACAS can bring about a significant reduction in the risk of
mid-air collision. (Paragraph 4.4.2.3)
10.2.2
Under Phase 1, it has been found that the risk of collision in European airspace will have been
reduced to 29.9% of the risk before ACAS was introduced (‘airspace risk ratio’). (Paragraph
4.4.2.1)
10.2.3
The theoretical performance of the logic considered in isolation has also been estimated. In
encounters between an ACAS equipped aircraft and an unequipped intruder, the logic risk ratio
is found to be 22.9%. In encounters between two ACAS equipped aircraft, both of which
respond as intended, the logic risk ratio is found to be 3.3%. In both cases, for these results it
was assumed that the aircraft reported altitude in 100ft increments. (Paragraphs 4.1.2.1, 4.1.3.1)
10.2.4
It has been found that the risk of an induced collision is a significant fraction of the risk of
collision that remains when ACAS is deployed. By itself, this induced risk could well be
unacceptable; it is only because it arises as an unavoidable consequence of gaining the overall
reduction in risk that it is accepted. Any change in the collision avoidance logic to reduce the
frequency of ACAS induced NMACs must be encouraged and promoted. (Paragraph 4.2.4.5)
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 57
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
10.3
Phase 2 and extending ACAS equipage
10.3.1
To achieve the full benefit of the ACAS mandate, it is critical that as high a proportion of
aircraft as possible are equipped with ACAS. Additional equipage has a direct effect on the
risk of collision in an airspace, but it also has a significant beneficial effect on the performance
of ACAS already deployed. (Paragraph 4.4.2.3)
10.3.2
Phase 2 is expected to further reduce the risk of collision in European airspace to 24.2% of the
risk before ACAS was first introduced. (Paragraph 4.4.2.1, 7.5.2)
10.3.3
The benefit to an aircraft of fitting ACAS is notably constant. The risk of collision faced by the
aircraft immediately after fitting ACAS varies between 26.7% and 27.8% of the risk
immediately before fitting ACAS, depending on precisely when the aircraft equips. An aircraft
that equips with ACAS at the end of Phase 2 still reduces its risk of collision by a factor of
more than three in spite of the protection afforded by ACAS on other aircraft. (Paragraph
4.4.2.3)
10.3.4
Concerns that performance limitations might limit the benefit of ACAS to small aircraft,
generally, have proved unfounded. (Paragraphs 7.3.1.3)
10.3.5
The marked safety benefit (for small aircraft and light aircraft) of fitting ACAS II to small
aircraft should be brought to the attention of European decision makers to support Phase 2.
(Paragraphs 7.6.1)
10.4
Pilot response to ACAS
10.4.1
The ACAS alerting functions (prompting visual search, or contact with the controller) almost
halve the risk of collision that would exist were RAs the sole function of ACAS (56.9% is
reduced to 29.9%). (Paragraph 4.5.2.2)
10.4.2
The responses of pilots to ACAS RAs (both whether they respond and the manner of their
response) have been found to be often deficient. (Paragraphs 6.2.2.4, 6.2.3.9)
10.4.3
Furthermore it has been confirmed that deficient response seriously degrades the safety
benefits afforded by ACAS RAs. It is critical that pilots respond accurately and promptly to
RAs. (Paragraphs 4.2.4.4, ):
· The performance of the logic is seven times worse when the response of the pilot is ‘slow’
than it is for the standard pilot response (63.3% vs. 9.0%). (Paragraph 6.3.2.1)
· A pilot who responds perfectly can reduce her risk to less than half that faced by typical
pilots of ACAS equipped aircraft. (Paragraph 6.4.3.3)
· A pilot who always ignores RAs exposes herself, and the unwitting pilots of the aircraft she
encounters, to a risk of collision more than one-and-a-half times that faced by average
pilots. (Paragraph 6.4.3.4)
10.4.4
As a consequence, pilot training is a crucial factor in achieving a successful implementation of
ACAS II. (Paragraph 6.3.2.4, 7.3.2.5)
10.5
The effect on ACAS safety of imperfect surveillance
10.5.1
Realistic but extreme surveillance errors have only a small effect on the efficacy of ACAS in
ACAS-ACAS encounters, given that both pilots respond correctly. The ACAS logic appears to
be robust in this respect. (Paragraph 5.4.1)
10.5.2
This result needs to be confirmed for encounters where pilots do not respond correctly, or only
one aircraft is equipped. When this is done, the effect of imperfect surveillance could be
introduced in the event tree. (Paragraphs 5.4.1, 9.2.5)
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 58
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
10.6
Multiple encounters
10.6.1
The frequency of multiple ACAS encounters is of the order of one in 1.7´105 flying-hours in
the busy airspace observed. (Paragraph 8.3.4)
10.6.2
From an operational point of view, version 7 performs well in the eleven multiple-aircraft
encounters that were collected, and is considered to perform better than version 6.04A.
(Paragraph 8.4.7)
10.6.3
The investigation of the effect of multiple encounters on the performance of ACAS was based
on very limited data, and a very conservative analysis of those data. Further studies of the
multiple-threat logic are desirable to ensure that ACAS is effective in multiple encounters and
to ensure that the assumption that they are not significant, in terms of the overall performance
of ACAS, is warranted. (Paragraph 8.5.7)
10.7
Other factors influencing the safety of ACAS
10.7.1
The availability of altitude data measured precisely and reported in 25ft increments, as
opposed to 100ft increments, significantly improves the performance of the ACAS logic. On
this basis, aircraft equipped with Mode S transponders should be encouraged to use a digital
data source and report altitude in 25ft increments. (Paragraph 4.3.4.1)
10.7.2
Aircraft should be encouraged to fit a transponder and use it to report altitude, so that ACAS
can avoid them. An aircraft that is not equipped with a transponder approximately halves its
risk of mid-air collision when it fits a transponder. (Paragraph 4.5.4.4)
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 59
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
11
Recommendations
11.1
General
11.1.1
In order to reduce the risk of collision for aircraft that do not have ACAS, and to maximise the
value of ACAS for aircraft already equipped, the fitting and use of ACAS should be promoted
vigorously.
11.1.2
The consequences in terms of collision risk following failure to respond to RAs, and to do so
accurately and promptly, must be emphasised to pilots by all practicable means.
11.1.3
Any change in the collision avoidance logic to reduce the frequency of ACAS induced
NMACs should be encouraged and promoted.
11.1.4
Aircraft equipped with Mode S transponders should be encouraged to use a digital data source
and report altitude in 25ft increments.
11.1.5
All aircraft should be required to fit a transponder and use it to report altitude.
11.1.6
The marked safety benefit of fitting ACAS II to small aircraft should be brought to the
attention of European decision makers to support Phase 2.
11.2
Further work
11.2.1
Further studies of the efficacy of the multiple-threat logic should be conducted, in order to
verify or contradict the assumption that multiple encounters are not significant to the overall
performance of ACAS.
11.2.2
The potential of the European ACAS safety encounter model to be used in other work
investigating the interaction of ACAS and ATM practices should be noted.
11.2.3
The structure of the Event Tree and the probabilities it receives as input should be reviewed
and refined.
11.2.4
The techniques developed to study the effects of imperfect surveillance should be extended to
further scenarios relevant to the performance of ACAS in European airspace.
11.2.5
The significance of Gilham encoding errors in Mode C altitude reports to the overall
assessment of the safety of ACAS should be examined.
11.3
Observation
11.3.1
Civil aviation authorities and ATS providers need to be open and accurate concerning
the risk of collision in the airspace for which they are responsible. It is not possible to
give accurate advice concerning the effect of ACAS without equally accurate advice
concerning the problem that ACAS addresses.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 60
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
12
References
[1]
‘The mathematics of collision avoidance in the air’, J Morrel, J. Inst. Nav., 11, p40, 1958
[2]
‘Flight trials of an experimental collision warning system’, S Ratcliffe, RRE Memo 1534,
October 1958
[3]
‘FAA BCAS concept’, E Koenke, FAA-EM-78-5, April 1978
[4]
‘Report on agenda item 3: development of draft SARPs … for Annex 10 relating to … ACAS’,
SICASP/4 WP/66, April 1989
[5]
‘Report on agenda item 4: development of proposals for amendments to … Annex 2, Annex 6,
Annex 11, PANS-RAC, PANS-OPS and ATS Planning Manual’, SICASP/4 WP/66, April 1989
[6]
‘Summary of ACAS II operational evaluation results’, Appendix A to the report on agenda item 2,
SICASP/5, November 1993
[7]
‘Summary of the forty-third meeting, Special Committee 147, MOPS for TCAS airborne
equipment’, RTCA Paper 502-93/SC147-578, November 1993
[8]
‘Minimum operational performance standards for Traffic alert and Collision Avoidance System II
airborne equipment’, RTCA DO-185A, 1997
[9]
‘RWG Recommended Modifications 1.0 to TSO C119B’, WJH FAA Technical Center ACT 350,
April 1999.
[10] ‘System Safety Study of Minimum TCAS II’, J Lebron et al., MITRE, MTR-83W241,
DOT/FAA/PM-83/36, December 1983
[11] ‘System Safety Study of Minimum TCAS II for Instrument Weather Conditions’, J Lebron &
S Mulder, MITRE, MTR-85W28, June 1985
[12] ‘ACAS II safety levels’, Rapporteur of WG2, SICASP/4-WP/35, April 1989
[13] ‘An evaluation of ACAS risk’, M Law & R Phillips, SICASP/WG2 WP2/220, October 1988
[14] ‘Induced NMAC risk estimate with upper bounds’, R Lejeune, SICASP/WG2 WP2/242,
January 1989
[15] ‘An update to the TCAS II System Safety Study’, N Spencer, MITRE, MTR-88W115,
December 1988
[16] ‘Summary of the predicted effect of TCAS II on ATC operations in the UK’, report prepared for
NATS, K Carpenter & H Hutchinson, DRA/LS(LSC4)/ACAS/CR/PAP/5, May 1995
[17] ‘Likely operation of TCAS II (Version 6.04A) in the UK’, report prepared for NATS,
H Hutchinson & K Carpenter, DRA/LS1(ATC)/ACAS/CR/PAP/8, January 1997
[18] ‘Safety of TCAS II Version 7 in the UK’, report prepared for NATS, H Hutchinson & K Carpenter,
DERA/LSB1/ATCSG/ACAS/CR/PAP/11, June 1999 (also SICASP/WG2 WP2/785)
[19] ‘Methodologies for
September 1990
evaluating
ACAS
safety’,
R Lejeune,
SICASP/WG2
WP2/291,
[20] ‘A safety analysis of the VVT logic’, R Lejeune, SICASP/WG2 WP2/320, October 1991
[21] ‘System safety study of ACAS II for logic version 6.04A’, A Zeitlin & M McLaughlin, WP2/376,
October 1992
[22] ‘Safety Study of TCAS II for Logic Version 6.04’, M McLaughlin & A Zeitlin, MITRE,
MTR-92W102, DOT/FAA/RD-92/22, RTCA 505-92/SC147-518, July 1992
[23] ‘Safety Study of TCAS II for Logic Version 6.04A’, M McLaughlin & A Zeitlin, MITRE,
MTR-95W234, November 1993
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 61
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
[24] ‘The effect of observed airspace differences on risk ratio’, K Carpenter & H Hutchinson,
SICASP/WG2 WP2/370, October 1992
[25] ‘ACAS I25I safety analysis results’, Appendix B to the report on agenda item 2, SICASP/5,
November 1993
[26] ‘Performance of the ACAS II collision avoidance logic’, Appendix A to the report on agenda
item 3, SICASP/6, February 1997
and also section 4.4 of ‘International standards and recommended practices: aeronautical
communications’, Annex 10 to the Convention on Civil Aviation, Volume IV, ‘Surveillance radar
and collision avoidance systems’, second edition, July 1998
[27] ‘TCAS 2 logic version 7.0: safety study’, T Arino & F Casaux, SICASP/WG2 WP2/646,
October 1997
[28] ‘Contribution of TCAS II Logic Version 7.0 in the European Airspace’, T Arino & F Casaux,
ATC Quarterly, 6, 4, p249–268, 1998
[29] ‘TCAS II Full System Safety Study’, EDS, under contract to NATS, C305340/R-01/20,
March 1995 (also SICASP/WG2 WP2/785)
[30] ‘UK full system safety study’, A Price, C Smith & K Carpenter, SICASP/WG2 WP2/485,
March 1995; and
‘UK full system safety study – update’, C Smith, SICASP/WG2 WP2/529, September 1995
[31] ‘Final report – Study of interaction between RVSM and ACAS within the European RVSM
airspace’, C Aveneau & B Bonnemaison, ACASA/WP3.6/185D, September 2001
[32] ‘The effect of the introduction of RVSM on the efficacy of ACAS above FL290’, H Hutchinson &
K Carpenter, ACASA/WP3.3/177, August 2001
[33] ‘Operational performance of TCAS 2 logic change 7 release 8 in European airspace’, T Arino &
F Casaux, SICASP/WG2 WP2/594, September 1996
[34] ‘European TEN Study: ACAS Analysis – Work Plan’, ACASA/WP10/001 version 1.3.1,
January 1999
[35] ‘The new multiaircraft logic design and testing’, J Yang, RTCA Paper 158-97/SC147-673,
May 1997
[36] ‘Human factors analysis’, C Seton-Watson, volume 4 of [29], April 1995
[37] ‘Risk ratio calculations based on the standard encounter model’, G Yates & H Hutchinson,
SICASP/WG2 WP2/809, October 1999
[38] ‘Vertical tracker issue with TCAS II logic version 7.0’, T Arino & F Casaux, SICASP/WG2
WP2/756, May 1999
[39] ‘Predicting the effect of TCAS II on safety’, T McLaughlin, ATC Quarterly, 7, 1, p1, 1999
[40] ‘Experience in the use of the ICAO ACAS encounter model’, presented by J Law, SICASP/7 WP/8,
September 2000. A fuller version is available as SICASP/WG2 WP2/841 or, better,
ACASA/WP1/109
25
With one exception: being on a collision course is not an independent event. Rather, the
precise value of its probability is an assumption that is required for the adjustment of the vmd
distribution which in turn is used to generate the encounters that are subsequently simulated to
obtain the logic event probabilities that relate to the operation of ACAS. The same value has
been assumed in the calculation of the both the primary and secondary values of the logic
events, consequently no secondary value of the probability of being on a collision course is
given.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 62
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
[41] ‘ACAS safety’, K Carpenter & H Hutchinson, SICASP/WG2 WP2/337, April 1992
[42] ‘ACAS SARPs’, ICAO Annex 10 second edition of volume IV, July 1998
[43] ‘TCAS performance with non-standard pilot responses’, H Hutchinson, SICASP/WG2 WP2/411,
February 1993
[44] ‘Statistics on the airborne implementation of Mode S’, Y Sagnier, SICASP-7/WP-23 Agenda
Item 3 & ACASA/WP-1.2/136 Version 1.1, July 2000
[45] ‘An analysis of pilot reaction to version 7 resolution advisories’, D Tillotson, SCRSP/WGA
IP/A/2-59, October 2001
13. Annexes
Annex A: Comments on the ICAO encounter model
Annex B: Event probabilities
Annex C: ACASA WP-1-working paper
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 63
ACAS PROGRAMME
ACASA PROJECT
Annex A to
Work Package 1
Final report on
Studies on the Safety of ACAS II
in Europe
Edition 1.3
March 2002
ACAS PROGRAMME, ACASA PROJECT
WP-1- Studies on the Safety of ACAS II in Europe
Annex A to
Work Package 1
Final Report on
Studies on the Safety of ACAS II
in Europe
Version 1.3, March 2002
ACASA/WP – 1.8/210D
Version 1.3
March 2002
Page i
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
DOCUMENT IDENTIFICATION SHEET
DOCUMENT DESCRIPTION
Document Title
ACAS PROGRAMME ACASA PROJECT
Annex A to
Work Package 1
Final Report on
Studies on the Safety of ACAS in Europe
EWP DELIVERABLE REFERENCE NUMBER:
PROGRAMME REFERENCE INDEX:
1.3
VERSION:
March 2002
EDITION DATE:
Abstract
ACAS II is an airborne avionics system designed to reduce the risk of mid-air collision. Its
carriage by a majority of aircraft within Europe is mandatory.
ACASA is a set of studies that has investigated several areas related to ACAS II operations in
Europe.
Work Package 1 of the ACASA project investigated safety. The safety of ACAS was assessed
purely in terms of the degree to which it achieved its stated aim of preventing mid-air collisions.
Work Package 1.3, Annex A contents comments on the ICAO encounter model.
Keywords
ACAS
TCAS
CONTACT PERSON:
Authors:
RVSM
Garfield Dean
TA
TEL: 33-1-6988 7587
UNIT:
RA
EEC Brétigny
Thierry Arino, Ken Carpenter, Stephan Chabert, Harry Hutchinson, Thierry Miquel,
Beatrice Raynaud, Kevin Rigotii and Eric Vallauri
DOCUMENT STATUS AND TYPE
STATUS
CLASSIFICATION
Working Draft
o
General Public
o
Draft
o
EATMP
þ
Proposed Issue
o
Restricted
o
Released Issue
þ
ELECTRONIC BACKUP
INTERNAL REFERENCE NAME:
HOST SYSTEM
Microsoft Windows
MEDIA
Type: Hard disk
Media Identification:
SOFTWARE
____________________________________________________________________________________________
ACASA/WP-1.8/210D
Version 1.3
Page ii
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
A
A.1
Comments on the ICAO encounter model
Introduction
The ACASA consortium found a number of difficulties during their
implementations of the standard ICAO encounter model. They are listed, and
very briefly described, in this Appendix. More complete details can be found
in the report to SICASP/7 [WP-1 109].
A.2
Technical problems encountered during implementation
A.2.1 Time of closest approach
The ICAO standard encounter model defines the trajectories in an encounter
with respect to a nominal time of closest approach between the two aircraft.
When the construction of each encounter is complete, the final time of closest
approach is only close to the initial, nominal value, and the constructed
encounter has to be rejected if the difference is too great. Using the time of
minimum horizontal separation as the reference point, which is what is done in
the new European ACAS safety encounter model, eliminates this curiosity,
and the vagueness of the test for acceptability.
A.2.2 Altitude distribution of encounters
The altitude distribution of encounters is defined in terms of discrete layers,
but the SARPs do not specify that encounters should be distributed uniformly
within each altitude layer. It is desirable that they should, for technical
reasons.
A.2.3 Floor and ceiling
The SARPs do not specify the lower limit of the lowest altitude layer and the
upper limit of the highest layer. These should be specified.
A.2.4
Ground speed
The ICAO specifications can result in ground speeds of less than 45kt, or more
than the speed of sound.
A.2.5
Turns
The way in which turns are specified can cause problems, mostly of
interpretation. It is possible to avoid these problems using the approach
adopted for the European ACAS safety encounter model.
A.2.6
Horizontal miss distance
There is an ambiguity in the construction of encounters that the ICAO
specification does not address. It corresponds to deciding whether the slower
aircraft passes behind the faster or in front of it.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 1
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
A.2.7
Vertical miss distance & crossing behaviour
The choice of vmd has to be made in a way that enables the generation of an
encounter with the correct altitude crossing characteristics. This is not
specified clearly in the SARPs. Furthermore, there is an ambiguity concerning
which aircraft is placed above the other for encounters where the aircraft do
not cross in altitude.
A.2.8
Negative altitudes
It is possible to construct trajectories with negative altitudes and the standards
do not state clearly how this feature is to be handled.
A.3
Operational realism of the model
A.3.1
NMACs
The encounter model carries with it an implied rate for the occurrence of
NMACs in the absence of ACAS. This should be the rate observed in the real
world, because the risk ratio is extremely sensitive to this single parameter.
The realism of the NMAC rate implied by the ICAO model was questioned
[WP-1 074]. The steps taken to ensure that this NMAC rate is realistic in the
European ACAS safety encounter model are discussed in section 3.5.
A.3.2
Correlation of the trajectories of the two aircraft
When one aircraft is climbing, the ICAO standard encounter model specifies
that other aircraft is equally likely to be climbing or descending, which is
manifestly unrealistic.
A.3.3 Correlation of horizontal and vertical manoeuvres
In the ICAO standard encounter model, the probability of a turn is
independent of whether the aircraft is climbing, descending or level.
A.3.4 Ground speeds
The distribution of ground speeds in the ICAO standard encounter model has
been found to be totally unrealistic for Europe.
A.3.5
Altitude distribution of encounters
The altitude distribution of encounters is not realistic for Europe. For example,
it contains too high a proportion of low-level encounters.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 2
ACAS PROGRAMME
ACASA PROJECT
Annex B to
Work Package 1
Final report on
Studies on the Safety of ACAS II
in Europe
Edition 1.0
March 2002
ACAS PROGRAMME, ACASA PROJECT
WP-1- Studies on the Safety of ACAS II in Europe
Annex B to
Work Package 1
Final Report on
Studies on the Safety of ACAS II
in Europe
Version 1.3, March 2002
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page i
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
TABLE OF CONTENTS
TABLE OF CONTENTS ......................................................................................................................................ii
B. Event probabilities ........................................................................................................................................... 1
B.1 Introduction .................................................................................................................................................... 1
B.2 Primary and secondary values........................................................................................................................ 1
B.3 Phases of the mandate..................................................................................................................................... 2
B.4 Description of the events................................................................................................................................. 2
B.4.1 Geometry events .......................................................................................................................................... 2
B.4.2 Equipment events......................................................................................................................................... 2
B.4.3 ATC events .................................................................................................................................................. 4
B.4.4 Human factors events................................................................................................................................... 4
B.4.5 Logic events ................................................................................................................................................ 6
B.5 Adopted values of the probabilities................................................................................................................. 8
B.5.1 Phase 1 probabilities .................................................................................................................................... 8
B.5.2 Phase 2 probabilities .................................................................................................................................. 11
____________________________________________________________________________________________
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page ii
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
B
Event probabilities
B.1
Introduction
B.1.1 The European ACAS safety encounter model developed as part of WP-1
enables risk ratios to be calculated that relate solely to the operation of the
ACAS collision avoidance logic and the reaction of pilots to the advisories it
generates. To evaluate the performance of ACAS when deployed in the real
world many other factors beyond the mere operation of the logic must be
included to arrive at a ‘full system’ risk ratio. To this end an ‘event tree’ was
developed, as mentioned in section 3.10, and whose structure is described in
[WP-1 197].
B.1.2 To enable the event tree to be exercised it was realised as an Excel spreadsheet
[WP-1 217]. It was then necessary to quantify the probabilities of the baselevel events in the tree, and this was done and reported in [WP-1 213].
B.1.3 This appendix summarises the base-level event probabilities: it elucidates their
context and indicates the values that were adopted. For a full discussion of the
determination of these values, and their sources, the reader is invited to refer
to the separate report [WP-1 213].
B.2
Primary and secondary values
B.2.1 Primary and secondary values of each probability, or set of complementary
probabilities, were determined.1 In each case the primary value was our best
estimate whilst our uncertainty in that estimate was expressed by choosing a
secondary value, that might be considered less likely but not improbable. The
secondary value was chosen to be pessimistic: i.e. adopting the value would
give a larger risk ratio relative to that resulting from adopting the primary
value. The values that were adopted are tabulated at the end of this appendix.
B.2.2 The sensitivity of the risk ratio to the uncertainty in each estimate was
analysed by adopting the secondary values in turn and noting the effect on the
risk ratio. This analysis is reported in an ACASA working paper [WP-1 216].
1
With one exception: being on a collision course is not an independent event.
Rather, the precise value of its probability is an assumption that is required for the
adjustment of the vmd distribution which in turn is used to generate the encounters
that are subsequently simulated to obtain the logic event probabilities that relate to
the operation of ACAS. The same value has been assumed in the calculation of the
both the primary and secondary values of the logic events, consequently no
secondary value of the probability of being on a collision course is given.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 1
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
B.3
Phases of the mandate
B.3.1 The European ACAS mandate is being introduced in two phases. The specific
requirements of the phases of the mandate directly affect the probability that a
randomly chosen aircraft in the airspace will be ACAS equipped. The
timescale of the phases of the mandate indirectly affect the likely transponder
equipage of a randomly chosen aircraft given that it is not equipped with
ACAS. Consequently two sets of values for these probabilities were
determined, corresponding to the two phases of the mandate.
B.4
Description of the events
B.4.1 Geometry events
GIMC – instrument meteorological conditions
The probability that visibility is such as to preclude the success of any attempt to
visually acquire the intruder, regardless of the relative disposition of the two aircraft.
GCCY – on collision course
The probability that, in an encounter considered by the event tree, two aircraft are on
a collision course.
GCCX – not on collision course
The converse of GCCY.
GLOS – there is no line of sight
The probability that an intruder is not within the field of view of the cockpit window
(or is obscured by some other part of the airframe) given that the aircraft are on a
collision course.
B.4.2 Equipment events
EAEY – aircraft is ACAS equipped
The probability that a given aircraft of interest is ACAS equipped (and is operating it
in full TA/RA mode).
EAEX – aircraft is not ACAS equipped
The converse of EAEY.
EMSY – aircraft is Mode S equipped
The probability that an aircraft that is not equipped with ACAS is equipped with a
Mode S transponder.
EMCY – aircraft is Mode C equipped
The probability that an aircraft that is not equipped with ACAS is equipped with a
Mode C transponder but not a Mode S transponder.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 2
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
ETXX – aircraft is not transponder equipped
The probability that an aircraft that is not equipped with ACAS is not equipped with a
Mode S transponder nor a Mode C transponder.
ETSY – ACAS tracks Mode S
The probability that ACAS surveillance will enable an intruder, which is not equipped
with ACAS but is equipped with a Mode S transponder, to be tracked.
ETSX – ACAS fails to track Mode S
The converse of ETSY.
ETCY – ACAS tracks Mode C
The probability that ACAS surveillance will enable an intruder, which is not equipped
with ACAS nor with a Mode S transponder but is equipped with a Mode C
transponder, to be tracked.
ETCX – ACAS fails to track Mode C
The converse of ETCY.
EDOY – display operational
The probability that the traffic display is in a state to be of use when attempting to
visually acquire a threat. A number of factors might make the display of no use: e.g.
hardware failure, sun glare, poorly adjusted brightness, inappropriate range scale
selected, and (in those installations in which the traffic display shares a screen with
other systems) the selection of another feature.
EDOX – no display
The converse of EDOY
EBDY – bearing data
The probability that the ACAS surveillance provides usable bearing data so that the
relative position of the threat can be shown on the traffic display.
EBDX – bearing data
The converse of EBDY.
EASY – altitude data from Mode S aircraft
The probability that an intruder, which is not equipped with ACAS but is equipped
with a Mode S transponder, will report altitude.
EASX – no altitude data from Mode S aircraft
The converse of EASY.
EACY – altitude data from Mode C aircraft
The probability that an intruder, which is not equipped with ACAS nor with a Mode S
transponder but is equipped with a Mode C transponder, will report altitude.
EACX – no altitude data from Mode C aircraft
The converse of EACY.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 3
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
B.4.3 ATC events
ACEY – there is a controller
The probability that an ACAS equipped aircraft, which receives an alert, is operating
with an air traffic control service.
ACEX – there is no controller
The converse of ACEY.
ACIY – controller is already involved
The probability that, when there is a controller, he is already involved in the encounter
(i.e. no avoidance instructions can be expected from the controller because he is ‘part
of the problem’).
ACIX – controller is not already involved
The converse of ACIY.
B.4.4 Human factors events
HRXA – controller instruction counter to RA when on collision course
The probability that, given there is a controller instruction when the aircraft are on a
collision course, the instruction will be counter to the RA.
HRXC – controller instruction counter to RA when not on collision course
The probability that, given there is a controller instruction when the aircraft are not on
a collision course and there is an RA, the instruction will be counter to the RA.
HCCY – pilot contacts controller
The probability that the pilot will contact the controller in response to an alert.
HCCX – pilot does not contact controller
The converse of HCCY.
HFAX – pilot does not look
The probability that the pilot does not conduct a visual search in response to an alert.
HFVX – pilot fails to act on visual
The probability that the pilot, having visually acquired the intruder, fails to act upon
the visual information.
HPRY – pilot responds to/notes RA
The probability that, given there is an RA, the pilot would respond to the RA in the
absence of a controller instruction.
HPRX – pilot ignores RA
The converse of HPRY.
HFCY – pilot follows/notes controller instruction
The probability that, given there is a controller instruction, the pilot would follow the
instruction in the absence of an RA.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 4
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
HFCX – pilot ignores controller instruction
The converse of HFCY.
HPCY – pilot prefers controller instruction
The probability that, given the pilot has received both an RA and a controller
instruction, she prefers the controller instruction and would have followed either had
it alone been received.
HPCX – pilot prefers RA
The converse of HPCY.
HPRP – pilot responds promptly
The probability that, given the pilot receives an RA and responds to it, she responds
with the standard, or even with a more vigorous, response.
HPRS – pilot responds slowly
The probability that, given the pilot receives an RA and responds to it, she responds
with a manoeuvre in the correct direction but too slowly or with insufficient strength.
HPRW – pilot responds with wrong sense
The probability that, given the pilot receives an RA and responds to it, she responds
with a manoeuvre that is in a direction contrary to the RA.
HAWD – pilot with display fails to acquire intruder
The probability that, given the aircraft are on a collision course and the pilot receives
an RA, she would fail to acquire the intruder in ideal viewing conditions (VMC and a
line of sight) with the aid of an available traffic display.
HAND – pilot without display fails to acquire intruder
The probability that, given the aircraft are on a collision course and the pilot receives
an RA, she would fail to acquire the intruder, in ideal viewing conditions (VMC and a
line of sight), without the aid of a traffic display.
HANA – pilot already has visual
The probability that, given the aircraft are on a collision course and the pilot receives
an RA, she would already visually acquired the intruder, in ideal viewing conditions,
by the time of the ACAS alert.
HVIW – visual information is wrong
The probability that, when a pilot believes she has visually acquired the threat, the
information is in some way incorrect. Amongst the factors to be considered here are
the possibility of visually acquired the wrong aircraft and, given acquisition of the
right aircraft, the misperception of its position and relative velocity.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 5
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
B.4.5 Logic events
Simulation based probabilities
B.4.5.1The majority of the logic events concern the interaction of the RAs generated
by the collision avoidance logic and the response of the pilot to those RAs. In
each case the value required is the probability that the RA will fail (i.e. a
collision results).
B.4.5.2Separate probabilities are determined for the case that the two aircraft are on a
collision course (ACAS fails to resolve the collision) and for the case that the
two aircraft are not originally on a collision course (ACAS induces a
collision).
B.4.5.3The probabilities are determined for the case of an ACAS aircraft
encountering an unequipped threat or for the case of two ACAS equipped
aircraft encountering each other. If the intruder is not ACAS equipped it may
be equipped with only a Mode C transponder (altitude reports in 100ft
increments) or may be equipped with a Mode S transponder (altitude reports in
25ft or 100ft increments). When two ACAS equipped aircraft encounter each
other, each aircraft may report altitude in either 25ft or 100ft increments:
additional the RAs in each aircraft will be co-ordinated.
B.4.5.4Four possible pilot reactions to RAs are considered: the pilot can ignore the
RA; the pilot can respond promptly; the pilot can respond slowly; or the pilot
can respond with the wrong sense. In encounters between two ACAS equipped
aircraft all combinations of these reactions are possible.
B.4.5.5Combining the possibilities outlined in the three previous paragraphs
generates 30 required probabilities. These have determined by analysis of
simulations of the performance of ACAS in encounters generated by the
European encounter model [WP-1 209] [WP-1 215].
Controller instruction probabilities
B.4.5.6The remaining four probabilities relate to the effect of controller instructions,
if followed.
LCIF – controller instruction fails to resolve collision
The probability, given the aircraft are on a collision course, that the pilot
contacts the controller, who provides an avoidance instruction, which if
correctly followed fails to resolve the collision.
LCIW – controller instruction prevents collision
The converse of LCIF.
LRXC – RA and controller instruction inconsistent: two evasive actions fail to
resolve collision
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 6
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
The probability, given the aircraft are on a collision course, that if one pilot
follows her RA but the other pilot follows a controller instruction and that
instruction is inconsistent with the RA, a collision will result.
LRXN – RA and controller instruction inconsistent: two evasive actions induce
collision
The probability, given the aircraft are not on a collision course, that if one pilot
follows her RA but the other pilot follows a controller instruction and that instruction
is inconsistent with the RA, a collision will be induced.
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 7
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
B.5
Adopted values of the probabilities
B.5.1 Phase 1 probabilities
The values that were adopted for the probabilities of the events, under Phase 1 of the
ACAS mandate, are tabulated below.
Geometry events
ref.
description
primary value
secondary value
GIMC
IMC
0.2000
0.3000
–3
GCCY on collision course
–
4.766´10
GCCX not on collision course
0.9952
–
GLOS
there is no line of sight
0.1926
0.2073
Equipment events
ref.
description
EAEY
aircraft ACAS equipped
EAEX
aircraft not ACAS equipped
EMCY aircraft Mode C equipped
EMSY aircraft Mode S equipped
ETXX
aircraft not transponder equipped
ETCY
ACAS tracks Mode C
ETCX
ACAS fails to track Mode C
ETSY
ACAS tracks Mode S
ETSX
ACAS fails to track Mode S
EDOY display operational
EDOX no display
EBDY bearing data
EBDX no bearing data
EACY altitude from Mode C aircraft
EACX no altitude from Mode C aircraft
EASY
altitude from Mode S aircraft
EASX
no altitude from Mode S aircraft
primary value
0.6290
0.3710
0.9030
0.0870
0.0100
0.9700
0.0300
0.9950
0.0050
0.9990
0.0010
0.9594
0.0406
0.9200
0.0800
0.9920
0.0080
secondary value
0.5949
0.4051
0.8969
0.0831
0.0200
0.9500
0.0500
0.9500
0.0500
0.9900
0.0100
0.7919
0.2081
0.7820
0.2180
0.9500
0.0500
ATC events
ref.
description
ACEY there is a controller
ACEX there is no controller
ACIY
controller is already involved
ACIX
controller is not already involved
primary value
0.9900
0.0100
0.2000
0.8000
secondary value
0.7778
0.2222
0.3333
0.6667
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 8
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
Human factors events
ref.
description
HRXA controller instruction counter to RA (on
collision course)
HRXC controller instruction counter to
unknown RA (not collision course)
HCCY pilot contacts controller
HCCX pilot does not contact controller
HFAX pilot does not look
HFVX pilot fails to act on visual
HPRY
pilot responds/notes RA
HPRX
pilot ignores RA
HFCY
pilot follows/notes controller instruction
HFCX
pilot ignores controller instruction
HPCY
pilot prefers controller instruction
HPCX
pilot prefers RA
HPRP
pilot responds promptly
HPRS
pilot responds slowly
HPRW pilot responds with wrong sense
HAWD pilot with display fails to acquire
intruder
HAND pilot without display fails to acquire
intruder
HANA pilot already has visual
HVIW visual information is wrong
Logic events
ref.
description
LSCC
RA fails to resolve collision,
ACAS (slow)/Mode C
LPCC
RA fails to resolve collision,
ACAS (prompt)/Mode C
LWCC RA fails to resolve collision,
ACAS (wrong)/Mode C
LSXC
RA fails to resolve collision,
ACAS (slow)/Mode S
LPXC
RA fails to resolve collision,
ACAS (prompt)/Mode S
LWXC RA fails to resolve collision,
ACAS (wrong)/Mode S
LSYC
RA fails to resolve collision,
ACAS (slow)/ACAS (no response)
LPYC
RA fails to resolve collision,
ACAS (prompt)/ACAS (no response)
LWYC RA fails to resolve collision,
ACAS (wrong)/ACAS (no response)
LSSC
RA fails to resolve collision,
ACAS (slow)/ACAS (slow)
ACASA/WP-1.8/210D
Version 1.3
primary value
0.0932
secondary value
0.2500
0.0725
0.1000
0.7500
0.2500
0.0500
0.0100
0.8214
0.1786
0.8446
0.1554
0.0883
0.9117
0.4546
0.5444
0.0010
0.0194
0.6462
0.3538
0.1000
0.0500
0.6618
0.3382
0.7391
0.2609
0.1160
0.8840
0.2400
0.6800
0.0800
0.1121
0.3719
0.6195
0.5105
0.0094
0.2639
0.0157
primary value
4.935´10–1
secondary value
5.413´10–1
9.200´10–2
9.728´10–2
3.592´10–1
4.080´10–1
4.841´10–1
5.309´10–1
8.873´10–2
9.209´10–2
3.592´10–1
4.080´10–1
1.494´10–1
1.768´10–1
4.180´10–2
4.947´10–2
3.592´10–1
4.080´10–1
2.021´10–1
2.216´10–1
March 2002
Page 9
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
RA fails to resolve collision,
ACAS (prompt)/ACAS (slow)
LWSC RA fails to resolve collision,
ACAS (wrong)/ACAS (slow)
LPPC
RA fails to resolve collision,
ACAS (prompt)/ACAS (prompt)
LWPC RA fails to resolve collision,
ACAS (wrong)/ACAS (prompt)
LWWC RA fails to resolve collision,
ACAS (wrong)/ACAS (wrong)
LSCN
RA induces collision,
ACAS (slow)/Mode C
LPCN
RA induces collision,
ACAS (prompt)/Mode C
LWCN RA induces collision,
ACAS (wrong)/Mode C
LSXN
RA induces collision,
ACAS (slow)/Mode S
LPXN
RA induces collision,
ACAS (prompt)/Mode S
LWXN RA induces collision,
ACAS (wrong)/Mode S
LSYN
RA induces collision,
ACAS (slow)/ACAS (no response)
LPYN
RA induces collision,
ACAS (prompt)/ACAS (no response)
LWYN RA induces collision,
ACAS (wrong)/ACAS (no response)
LSSN
RA induces collision,
ACAS (slow)/ACAS (slow)
LPSN
RA induces collision,
ACAS (prompt)/ACAS (slow)
LWSN RA induces collision,
ACAS (wrong)/ACAS (slow)
LPPN
RA induces collision, ACAS
(prompt)/ACAS (prompt)
LWPN RA induces collision,
ACAS (wrong)/ACAS (prompt)
LWWN RA induces collision,
ACAS (wrong)/ACAS (wrong)
3.688´10–2
4.237´10–2
3.718´10–1
4.078´10–1
1.130´10–2
1.356´10–2
3.691´10–1
4.070´10–1
3.592´10–1
4.080´10–1
1.358´10–3
1.489´10–3
7.057´10–4
7.462´10–4
3.186´10–2
3.619´10–2
1.073´10–3
1.177´10–3
5.676´10–4
5.896´10–4
3.186´10–2
3.619´10–2
2.361´10–3
2.794´10–3
5.033´10–4
9.520´10–4
3.186´10–2
3.619´10–2
1.614´10–3
1.771´10–3
1.952´10–4
2.073´10–4
1.966´10–2
2.156´10–2
6.656´10–5
6.806´10–5
1.674´10–2
1.846´10–2
3.186´10–2
3.619´10–2
LCIF
LCIW
LRXC
LRXN
0.0920
0.9080
0.3691
0.0167
0.0973
0.9027
0.4070
0.0185
LPSC
controller instruction fails
controller instruction works
two evasive actions cancel
two actions produce collision
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 10
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
B.5.2 Phase 2 probabilities
The majority of the event probabilities remain the same when Phase 2 is modelled.
However, as mentioned in section B.3, two of the equipment probabilities are directly
affected by the mandate and three are indirectly affected. The increase in the level of
ACAS equipage also implies an improvement in altimetry, and thus reduces the
probability of being on a collision course. These different values are tabulated below.
Geometry events (Phase 2)
ref.
GCCY
GCCX
description
on collision course
not on collision course
primary value
4.607´10–3
0.9954
secondary value
–
–
primary value
0.7015
0.2985
0.5940
0.3960
0.0100
secondary value
0.6709
0.3291
0.6270
0.3530
0.0200
Equipment events (Phase 2)
ref.
EAEY
EAEX
EMCY
EMSY
ETXX
description
aircraft ACAS equipped
aircraft not ACAS equipped
aircraft Mode C equipped
aircraft Mode S equipped
aircraft not transponder equipped
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page 11
ACAS PROGRAMME
ACASA PROJECT
Annex C to
Work Package 1
Final report on
Studies on the Safety of ACAS II
in Europe
Edition 1.0
March 2002
ACAS PROGRAMME, ACASA PROJECT
WP-1- Studies on the safety of ACAS II in Europe
Annex C to
Work Package 1
Final Report on
Studies on the Safety of ACAS II
in Europe
Version 1.3, March 2002
ACASA/WP1.8/210D
Version 1.3
March 2002
Page i
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 – Studies on the safety of ACAS II in Europe
TABLE OF CONTENTS
TABLE OF CONTENTS ......................................................................................................................................ii
C ACASA WP-1 working papers................................................................................................................ 1-2-3-4
____________________________________________________________________________________________
ACASA/WP-1.8/210D
Version 1.3
March 2002
Page ii
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
C
ACASA WP-1 working papers
Reference is made to the papers in this list using the notation [WP-1 nnn]. Other references require only one or two digits.
work
package
#
paper title
authorship
005
Preliminary results for WP-1.2
Thierry Arino
1.2 1.0
3 September 1998
024
Preliminary results for WP1.6 ACAS safety study collision risk fault tree
Gillian Yates
1.6 1.0
10 March 1999
027
Specification of follow-on work for 1.1.2
Kevin Rigotti
1.1 1.0
27 January 1999
029
Analytical expression of horizontal positions during accelerated turn
Thierry Miquel
1.1 1.0
29 March 1999
032
The ACASA Safety Study: Progress to date…(SICASP WG2 paper 745)
Ken Carpenter
035
Proposed modifications to avoid parameter reselection
Kevin Rigotti
1.1 1.0
28 May 1999
036
Investigation of the effects of modelling altimetry error by analytical and stochastic approaches
Gillian Yates
1.1 2.0
21 June 1999
038
Changes required in the ICAO encounter model to fit it for Europe
Ken Carpenter
1.1 1.0
28 May 1999
041
Comments on the proposed modifications to avoid parameter reselection
Thierry Miquel
1.1 1.0
31 May 1999
052
Status on the repetition of the ICAO risk ratio calculations
Thierry Arino & Christian Aveneau
1.1 1.0
20 July 1999
062
Source of discrepancy between DERA and CENA implementations of the ICAO encounter model
Thierry Miquel
1.1 1.1
28 August 1999
065
Updated status on the repetition of the ICAO risk ratio calculations
Thierry Arino
1.1 1.0
26 August 1999
067
Two encounter model issues
Ken Carpenter
1.1 1.0
13 August 1999
068
We can do it two ways
Ken Carpenter
1.1 1.0
19 August 1999
069
What radar data for the encounter model?
Ken Carpenter
1.1 1.1
16 September 1999
070
Plans for the construction of the European encounter model
Ken Carpenter
1.1 1.4
15 September 1999
073
Differences between the CENA and DERA: calculations of risk ratio
Ken Carpenter & Harry Hutchinson
1.1 1.1
24 September 1999
074
Some considerations on the encounter model effect on risk ratio figures
Thierry Arino & Thierry Miquel
1.1 1.0
22 October 1999
075
Illustration of various encounter selection criteria
Béatrice Bonnemaison
1.1 1.0
18 November 1999
078
Capture criteria for encounters in radar data
Harry Hutchinson & Ken Carpenter
1.1 1.3
22 November 1999
ACASA/WP-1.8/210D
Version 1.3
version
date
28 April 1999
Page -1-
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
080
Fault tree for unresolved risk ratio
Ken Carpenter
1.6 0.1
18 November 1999
081
Risk ratio calculations based on the standard encounter model (SICASP WG2/809)
Gillian Yates & Harry Hutchinson
1.1 1.0
21 October 1999
082
Review of draft fault tree
Thierry Arino
1.6 1.0
22 November 1999
085
Radar data processing specifications
Ken Carpenter
1.1 0.4
30 November 2000
086
Specifications of the back-end software to derive parameters for the safety model
Thierry Miquel
1.1 1.8
5 December 2000
088
Comments on the CENA review of fault tree for unresolved collision risk ratio
Ken Carpenter
1.6 0.1
17 December 1999
089
DERA radar data processing specifications
Ken Carpenter
1.1 0.2
26 February 2000
090
CENA radar data processing for the European Encounter Model
Bernard Gayraud
1.1 1.0
7 February 2000
091
Radar data requirements.
Ken Carpenter
1.1
093
Plans for WP1.7: human reaction to ACAS alerts and displays
Ken Carpenter
1.7 0.1
26 January 2000
094
Risk ratio models
Ken Carpenter
1.1 0.1
14 February 2000
097
Selected encounters from French radar data
Béatrice Bonnemaison
1.1 1.0
15 February 2000
101
Draft specification of the European Encounter Model – a précis.
Ken Carpenter
1.1 1.0
26 February 2000
102
Human response probabilities
Ken Carpenter
1.7 1.0
8 March 2000
109
Experience in the use of the ICAO ACAS encounter model (SICASP WG2/841 & (SICASP/7 WP8)
presented by John Law
1.1 1.1
March 2000
115
NMAC Rate
Ken Carpenter
1.1 2.3
19 November 2000
116
CENA Safety Encounter Model based on the French radar data
Béatrice Bonnemaison
1.1 2.1
26 September 2000
117
WP-1.7: revised work plan
Thierry Arino
1.7 1.0
14 June 2000
126
From an observed encounter to simulated encounters
Thierry Miquel
1.1 1.0
6 September 2000
127
NMAC rate for the current “Set a” of French radar data recordings
Thierry Miquel
1.1 1.4
4 December 2000
130
ACAS and multiple encounters: use of radar recorded encounter data
Ken Carpenter
1.5 1.1
28 September 2000
140
Exact solution of contributions to NMAC rate due to altimetry error
Harry Hutchinson
1.1 1.0
6 October 2000
142
Work plan for WP-1.3: effect of surveillance performance on the ACAS logic
Thierry Miquel
1.3 2.1
18 January 2001
ACASA/WP-1.8/210D
Version 1.3
1
2 December 1999
Page -2-
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
145
Work plan and methodology for the assessment of the benefits of fitting ACAS to smaller aircraft
146
TCAS operational performance on multiple-aircraft encounters
148
Study of UK and French radar data differences
151
1.2 2.0
1 December 2000
1.5 2.0
20 November 2000
Stéphan Chabert
1.1 1.1
20 November 2000
Review of methods for adjusting weightings
Harry Hutchinson
1.1 2.0
1 December 2000
155
Specifications of ACAS surveillance error model
Thierry Miquel
1.3 2.1
12 March 2001
158
Adjusting the vmd bin distributions “in anger”
Harry Hutchinson
1.1 0.1
6 February 2001
160
CENA/Sofréavia risk ratio calculations with the European encounter model
Thierry Arino
1.1 1.0
11 January 2001
161
Analysis of pilot reactions based on airborne recorded data
Eric Vallauri
1.7 1.1
2 March 2001
169
WP-1.2 extension: work plan for airborne risk ratios
Thierry Arino
1.2 1.0
21 February 2001
171
Pilot model specification and risk ratio calculation
Eric Vallauri
1.7 1.1
16 March 2001
172
Assessment of the benefit of fitting ACAS to smaller aircraft
Stéphan Chabert
1.2 1.0
30 March 2001
173
Report on the effect of imperfect surveillance performance on the efficacy of ACAS
Thierry Arino
1.3 1.0
5 April 2001
182
Effect on risk ratio of removing aircraft type modelling from the European Encounter Model.
Thierry Arino
1.1 2.0
2 July 2001
183
CENA’s view on the VMD distribution adjustment algorithms
Thierry Miquel
1.1 1.0
25 June 2001
184
Methodology and workplan for the assessment of the benefits of fitting ACAS to smaller aircraft.
Stéphan Chabert
1.2 1.0
28 June 2001
186
European Encounter Model: specifications and probability tables
Thierry Miquel & Kevin Rigotti
1.1 2.1
4 December 2001
187
Problem encounter geometry
Harry Hutchinson & Ken Carpenter
1.1 1.0
20 July 2000
192
ACAS and multiple encounters: use of radar recorded encounter data
Bill Booth & Ken Carpenter
1.5 0.3
10 July 2001
193
Estimating the frequency of events
Ken Carpenter
1.5 1.0
20 July 2001
197
Notes on the event tree for the ACAS collision risk ratio
Ken Carpenter
1.6 2.2
12 September 2001
204
Assessment of the benefits of fitting ACAS to smaller aircraft: the airborne risk ratio approach
Stéphan Chabert
1.2 1.0
10 September 2001
205
Timing of a late controller instruction
Ken Carpenter
1.6 1.0
2 August 2001
207
Aircraft centred fault tree
Ken Carpenter
1.6 1.0
30 July 2001
ACASA/WP-1.8/210D
Version 1.3
Stéphan Chabert
Bernard Hasquenoph & Stéphan Chabert
Page -3-
ACAS PROGRAMME, ACASA PROJECT
Work Package 1 - Studies on the safety of ACAS II in Europe
208
Pilot response models
Harry Hutchinson
1.6 0.2
8 August 2001
209
Determination of the probabilities of ACAS logic events
Harry Hutchinson
1.6 1.1
5 September 2001
212
Multiple encounter analysis
Harry Hutchinson, James Yau & Ken Carpenter
1.5 1.0
1 November 2001
213
Event tree probabilities
Harry Hutchinson
1.6 3.0
12 November 2001
215
Logic simulations
Harry Hutchinson
1.6 1.0
17 October 2001
216
Sensitivity analysis of risk ratios using the event tree
Harry Hutchinson
1.6 2.0
12 November 2001
217
Event tree: Excel spreadsheet implementation
ATC Systems Group, Malvern
1.6 4.0
13 December 2001
ACASA/WP-1.8/210D
Version 1.3
Page -4-
Was this manual useful for you? yes no
Thank you for your participation!

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement