What Matters in the Effectiveness of Airborne Collision Avoidance Systems? Monte Carlo Simulation of Uncertainties for TCAS II and ACAS Xa

TCAS II is a rule-based airborne collision avoidance system (ACAS) that is used in current commercial air transport operations, and ACAS Xa is a new optimization-based system. Operational validation studies have mainly used deterministic simulations of ACAS performance using various sets of encounters. Recently a new approach was developed, which employs Monte Carlo (MC) simulation of agent-based models to evaluate the impact of sensor errors and pilot response variability. This paper contrasts the results of both approaches in a comparison of TCAS II and ACAS Xa for various types of synthetic encounters. It was found that conventional estimates of near mid-air collision (NMAC) probabilities are often lower than the estimates achieved using MC simulation, and that the biases in the P(NMAC) estimates are consistently larger for ACAS Xa than for TCAS II. Contributions to unresolved risk are largest for pilot performance, then for encounter types, and finally for sensor errors. The contribution of non-responding pilots is much larger than the differences between TCAS II and ACAS Xa. It is concluded that the agent-based MC simulation overcomes the limitations in traditional evaluation of altimetry errors and pilot response, providing an independent means to effectively analyze the robustness of ACASs.