Air traffic controllers' mental fatigue recognition: A multi-sensor information fusion-based deep learning approach

With the growing density of air passenger traffic, accurately recognizing the level of mental fatigue (MF) experienced by air traffic controllers (ATCOs) is crucial for developing intelligent ATCOs' mental state monitoring systems, which can achieve a more effective and safer human–machine cooperative pattern. However, the existing methods for recognizing ATCOs' MF face significant challenges due to pattern variations between ATCOs and sensor artifacts. This study introduces a framework for ATCOs' MF recognition, utilizing a deep neural network called RecMF, which incorporates multi-sensor information fusion to enhance the performance of MF detection. Specifically, the RecMF employs an attention-enabled CNN-LSTM architecture that simultaneously captures time-series feature representations of electroencephalogram (EEG) signals and eye movements. To validate the effectiveness of RecMF, a fatigue-inducing experiment is conducted involving 28 subjects who are tasked with performing a series of air traffic control (ATC) tasks. The model's performance is evaluated across various time horizons and typical cognitive tasks to gain insights into its capabilities. The evaluation results indicate that the proposed model outperforms other existing methods, thereby confirming its feasibility and effectiveness. Additionally, the effects of MF on ATCOs' cognitive performance are analyzed using analysis of variance (ANOVA). The results reveal that higher levels of MF significantly reduce ATCOs' reaction speed and operational accuracy.