Abstract
Flight safety remains a paramount concern in the aviation industry, with pilots’ mental workload playing a significant role, particularly during the approach and landing phase under various weather conditions. Monitoring pilots’ mental workload in a real-time and noncontact approach is essential for preventing errors and improving flight safety. This study employs noncontact facial expression analysis using advanced sensor technology, specifically FaceReader and GoPro camera sensors, to classify pilots’ mental workload during simulated approach and landing in diverse weather scenarios. Data were collected from 21 pilot cadets using a Cessna 172 simulator under three specific conditions: daytime with crosswind, nighttime with crosswind, and daytime without crosswind. The NASA-TLX scale was employed to assess subjective mental workloads. Deep learning and machine learning models including Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), K-nearest Neighbors (KNN), Decision Tree (DT), Quadratic Discriminant Analysis (QDA), Linear Discriminant Analysis (LDA), and Naive Bayes (NB) were trained on the FaceReader data. The CNN model achieved the highest performance, with an accuracy of 99.87%. SHAP (SHapley Additive exPlanations) values were utilized to interpret the model outputs, identifying ‘Fearful’, ' Lips Part ‘, and ' Chin Raiser ' as critical features. This study demonstrates the efficacy of noncontact data collection using camera sensor technology and advanced machine learning techniques in classifying mental workloads, which will enhance flight safety and improve pilot performance. Future research will focus on validating these findings in real-world scenarios and exploring the temporal evolution of mental workloads.
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03619-x/MediaObjects/11760_2024_3619_Fig1_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03619-x/MediaObjects/11760_2024_3619_Fig2_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03619-x/MediaObjects/11760_2024_3619_Fig3_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03619-x/MediaObjects/11760_2024_3619_Fig4_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03619-x/MediaObjects/11760_2024_3619_Fig5_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03619-x/MediaObjects/11760_2024_3619_Fig6_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03619-x/MediaObjects/11760_2024_3619_Fig7_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03619-x/MediaObjects/11760_2024_3619_Fig9_HTML.png)
![](http://media.springernature.com/m312/springer-static/image/art%3A10.1007%2Fs11760-024-03619-x/MediaObjects/11760_2024_3619_Fig8_HTML.png)
Similar content being viewed by others
Data availability
No datasets were generated or analysed during the current study.
References
Park, P., Di Marco, P., Nah, J., Fischione, C.: Wireless avionics intracommunications: A survey of benefits, challenges, and solutions. IEEE Internet Things J. 8(10), 7745–7767 (2020)
Shekhar, A.C., Blumen, I.J.: Fatal air medical accidents in the United States (2000–2020). Prehosp. Disaster Med. 38(2), 259–263 (2023)
Orru, G., Longo, L.: The evolution of cognitive load theory and the measurement of its intrinsic, extraneous and germane loads: a review. Human Mental Workload: Models and Applications: Second International Symposium, H-WORKLOAD. Amsterdam, The Netherlands, September 20–21, 2018, Revised Selected Papers 2: Springer; 2019. pp. 23–48. (2018)
van Ruysseveldt, J., van Dijke, M.: When are workload and workplace learning opportunities related in a curvilinear manner? The moderating role of autonomy. J. Vocat. Behav. 79(2), 470–483 (2011)
Taheri Gorji, H., Wilson, N., VanBree, J., Hoffmann, B., Petros, T., Tavakolian, K.: Using machine learning methods and EEG to discriminate aircraft pilot cognitive workload during flight. Sci. Rep. 13(1), 2507 (2023)
Wang, H., Jiang, N., Pan, T., Si, H., Li, Y., Zou, W.: Cognitive load identification of pilots based on physiological-psychological characteristics in Complex environments. J. Adv. Transp. 2020(1), 5640784 (2020)
Gao, S., Wang, L.: Effects of mental workload and risk perception on pilots’ safety performance in adverse weather contexts. Engineering Psychology and Cognitive Ergonomics Cognition and Design: 17th International Conference, EPCE. Held as Part of the 22nd HCI International Conference, HCII 2020, Copenhagen, Denmark, July 19–24, 2020, Proceedings, Part II 22: Springer; 2020. pp. 278 – 91. (2020)
McKendrick, R., Harwood, A.: Cognitive workload and workload transitions elicit curvilinear hemodynamics during spatial working memory. Front. Hum. Neurosci. 13, 405 (2019)
Minoretti, P., Serrano, M.G., Riera, M.L., Sáez, A.S., Martín, Á.G.: Occupational Health Challenges for Aviation workers amid the changing climate: A narrative review. Cureus ;16(3). (2024)
Zhang, C., Liu, C., Liu, H., Jiang, C., Fu, L., Wen, C., et al.: Incorporation of pilot factors into risk analysis of Civil Aviation accidents from 2008 to 2020: A data-driven bayesian Network Approach. Aerospace. 10(1), 9 (2022)
Sharma, L.D., Saraswat, R.K., Sunkaria, R.K.: Cognitive performance detection using entropy-based features and lead-specific approach. Signal. Image Video Process. 15(8), 1821–1828 (2021)
Charles, R.L., Nixon, J.: Measuring mental workload using physiological measures: A systematic review. Appl. Ergon. 74, 221–232 (2019)
Lee, D.S., Chong, T.W., Lee, B.G.: Stress events detection of driver by wearable glove system. IEEE Sens. J. 17(1), 194–204 (2016)
Uhlarik, J., Comerford, D.A.: A review of situation awareness literature relevant to pilot surveillance functions. (2002)
Magnusdottir, E.H., Johannsdottir, K.R., Majumdar, A., Gudnason, J.: Assessing Cognitive workload using Cardiovascular measures and Voice. Sensors. 22(18), 6894 (2022)
Boehm, U., Matzke, D., Gretton, M., Castro, S., Cooper, J., Skinner, M., et al.: Real-time prediction of short-timescale fluctuations in cognitive workload. Cogn. Research: Principles Implications. 6, 1–29 (2021)
Khan, M. A., Asadi, H., Zhang, L., Qazani, M. R. C., Oladazimi, S., Kiong, L. C., ... Nahavandi, S.: Application of artificial intelligence in cognitive load analysis using functional near-infrared spectroscopy: A systematic review. Expert Syst. Appl. 123717 (2024)
Zhang, C., Yuan, J., Jiao, Y., Liu, H., Fu, L., Jiang, C., et al.: Variation of Pilots’ Mental Workload under Emergency Flight conditions Induced by different equipment failures: A Flight Simulator Study. Transp. Res. Rec. 2678(4), 365–377 (2024)
Cao, X., MacNaughton, P., Cadet, L.R., Cedeno-Laurent, J.G., Flanigan, S., Vallarino, J., et al.: Heart rate variability and performance of commercial airline pilots during flight simulations. Int. J. Environ. Res. Public Health. 16(2), 237 (2019)
Innes, R.J., Howard, Z.L., Thorpe, A., Eidels, A., Brown, S.D.: The effects of increased visual information on cognitive workload in a helicopter simulator. Hum. Factors. 63(5), 788–803 (2021)
Ren, B., Zhou, Q., Chen, J.: Assessing cognitive mental workloads of assembly workers during multi-task switching. (2023)
Knisely, B.M., Joyner, J.S., Vaughn-Cooke, M.: Cognitive task analysis and workload classification. MethodsX. 8, 101235 (2021)
Ahlstrom, U., Caddigan, E., Schulz, K., Ohneiser, O., Bastholm, R., Dworsky, M.: The Effect of Weather State-Change Notifications on General Aviation Pilots’ Behavior, Cognitive Engagement, and Weather Situation Awareness. (2015)
Aygun, A., Nguyen, T., Haga, Z., Aeron, S., Scheutz, M.: Investigating methods for cognitive workload estimation for assistive robots. Sensors. 22(18), 6834 (2022)
Hancock, P.A., Matthews, G.: Workload and performance: Associations, insensitivities, and dissociations. Hum. Factors. 61(3), 374–392 (2019)
Seshadri, D.R., Li, R.T., Voos, J.E., Rowbottom, J.R., Alfes, C.M., Zorman, C.A., et al.: Wearable sensors for monitoring the internal and external workload of the athlete. NPJ Digit. Med. 2(1), 71 (2019)
Golenia, J.-E., Wenzel, M.A., Bogojeski, M., Blankertz, B.: Implicit relevance feedback from electroencephalography and eye tracking in image search. J. Neural Eng. 15(2), 026002 (2018)
Hetland, A., Kjelstrup, E., Mittner, M., Vittersø, J.: The thrill of speedy descents: A pilot study on differences in facially expressed online emotions and retrospective measures of emotions during a downhill mountain-bike descent. Front. Psychol. 10, 370400 (2019)
Hetland, A.: Feeling and thinking about it are two different things: How to capture momentary emotions of Extreme sports in the field. Int. J. Environ. Res. Public Health. 19(3), 1290 (2022)
Xu, N., Guo, G., Lai, H., Chen, H.: Usability study of two in-vehicle information systems using finger tracking and facial expression recognition technology. Int. J. Human–Computer Interact. 34(11), 1032–1044 (2018)
Skaramagkas, V., Giannakakis, G., Ktistakis, E., Manousos, D., Karatzanis, I., Tachos, N.S., et al.: Review of eye tracking metrics involved in emotional and cognitive processes. IEEE Rev. Biomed. Eng. 16, 260–277 (2021)
Lapierre, H.G., Charland, P., Léger, P.-M.: Looking under the hood of learning computer programming: The emotional and cognitive differences between novices and beginners. Comput. Sci. Educ. 34(3), 331–352 (2024)
Sharma, K., Niforatos, E., Giannakos, M., Kostakos, V.: Assessing cognitive performance using physiological and facial features: Generalizing across contexts. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4(3), 1–41 (2020)
Nie, Y., Tong, S., Li, J., Zhang, Y., Zheng, C., Fan, B.: Time identification of design knowledge push based on cognitive load measurement. Adv. Eng. Inform. 54, 101783 (2022)
Dehais, F., Duprès, A., Blum, S., Drougard, N., Scannella, S., Roy, R.N., et al.: Monitoring pilot’s mental workload using ERPs and spectral power with a six-dry-electrode EEG system in real flight conditions. Sensors. 19(6), 1324 (2019)
Li, B., Tong, L., Zhang, C., Chen, P., Wang, L., Yan, B.: Prediction of image interpretation cognitive ability under different mental workloads: A task-state fMRI study. Cereb. Cortex. 34(3), bhae100 (2024)
Xi, P., Law, A., Goubran, R., Shu, C.: Pilot workload prediction from ECG using deep convolutional neural networks. 2019 IEEE International Symposium on Medical Measurements and Applications (MeMeA): IEEE; pp. 1–6. (2019)
Belkhiria, C., Peysakhovich, V.: EOG metrics for cognitive workload detection. Procedia Comput. Sci. 192, 1875–1884 (2021)
Feng, C., Wanyan, X., Yang, K., Zhuang, D., Wu, X.: A comprehensive prediction and evaluation method of pilot workload. Technol. Health Care. 26(S1), 65–78 (2018)
Park, J., Berman, J., Dodson, A., Liu, Y., Armstrong, M., Huang, H., et al.: Assessing workload in using electromyography (EMG)-based prostheses. Ergonomics. 67(2), 257–273 (2024)
Zhang, H., Wang, Z., Zhuang, Y., Yin, S., Chen, Z., Liang, Y.: Assessment of Mental Workload Level based on PPG Signal Fusion Continuous Wavelet Transform and cardiopulmonary Coupling Technology. Electronics. 13(7), 1238 (2024)
Liu, C., Zhang, C., Sun, L., Liu, K., Liu, H., Zhu, W., et al.: Detection of pilot’s mental workload using a wireless EEG headset in airfield traffic pattern tasks. Entropy. 25(7), 1035 (2023)
von Janczewski, N., Kraus, J., Engeln, A., Baumann, M.: A subjective one-item measure based on NASA-TLX to assess cognitive workload in driver-vehicle interaction. Transp. Res. part. F: Traffic Psychol. Behav. 86, 210–225 (2022)
Albadr, M.A.A., Tiun, S., Ayob, M., Al-Dhief, F.T., Omar, K., Hamzah, F.A.: Optimised genetic algorithm-extreme learning machine approach for automatic COVID-19 detection. PloS One. 15(12), e0242899 (2020)
Albadr, M.A.A., Tiun, S., Ayob, M., Al-Dhief, F.T.: Particle swarm optimization-based extreme learning machine for covid-19 detection. Cogn. Comput. 16(4), 1858–1873 (2024)
Rahman, M.A., Hamada, M., Sharmin, S., Rimi, T.A., Talukder, A.S., Imran, N., ... Ali, M. A.: Enhancing early breast cancer detection through advanced data analysis. IEEE Access. (2024)
Dickinson, Q., Meyer, J.G.: Positional SHAP (PoSHAP) for interpretation of machine learning models trained from biological sequences. PLoS Comput. Biol. 18(1), e1009736 (2022)
Speirs, A., Ramée, C., Payan, A.P., Mavris, D., Feigh, K.M.: Impact of adverse weather on commercial helicopter pilot decision-making and standard operating procedures. AIAA Aviation 2021 Forum2021. p. 2771
Kosch, T., Karolus, J., Zagermann, J., Reiterer, H., Schmidt, A., Woźniak, P.W.: A survey on measuring cognitive workload in human-computer interaction. ACM Comput. Surveys. 55(13s), 1–39 (2023)
Longo, L.: Experienced mental workload, perception of usability, their interaction and impact on task performance. PloS One. 13(8), e0199661 (2018)
Albadr, M.A.A., Ayob, M., Tiun, S., Al-Dhief, F.T., Arram, A., Khalaf, S.: Breast cancer diagnosis using the fast learning network algorithm. Front. Oncol. 13, 1150840 (2023)
Duncan, J.A., Alambeigi, F., Pryor, M.W.: A Survey of Multimodal Perception Methods for Human-Robot Interaction in Social Environments. ACM Transactions on Human-Robot Interaction (2024)
Wang, C., Guragain, B., Verma, A.K., Archer, L., Majumder, S., Mohamud, A., et al.: Spectral analysis of EEG during microsleep events annotated via driver monitoring system to characterize drowsiness. IEEE Trans. Aerosp. Electron. Syst. 56(2), 1346–1356 (2019)
Schindler, S., Caldarone, F., Bruchmann, M., Moeck, R., Straube, T.: Time-dependent effects of perceptual load on processing fearful and neutral faces. Neuropsychologia. 146, 107529 (2020)
Gu, Y., Weng, Y., Wang, Y., Wang, M., Zhuang, G., Huang, J., ... Ren, F.: EmoTake: Exploring Drivers’ Emotion for Takeover Behavior Prediction. IEEE Trans. Affect. Comput. (2024)
Kerdvibulvech, C.: A digital human emotion modeling application using metaverse technology in the post-COVID-19 era. International Conference on Human-Computer Interaction: Springer; pp. 480-9. (2023)
Acknowledgements
The authors wish to express their gratitude for the collaborative efforts that contributed to the completion of this paper. We are grateful for the participation of the pilot cadets from the Flight Technology College of Civil Aviation Flight University of China and the support of Noldus Information Technology BV.
Funding
This work was supported by the China Scholarship Council (No. 202307000068); the Open Fund of Key Laboratory of Flight Techniques and Flight Safety, CAAC (No. FZ2021KF09); the Special Fund of Key Laboratory of Flight Techniques and Flight Safety, CAAC (No. FZ2022ZX04 and FZ2022ZX54).
Author information
Authors and Affiliations
Contributions
The authors confirm contribution to the paper as follows: study conception and design: Chenyang Zhang, Chaozhe Jiang; data collection: Chenyang Zhang, Shihan Luo, Jiajun Yuan; analysis and interpretation of results: Wenbing Zhu, Shihan Luo, Hua Chen; draft manuscript preparation: Qinyang Li, Tong Wang, Chenyang Zhang, Shihan Luo. All authors reviewed the results and approved the final version of the manuscript.
Corresponding author
Ethics declarations
Ethical approval
The study adhered to the principles of the Helsinki Declaration and was approved by the Ethical Review Board of Southwest Jiaotong University (SWJTU-2109-001-QT). Written informed consent was obtained from all the participants prior for the experiments of this study.
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Luo, S., Zhang, C., Zhu, W. et al. Noncontact perception for assessing pilot mental workload during the approach and landing under various weather conditions. SIViP 19, 98 (2025). https://doi.org/10.1007/s11760-024-03619-x
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11760-024-03619-x