skip to main content
10.1145/3655532.3655539acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicrsaConference Proceedingsconference-collections
research-article
Open access

A Panoramic Review of Situational Awareness Monitoring Systems

Published: 28 June 2024 Publication History

Abstract

Autonomous vehicles require a human-machine interaction (HMI) system for safe control transition during takeover requests (TORs). Vital to the HMI system is real-time monitoring of drivers’ situational awareness, across diverse scenarios, without pre-calibration. This paper reviews situational awareness metrics, and considers sensor and computational model limitations, highlighting current gaps. We present a classification model categorizing awareness measurement methods—session-specific, online, offline—suitable for various applications, aiding researchers in choosing appropriate models.

References

[1]
Jibran A Abbasi, Darragh Mullins, Nicolas Ringelstein, Patrice Reilhac, Edward Jones, and Martin Glavin. 2021. An Analysis of Driver Gaze Behaviour at Roundabouts. IEEE Transactions on Intelligent Transportation Systems (2021).
[2]
Amirhossein S Aghaei, Birsen Donmez, Cheng Chen Liu, Dengbo He, George Liu, Konstantinos N Plataniotis, Huei-Yen Winnie Chen, and Zohreh Sojoudi. 2016. Smart driver monitoring: when signal processing meets human factors: in the driver’s seat. IEEE Signal Processing Magazine 33, 6 (2016), 35–48.
[3]
Shubham Agrawal and Srinivas Peeta. 2021. Evaluating the impacts of situational awareness and mental stress on takeover performance under conditional automation. Transportation research part F: traffic psychology and behaviour 83 (2021), 210–225.
[4]
Ioannis Agtzidis, Mikhail Startsev, and Michael Dorr. 2019. A ground-truth data set and a classification algorithm for eye movements in 360-degree videos. arXiv preprint arXiv:1903.06474 (2019).
[5]
Sangtae Ahn, Thien Nguyen, Hyojung Jang, Jae G Kim, and Sung C Jun. 2016. Exploring neuro-physiological correlates of drivers’ mental fatigue caused by sleep deprivation using simultaneous EEG, ECG, and fNIRS data. Frontiers in human neuroscience 10 (2016), 219.
[6]
Pavlo Antonenko, Fred Paas, Roland Grabner, and Tamara Van Gog. 2010. Using electroencephalography to measure cognitive load. Educational psychology review 22, 4 (2010), 425–438.
[7]
Tobias Appel, Natalia Sevcenko, Franz Wortha, Katerina Tsarava, Korbinian Moeller, Manuel Ninaus, Enkelejda Kasneci, and Peter Gerjets. 2019. Predicting cognitive load in an emergency simulation based on behavioral and physiological measures. In 2019 International Conference on Multimodal Interaction. 154–163.
[8]
Chris Berka, Daniel J Levendowski, Gene Davis, Melissa Whitmoyer, Kelly Hale, and K Fuchs. 2006. Objective measures of situational awareness using neurophysiology technology. Augmented Cognition: Past, Present and Future (2006), 145–154.
[9]
Miriam Bongo and Rosemary Seva. 2022. Effect of Fatigue in Air Traffic Controllers’ Workload, Situation Awareness, and Control Strategy. The International Journal of Aerospace Psychology 32, 1 (2022), 1–23.
[10]
Sarah-Maria Castritius, Patric Schubert, Christoph Dietz, Heiko Hecht, Lynn Huestegge, Magnus Liebherr, and Christian T Haas. 2021. Driver Situation Awareness and Perceived Sleepiness during Truck Platoon Driving–Insights from Eye-tracking Data. International Journal of Human–Computer Interaction 37, 15 (2021), 1467–1477.
[11]
Di Catherwood, Graham K Edgar, Dritan Nikolla, Chris Alford, David Brookes, Steven Baker, and Sarah White. 2014. Mapping brain activity during loss of situation awareness: an EEG investigation of a basis for top-down influence on perception. Human factors 56, 8 (2014), 1428–1452.
[12]
Lan-lan Chen, Yu Zhao, Peng-fei Ye, Jian Zhang, and Jun-zhong Zou. 2017. Detecting driving stress in physiological signals based on multimodal feature analysis and kernel classifiers. Expert Systems with Applications 85 (2017), 279–291.
[13]
Baoquan Cheng, Chaojie Fan, Hanliang Fu, Jianling Huang, Huihua Chen, and Xiaowei Luo. 2022. Measuring and computing cognitive statuses of construction workers based on electroencephalogram: a critical review. IEEE Transactions on Computational Social Systems (2022).
[14]
Hsiu-Sen Chiang. 2015. Ecg-based mental stress assessment using fuzzy computing and associative petri net. Journal of Medical and Biological Engineering 35, 6 (2015), 833–844.
[15]
Simone Dari, Nikolay Kadrileev, and Eyke Hüllermeier. 2020. A neural network-based driver gaze classification system with vehicle signals. In 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 1–7.
[16]
Amir Dini, Cornelia Murko, Saeed Yahyanejad, Ursula Augsdörfer, Michael Hofbaur, and Lucas Paletta. 2017. Measurement and prediction of situation awareness in human-robot interaction based on a framework of probabilistic attention. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 4354–4361.
[17]
Mica R Endsley. 1995. Measurement of situation awareness in dynamic systems. Human factors 37, 1 (1995), 65–84.
[18]
Mica R Endsley and Daniel J Garland. 2000. Pilot situation awareness training in general aviation. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 44. SAGE Publications Sage CA: Los Angeles, CA, 357–360.
[19]
Alexander Eriksson and Neville A Stanton. 2017. Takeover time in highly automated vehicles: noncritical transitions to and from manual control. Human factors 59, 4 (2017), 689–705.
[20]
Raul Fernandez Rojas, Essam Debie, Justin Fidock, Michael Barlow, Kathryn Kasmarik, Sreenatha Anavatti, Matthew Garratt, and Hussein Abbass. 2019. Encephalographic assessment of situation awareness in teleoperation of human-swarm teaming. In International Conference on Neural Information Processing. Springer, 530–539.
[21]
L Fridman, P Langhans, J Lee, and B Reimer. 2019. Driver gaze region estimation without using eye movement. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
[22]
Lex Fridman, Joonbum Lee, Bryan Reimer, and Trent Victor. 2016. ‘Owl’and ‘Lizard’: Patterns of head pose and eye pose in driver gaze classification. IET Computer Vision 10, 4 (2016), 308–314.
[23]
Xiaofeng Gao, Xingwei Wu, Samson Ho, Teruhisa Misu, and Kumar Akash. 2022. Effects of Augmented-Reality-Based Assisting Interfaces on Drivers’ Object-wise Situational Awareness in Highly Autonomous Vehicles. arXiv preprint arXiv:2206.02332 (2022).
[24]
Leo J Gugerty. 1997. Situation awareness during driving: Explicit and implicit knowledge in dynamic spatial memory.Journal of Experimental Psychology: Applied 3, 1 (1997), 42.
[25]
Eija Haapalainen, SeungJun Kim, Jodi F Forlizzi, and Anind K Dey. 2010. Psycho-physiological measures for assessing cognitive load. In Proceedings of the 12th ACM international conference on Ubiquitous computing. 301–310.
[26]
Sogand Hasanzadeh, Behzad Esmaeili, and Michael D Dodd. 2016. Measuring construction workers’ real-time situation awareness using mobile eye-tracking. In Construction Research Congress 2016. 2894–2904.
[27]
Markus Hofbauer, Christopher B Kuhn, Jiaming Meng, Goran Petrovic, and Eckehard Steinbach. 2020. Multi-view region of interest prediction for autonomous driving using semi-supervised labeling. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC). IEEE, 1–6.
[28]
Markus Hofbauer, Christopher B Kuhn, Lukas Püttner, Goran Petrovic, and Eckehard Steinbach. 2020. Measuring driver situation awareness using region-of-interest prediction and eye tracking. In 2020 IEEE International Symposium on Multimedia (ISM). IEEE, 91–95.
[29]
Maarten A Hogervorst, Anne-Marie Brouwer, and Jan BF Van Erp. 2014. Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload. Frontiers in neuroscience 8 (2014), 322.
[30]
Mike Howard, Rashmi Sundareswara, Mike Daily, Rajan Bhattacharyya, Sam Kaplan, Nathan Mundhenk, Craig Lee, and Howard Neely. 2013. Using tactile displays to maintain situational awareness during driving. In 2013 IEEE International Multi-Disciplinary Conference on Cognitive Methods in Situation Awareness and Decision Support (CogSIMA). IEEE, 228–237.
[31]
Yifei Huang, Minjie Cai, Zhenqiang Li, and Yoichi Sato. 2018. Predicting gaze in egocentric video by learning task-dependent attention transition. In Proceedings of the European conference on computer vision (ECCV). 754–769.
[32]
Lisheng Jin, Qingning Niu, Haijing Hou, Huacai Xian, Yali Wang, and Dongdong Shi. 2012. Driver cognitive distraction detection using driving performance measures. Discrete Dynamics in Nature and Society 2012 (2012).
[33]
Jan Luca Kästle, Bani Anvari, Jakub Krol, and Helge A Wurdemann. 2021. Correlation between Situational Awareness and EEG signals. Neurocomputing 432 (2021), 70–79.
[34]
Jesus L Lobo, Javier Del Ser, Flavia De Simone, Roberta Presta, Simona Collina, and Zdenek Moravek. 2016. Cognitive workload classification using eye-tracking and EEG data. In Proceedings of the International Conference on Human-Computer Interaction in Aerospace. 1–8.
[35]
Wing-Yue Geoffrey Louie, Iyad Mansour, 2019. Towards a driver monitoring system for estimating driver situational awareness. In 2019 28th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, 1–6.
[36]
Staffan Magnusson Nählinder. 2009. Flight simulator training: Assessing the potential. Ph. D. Dissertation. Linköping University Electronic Press.
[37]
Michael D Matthews and Scott A Beal. 2002. Assessing situation awareness in field training exercises. Technical Report. Military Academy West Point NY Office of Military Psychology and Leadership.
[38]
Atsuo Murata. 2005. An attempt to evaluate mental workload using wavelet transform of EEG. Human Factors 47, 3 (2005), 498–508.
[39]
Rizwan Ali Naqvi, Muhammad Arsalan, Ganbayar Batchuluun, Hyo Sik Yoon, and Kang Ryoung Park. 2018. Deep learning-based gaze detection system for automobile drivers using a NIR camera sensor. Sensors 18, 2 (2018), 456.
[40]
Mahesh Pal. 2005. Random forest classifier for remote sensing classification. International journal of remote sensing 26, 1 (2005), 217–222.
[41]
Cristina Palmero, Javier Selva, Mohammad Ali Bagheri, and Sergio Escalera. 2018. Recurrent cnn for 3d gaze estimation using appearance and shape cues. arXiv preprint arXiv:1805.03064 (2018).
[42]
Seonwook Park, Shalini De Mello, Pavlo Molchanov, Umar Iqbal, Otmar Hilliges, and Jan Kautz. 2019. Few-shot adaptive gaze estimation. In Proceedings of the IEEE/CVF international conference on computer vision. 9368–9377.
[43]
Luke Petersen, Lionel Robert, X Jessie Yang, and Dawn M Tilbury. 2019. Situational awareness, drivers trust in automated driving systems and secondary task performance. arXiv preprint arXiv:1903.05251 (2019).
[44]
Karl Popper. 2005. The logic of scientific discovery. Routledge.
[45]
Valentina Quaresima and Marco Ferrari. 2019. A mini-review on functional near-infrared spectroscopy (fNIRS): where do we stand, and where should we go?. In Photonics, Vol. 6. MDPI, 87.
[46]
Akshay Rangesh, Nachiket Deo, Kevan Yuen, Kirill Pirozhenko, Pujitha Gunaratne, Heishiro Toyoda, and Mohan M Trivedi. 2018. Exploring the situational awareness of humans inside autonomous vehicles. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC). IEEE, 190–197.
[47]
Nataniel Ruiz, Eunji Chong, and James M Rehg. 2018. Fine-grained head pose estimation without keypoints. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2074–2083.
[48]
Paul M Salmon, Neville A Stanton, Guy H Walker, Daniel Jenkins, Darshna Ladva, Laura Rafferty, and Mark Young. 2009. Measuring Situation Awareness in complex systems: Comparison of measures study. International Journal of Industrial Ergonomics 39, 3 (2009), 490–500.
[49]
Tommy Strandvall. 2009. Eye tracking in human-computer interaction and usability research. In IFIP Conference on Human-Computer Interaction. Springer, 936–937.
[50]
David L Strayer, Jonna Turrill, Joel M Cooper, James R Coleman, Nathan Medeiros-Ward, and Francesco Biondi. 2015. Assessing cognitive distraction in the automobile. Human factors 57, 8 (2015), 1300–1324.
[51]
Lindsay Sturre, Dan Chiappe, Kim-Phuong L Vu, and Thomas Z Strybel. 2015. Using Eye Movements to Test Assumptions of the Situation Present Assessment Method. In International Conference on Human Interface and the Management of Information. Springer, 45–52.
[52]
Koen van de Merwe, Henk van Dijk, and Rolf Zon. 2012. Eye movements as an indicator of situation awareness in a flight simulator experiment. The International Journal of Aviation Psychology 22, 1 (2012), 78–95.
[53]
Arie P Van Den Beukel and Mascha C Van Der Voort. 2013. The influence of time-criticality on situation awareness when retrieving human control after automated driving. In 16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013). IEEE, 2000–2005.
[54]
Sourabh Vora, Akshay Rangesh, and Mohan Manubhai Trivedi. 2018. Driver gaze zone estimation using convolutional neural networks: A general framework and ablative analysis. IEEE Transactions on Intelligent Vehicles 3, 3 (2018), 254–265.
[55]
H Wang, Z Chen, and Y Zhou. [n. d.]. Hybrid coarse-fine classification for head pose estimation, arXiv. org. URL http://search. proquest. com/docview/2170061213 ([n. d.]).
[56]
Glenn F Wilson and Christopher A Russell. 2003. Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Human factors 45, 4 (2003), 635–644.
[57]
Guosheng Yang, Yingzi Lin, and Prabir Bhattacharya. 2010. A driver fatigue recognition model based on information fusion and dynamic Bayesian network. Information Sciences 180, 10 (2010), 1942–1954.
[58]
Shiyan Yang, Jonny Kuo, and Michael G Lenné. 2018. Analysis of gaze behavior to measure cognitive distraction in real-world driving. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 62. SAGE Publications Sage CA: Los Angeles, CA, 1944–1948.
[59]
Shiyan Yang, Kyle M Wilson, Trey Roady, Jonny Kuo, and Michael G Lenné. 2022. Evaluating driver features for cognitive distraction detection and validation in manual and Level 2 automated driving. Human factors 64, 4 (2022), 746–759.
[60]
Chung-san Yu, Eric Min-yang Wang, Wen-Chin Li, and Graham Braithwaite. 2014. Pilots’ visual scan patterns and situation awareness in flight operations. Aviation, space, and environmental medicine 85, 7 (2014), 708–714.
[61]
Guangtao Zhang, Katsumi Minakata, and John Paulin Hansen. 2019. Enabling Real-Time measurement of situation awareness in robot teleoperation with a head-mounted display. In Human Factors Society Conference. 169.
[62]
Chunlin Zhao, Chongxun Zheng, Min Zhao, Yaling Tu, and Jianping Liu. 2011. Multivariate autoregressive models and kernel learning algorithms for classifying driving mental fatigue based on electroencephalographic. Expert Systems with Applications 38, 3 (2011), 1859–1865.
[63]
Feng Zhou, X Jessie Yang, and Joost CF de Winter. 2021. Using eye-tracking data to predict situation awareness in real time during takeover transitions in conditionally automated driving. IEEE Transactions on Intelligent Transportation Systems 23, 3 (2021), 2284–2295.
[64]
Haibei Zhu, Teruhisa Misu, Sujitha Martin, Xingwei Wu, and Kumar Akash. 2021. Improving driver situation awareness prediction using human visual sensory and memory mechanism. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 6210–6216.

Index Terms

  1. A Panoramic Review of Situational Awareness Monitoring Systems

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    ICRSA '23: Proceedings of the 2023 6th International Conference on Robot Systems and Applications
    September 2023
    335 pages
    ISBN:9798400708039
    DOI:10.1145/3655532
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 28 June 2024

    Check for updates

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    ICRSA 2023

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • 0
      Total Citations
    • 149
      Total Downloads
    • Downloads (Last 12 months)149
    • Downloads (Last 6 weeks)27
    Reflects downloads up to 20 Feb 2025

    Other Metrics

    Citations

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media