skip to main content
10.1145/3025453.3025822acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Toward Measurement of Situation Awareness in Autonomous Vehicles

Published: 02 May 2017 Publication History

Abstract

Until vehicles are fully autonomous, safety, legal and ethical obligations require that drivers remain aware of the driving situation. Key decisions about whether a driver can take over when the vehicle is confused, or its capabilities are degraded, depend on understanding whether he or she is responsive and aware of external conditions. The leading techniques for measuring situation awareness in simulated environments are ill-suited to autonomous driving scenarios, and particularly to on-road testing. We have developed a technique, named Daze, to measure situation awareness through real-time, in-situ event alerts. The technique is ecologically valid: it resembles applications people use in actual driving. It is also flexible: it can be used in both simulator and on-road research settings. We performed simulator-based and on-road test deployments to (a) check that Daze could characterize drivers' awareness of their immediate environment and (b) understand practical aspects of the technique's use. Our contributions include the Daze technique, examples of collected data, and ways to analyze such data.

Supplementary Material

suppl.mov (pn2824-file3.mp4)
Supplemental video
suppl.mov (pn2824p.mp4)
Supplemental video
MP4 File (p405-sirkin.mp4)

References

[1]
L. Paige Bacon and Thomas Strybel. 2013. Assessment of the validity and intrusiveness of online-probe questions for situation awareness in a simulated air-traffic-management task with student air-traffic controllers. Safety Science 56 (2013), 89--95. http://dx.doi.org/10.1016/j.ssci.2012.06.019
[2]
Sonia Baltodano, Srinath Sibi, Nikolas Martelaro, Nikhil Gowda, and Wendy Ju. 2015. The RRADS platform: A real road autonomous driving simulation. In Proc. of the ACM/IEEE Intl. Conf. on Automotive User Interfaces and Interactive Vehicular Interactions. ACM, 281--288. http://dl.acm.org/citation.cfm?id=2799250.2799288
[3]
Cheryl A. Bolstad, Haydee M. Cuevas, Jingjing Wang-Costello, Mica R. Endsley, and Linda S. Angell. 2009. Measurement of situation awareness for automobile technologies of the future. In Performance Metrics for Assessing Driver Distraction. SAE, 195--213.
[4]
G. Borghini, G. Vecchiato, J. Toppi, L. Astolfi, A. Maglione, R. Isabella, C. Caltagirone, W. Kong, D. Wei, Z. Zhou, and L. Polidori. 2012. Assessment of mental fatigue during car driving by using high resolution EEG activity and neurophysiologic indices. In Engineering in Medicine and Biology Society (EMBC), 2012 Annual Intl. Conf. of the IEEE. IEEE, 6442--6445. http://ieeexplore. ieee.org/xpls/abs_all.jsp?arnumber=6347469
[5]
Karel A. Brookhuis, Dick De Waard, and Wiel H. Janssen. 2001. Behavioural impacts of advanced driver assistance systems--an overview. European Journal of Transport and Infrastructure Research 1, 3 (2001), 245--253. http://www.ejtir.tbm.tudelft.nl/issues/2001_ 03/pdf/2001_03_02.pdf
[6]
Christopher Chabris and Daniel Simons. 2011. The invisible gorilla: And other ways our intuitions deceive us. Broadway Books. http://dx.doi.org/10.1080/17439760.2012.753466
[7]
Trafton Drew, Melissa L. H. Võ, and Jeremy M. Wolfe. 2013. The invisible gorilla strikes again: Sustained inattentional blindness in expert observers. Psychological Science 24, 9 (2013), 1848--1853. https://doi.org/10.1177/0956797613479386
[8]
Frank T. Durso, Andrew R. Dattel, S. Banbury, and S. Tremblay. 2004. SPAM: The real-time assessment of SA, S. Banbury and S. Tremblay (Eds.). A Cognitive Approach to Situation Awareness: Theory and Application (2004), 137--154.
[9]
Mica R. Endsley. 1995a. Measurement of situation awareness in dynamic systems. Human Factors: The Journal of the Human Factors and Ergonomics Society 37, 1 (1995), 65--84. http://hfs.sagepub.com/content/37/1/65.short
[10]
Mica R. Endsley. 1995b. Toward a theory of situation awareness in dynamic systems. Human Factors: The Journal of the Human Factors and Ergonomics Society 37, 1 (1995), 32--64. http://hfs.sagepub.com/content/37/1/32.short
[11]
Mica R. Endsley and David B. Kaber. 1999. Level of automation effects on performance, situation awareness and workload in a dynamic control task. Ergonomics 42, 3 (1999), 462--492. http://dx.doi.org/10.1080/001401399185595
[12]
P. A. Hancock. 2014. Automation: How much is too much? Ergonomics 57, 3 (2014), 449--454. http://dx.doi.org/10.1080/00140139.2013.816375
[13]
Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in psychology 52 (1988), 139--183. http://dx.doi.org/10.1016/S0166--4115(08)62386--9
[14]
Key Jung Lee, Yeon Kyoung Joo, and Clifford Nass. 2014. Partially intelligent automobiles and driving experience at the moment of system transition. In Proc. of the SIGCHI Conf. on Human Factors in Computing Systems. ACM, 3631--3634. http://dl.acm.org/citation.cfm?id=2557370
[15]
Kwan Min Lee. 2004. Presence, explicated. Communication Theory 14, 1 (2004), 27--50. http://onlinelibrary.wiley.com/doi/10.1111/j. 1468--2885.2004.tb00302.x/full
[16]
R. Ma and David B. Kaber. 2007. Situation awareness and driving performance in a simulated navigation task. Ergonomics 50, 8 (2007), 1351--1364. http://dx.doi.org/10.1080/00140130701318913
[17]
Nikolas Martelaro, David Sirkin, and Wendy Ju. 2015. Daze: A real-time situation awareness measurement tool for driving. In Proc. of the ACM/IEEE Intl. Conf. on Automotive User Interfaces and Interactive Vehicular Interactions. ACM, 158--163. http://dl.acm.org/citation.cfm?doid=2809730.2809753
[18]
M. H. Martens and W. Van Winsum. 2000. Measuring distraction: the peripheral detection task. TNO Human Factors, Soesterberg, Netherlands (2000). http://dx.doi.org/10.1016/j.trf.2005.04.009
[19]
Brian Ka-Jun Mok, Mishel Johns, Key Jung Lee, Hillary Page Ive, David Miller, and Wendy Ju. 2015a. Timing of unstructured transitions of control in automated driving. In Intelligent Vehicles Symposium. IEEE, 1167--1172. http://dx.doi.org/10.1109/IVS.2015.7225841
[20]
Brian Ka-Jun Mok, David Sirkin, Srinath Sibi, David Bryan Miller, and Wendy Ju. 2015b. Understanding driver-automated vehicle interactions through Wizard of Oz design improvisation. In Proc. Intl. Driving Symposium on Human Factors in Driver Assessment, Training, and Vehicle Design. 386--392.
[21]
National Highway Traffic Safety Administration and U.S. Department of Transportation. 2016. Federal Automated Vehicles Policy: Accelerating the Next Revolution in Roadway Safety. Washington, DC.
[22]
National Highway Traffic Safety Administration and others. 2013. Preliminary Statement of Policy Concerning Automated Vehicles. Washington, DC.
[23]
Fred Paas, Juhani E. Tuovinen, Huib Tabbers, and Pascal WM Van Gerven. 2003. Cognitive load measurement as a means to advance cognitive load theory. Educational Psychologist 38, 1 (2003), 63--71. http:// www.tandfonline.com/doi/abs/10.1207/S15326985EP3801_8
[24]
Raja Parasuraman, Thomas B. Sheridan, and Christopher D. Wickens. 2008. Situation awareness, mental workload, and trust in automation: Viable, empirically supported cognitive engineering constructs. Journal of Cognitive Engineering and Decision Making 2, 2 (2008), 140--160. http://edm.sagepub.com/content/2/2/140.short
[25]
Russell Pierce, Kim-Phuong L. Vu, Jimmy Nguyen, and Thomas Strybel. 2008. The relationship between SPAM, workload, and task performance on a simulated ATC task. In Proc. of the Human Factors Ergonomics Society. 34--38. https://doi.org/10.1177/154193120805200109
[26]
A. R. Quimby and G. R. Watts. 1981. Human factors and driving performance. Technical Report. http://trid.trb.org/view.aspx?id=174634
[27]
Thomas A. Ranney. 1994. Models of driving behavior: A review of their evolution. Accident Analysis & Prevention 26, 6 (1994), 733--750. http://www.sciencedirect.com/ science/article/pii/0001457594900515
[28]
Paul Salmon, Neville Stanton, Guy Walker, and Damian Green. 2006. Situation awareness measurement: A review of applicability for C4i environments. Applied Ergonomics 37, 2 (2006), 225--238. http://www.sciencedirect.com/science/article/pii/ S0003687005000554
[29]
Angelo Sassaroli, Feng Zheng, Leanne M. Hirshfield, Audrey Girouard, Erin Treacy Solovey, Robert JK Jacob, and Sergio Fantini. 2008. Discrimination of mental workload levels in human subjects with functional near-infrared spectroscopy. Journal of Innovative Optical Health Sciences 1, 02 (2008), 227--237. http://www. worldscientific.com/doi/abs/10.1142/S1793545808000224
[30]
Thomas B. Sheridan. 2006. Supervisory control. Handbook of Human Factors and Ergonomics (2006), 1025--1052. http://onlinelibrary.wiley.com/doi/10. 1002/0470048204.ch38/summary
[31]
Thiago H. Silva, Pedro O.S. Vaz de Melo, Aline Carneiro Viana, Jussara M. Almeida, Juliana Salles, and Antonio AF Loureiro. 2013. Traffic condition is more than colored lines on a map: Characterization of Waze alerts. In Social Informatics. Springer, 309--318. http://link.springer.com/chapter/10.1007/ 978--3--319-03260--3_27
[32]
M. W. Smolensky. 1993. Toward the physiological measurement of situation awareness: The case for eye movement measurements. In Proc. of the Human Factors and Ergonomics Society, Vol. 41.
[33]
Erin Treacy Solovey, Audrey Girouard, Krysta Chauncey, Leanne M. Hirshfield, Angelo Sassaroli, Feng Zheng, Sergio Fantini, and Robert JK Jacob. 2009. Using fNIRS brain sensing in realistic HCI settings: Experiments and guidelines. In Proc. ACM Symposium on User Interface Software and Technology. ACM, 157--166. http://dl.acm.org/citation.cfm?id=1622207
[34]
Holly A. Taylor, Susan J. Naylor, and Nicholas A. Chechile. 1999. Goal-specific influences on the representation of spatial perspective. Memory & Cognition 27, 2 (1999), 309--319. http://link.springer.com/article/10.3758/BF03211414
[35]
R. M. Taylor. 1990. Situational Awareness Rating Technique (SART): The development of a tool for aircrew systems design. AGARD, Situational Awareness in Aerospace Operations 17 (1990).
[36]
Kristie L. Young, Michael A. Regan, and John D. Lee. 2009. Measuring the effects of driver distraction: Direct driving performance methods and measures. CRC Press, Boca Raton, FL. http: //www.crcnetbase.com/doi/abs/10.1201/9781420007497.ch7
[37]
Mark S. Young and Neville A. Stanton. 2002. Attention and automation: New perspectives on mental underload and performance. Theoretical Issues in Ergonomics Science 3, 2 (2002), 178--194. http://www.tandfonline. com/doi/abs/10.1080/14639220210123789

Cited By

View all
  • (2024)Beyond "Taming Electric Scooters": Disentangling Understandings of Micromobility Naturalistic RidingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785138:3(1-24)Online publication date: 9-Sep-2024
  • (2024)Investigating Drivers’ Awareness of Pedestrians Using Virtual Reality towards Modeling the Impact of External Factors2024 IEEE Intelligent Vehicles Symposium (IV)10.1109/IV55156.2024.10588734(3001-3008)Online publication date: 2-Jun-2024
  • (2024)The “Trolley Problem” in Fully Automated AI-Driven Media: A Challenge Beyond Autonomous DrivingJournal of Media Ethics10.1080/23736992.2024.241805339:4(244-262)Online publication date: 9-Nov-2024
  • Show More Cited By

Index Terms

  1. Toward Measurement of Situation Awareness in Autonomous Vehicles

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
    May 2017
    7138 pages
    ISBN:9781450346559
    DOI:10.1145/3025453
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 May 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. autonomous vehicles
    2. interaction design
    3. measurement
    4. situation awareness

    Qualifiers

    • Research-article

    Funding Sources

    • Center for Design Research Affiliates Program

    Conference

    CHI '17
    Sponsor:

    Acceptance Rates

    CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)164
    • Downloads (Last 6 weeks)6
    Reflects downloads up to 23 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Beyond "Taming Electric Scooters": Disentangling Understandings of Micromobility Naturalistic RidingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785138:3(1-24)Online publication date: 9-Sep-2024
    • (2024)Investigating Drivers’ Awareness of Pedestrians Using Virtual Reality towards Modeling the Impact of External Factors2024 IEEE Intelligent Vehicles Symposium (IV)10.1109/IV55156.2024.10588734(3001-3008)Online publication date: 2-Jun-2024
    • (2024)The “Trolley Problem” in Fully Automated AI-Driven Media: A Challenge Beyond Autonomous DrivingJournal of Media Ethics10.1080/23736992.2024.241805339:4(244-262)Online publication date: 9-Nov-2024
    • (2024)An interpretable prediction framework for multi-class situational awareness in conditionally automated drivingAdvanced Engineering Informatics10.1016/j.aei.2024.10268362(102683)Online publication date: Oct-2024
    • (2024)Exploring the Impact of Interpretable Information Types on Driver's Situational Awareness and Performance During Driving Take-OverCross-Cultural Design10.1007/978-3-031-60913-8_8(99-114)Online publication date: 1-Jun-2024
    • (2023)A Visual Protocol Analysis Method for Collecting Driving Cognitions in Multi-user Driving Simulator StudiesProceedings of the Eleventh International Symposium of Chinese CHI10.1145/3629606.3629630(263-272)Online publication date: 13-Nov-2023
    • (2023)Physiological Indices to Predict Driver Situation Awareness in VRAdjunct Proceedings of the 2023 ACM International Joint Conference on Pervasive and Ubiquitous Computing & the 2023 ACM International Symposium on Wearable Computing10.1145/3594739.3610687(40-45)Online publication date: 8-Oct-2023
    • (2023)A study of the influence of AR on the perception, comprehension and projection levels of situation awareness2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR55154.2023.00069(541-551)Online publication date: Mar-2023
    • (2023)Human-Vehicle Interaction to Support Driver's Situation Awareness in Automated Vehicles: A Systematic ReviewIEEE Transactions on Intelligent Vehicles10.1109/TIV.2022.32008268:3(2551-2567)Online publication date: Mar-2023
    • (2023)Situational awareness in the automotive domain - Effects on driving performance and driver's behavior2023 3rd International Conference on Electrical, Computer, Communications and Mechatronics Engineering (ICECCME)10.1109/ICECCME57830.2023.10253014(1-5)Online publication date: 19-Jul-2023
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media