Skip to main content
SpringerLink
Log in

Estimating Cognitive Load and Cybersickness of Pilots in VR Simulations via Unobtrusive Physiological Sensors

  • Conference paper
  • First Online:
Virtual, Augmented and Mixed Reality: Applications in Education, Aviation and Industry (HCII 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13318))

Included in the following conference series:

Abstract

Predicting real-time estimates of cognitive load in pilots assists intelligent flight systems in alleviating high workloads, thereby averting accidents and directly impacting safety in aviation. Virtual Reality (VR) flight simulations provide an immersive stage to evaluate physiological measures and identify their cognitive correlates. In this work, unobtrusive sensors such as eye-tracking, pupillometry, and photoplethysmography (PPG) record physiological data while six participants perform six flying tasks of varying complexity in VR. The extracted feature sets such as pupil diameter change, number of fixations and saccades, and heart rate variability (HRV) are compared to the Pilot Inceptor Workload (PIW) measures, specifically duty cycle and aggressiveness. The PIW, number of saccades, and the self-reported workload measures were significantly affected by the tasks. However, the number of saccades measure demonstrated a significant negative correlation with the PIW’s measures, contradicting prior work. The remaining feature sets, including the pupil diameter change and the number of fixations, display a nearly identical trend to the PIW measure, though no significance was detected.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Arjun, S., Rajshekar Reddy, G., Mukhopadhyay, A., Vinod, S., Biswas, P.: Evaluating visual variables in a virtual reality environment. In: 34th British HCI Conference 34, pp. 11–22 (2021)

    Google Scholar 

  2. Babu, M.D.: Investigation of pilot inceptor workload and workload buildup technique through simulator and in-flight studies. Int. J. Aerosp. Psychol. 32(2–3), 65–94 (2022)

    Google Scholar 

  3. Babu, M.D., JeevithaShree, D., Prabhakar, G., Saluja, K.P.S., Pashilkar, A., Biswas, P.: Estimating pilots’ cognitive load from ocular parameters through simulation and in-flight studies. J. Eye Mov. Res. 12(3) (2019)

    Google Scholar 

  4. Beatty, J., Lucero-Wagoner, B., et al.: The pupillary system. In: Handbook of Psychophysiology, 2nd edn, pp. 142–162 (2000)

    Google Scholar 

  5. Casner, S.M., Gore, B.F.: Measuring and evaluating workload: a primer. NASA Technical Memorandum, NASA–TM 2010-216395 (2010)

    Google Scholar 

  6. Chen, S., Epps, J.: Automatic classification of eye activity for cognitive load measurement with emotion interference. Comput. Methods Programs Biomed. 110(2), 111–124 (2013)

    Article  Google Scholar 

  7. Coral, M.P.: Analyzing cognitive workload through eye-related measurements: a meta-analysis. Ph.D. thesis, Wright State University (2016)

    Google Scholar 

  8. Cowley, B.U., et al.: The psychophysiology primer: a guide to methods and a broad review with a focus on human-computer interaction. Found. Trends Hum. Comput. Interact. 9(3–4), 151–308 (2016)

    Article  Google Scholar 

  9. Davis, S., Nesbitt, K., Nalivaiko, E.: A systematic review of cybersickness. In: Proceedings of the 2014 Conference on Interactive Entertainment, IE 2014, pp. 1–9. Association for Computing Machinery, New York (2014). https://doi.org/10.1145/2677758.2677780

  10. Di Nocera, F., Camilli, M., Terenzi, M.: Using the distribution of eye fixations to assess pilots’ mental workload. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 50, pp. 63–65. SAGE Publications, Los Angeles (2006)

    Google Scholar 

  11. Feng, C., Wanyan, X., Yang, K., Zhuang, D., Wu, X.: A comprehensive prediction and evaluation method of pilot workload. Technol. Health Care 26(S1), 65–78 (2018)

    Article  Google Scholar 

  12. Forte, G., Favieri, F., Casagrande, M.: Heart rate variability and cognitive function: a systematic review. Front. Neurosci. 13, 710 (2019)

    Article  Google Scholar 

  13. Gianaros, P.J., Muth, E.R., Mordkoff, J.T., Levine, M.E., Stern, R.M.: A questionnaire for the assessment of the multiple dimensions of motion sickness. Aviat. Space Environ. Med. 72(2), 115 (2001)

    Google Scholar 

  14. Golding, J.F.: Motion sickness susceptibility questionnaire revised and its relationship to other forms of sickness. Brain Res. Bull. 47(5), 507–516 (1998)

    Article  Google Scholar 

  15. Grant, T., Dhruv, K., Eloy, L., Hayne, L., Durkee, K., Hirshfield, L.: A neurophysiological sensor suite for real-time prediction of pilot workload in operational settings. In: Stephanidis, C., et al. (eds.) HCII 2020. LNCS, vol. 12425, pp. 60–77. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60128-7_5

    Chapter  Google Scholar 

  16. Gray, W.: Boundary-escape tracking: a new conception of hazardous PIO. Technical report, Air Force Flight Test Center, Edwards AFB, CA (2004)

    Google Scholar 

  17. Gray, W.: Boundary avoidance tracking: a new pilot tracking model. In: AIAA Atmospheric Flight Mechanics Conference and Exhibit, p. 5810 (2005)

    Google Scholar 

  18. Gray, W.: A generalized handling qualities flight test technique utilizing boundary avoidance tracking. In: 2008 US Air Force T&E Days, p. 1648. Aerospace Research Central (2008)

    Google Scholar 

  19. Gray, W.: A boundary avoidance tracking flight test technique for performance and workload assessment. In: Proceedings of the 38th Symposium of Society of Experimental Test Pilots, San Diego (2007)

    Google Scholar 

  20. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv. Psychol. 52, 139–183 (1988)

    Article  Google Scholar 

  21. Hebbar, P.A., Bhattacharya, K., Prabhakar, G., Pashilkar, A.A., Biswas, P.: Correlation between physiological and performance-based metrics to estimate pilots’ cognitive workload. Front. Psychol. 12, 954 (2021)

    Article  Google Scholar 

  22. HP Development Company: Eye Tracking Calibration (2022). https://developers.hp.com/omnicept/eye-tracking-calibration. Accessed 23 Feb 2022

  23. HP Development Company: Fundamentals (2022). https://developers.hp.com/omnicept/docs/fundamentals?language=es-un. Accessed 23 Feb 2022

  24. HP Development Company: HP Reverb G2 Omnicept Edition (2022). https://www.hp.com/us-en/vr/reverb-g2-vr-headset-omnicept-edition.html. Accessed 23 Feb 2022

  25. Huttunen, K., Keränen, H., Väyrynen, E., Pääkkönen, R., Leino, T.: Effect of cognitive load on speech prosody in aviation: evidence from military simulator flights. Appl. Ergon. 42(2), 348–357 (2011)

    Article  Google Scholar 

  26. Itoh, Y., Hayashi, Y., Tsukui, I., Saito, S.: The ergonomic evaluation of eye movement and mental workload in aircraft pilots. Ergonomics 33(6), 719–732 (1990)

    Article  Google Scholar 

  27. Kim, H.K., Park, J., Choi, Y., Choe, M.: Virtual reality sickness questionnaire (VRSQ): motion sickness measurement index in a virtual reality environment. Appl. Ergon. 69, 66–73 (2018)

    Article  Google Scholar 

  28. Kim, Y.Y., Kim, H.J., Kim, E.N., Ko, H.D., Kim, H.T.: Characteristic changes in the physiological components of cybersickness. Psychophysiology 42(5), 616–625 (2005)

    Google Scholar 

  29. Krejtz, K., Duchowski, A.T., Niedzielska, A., Biele, C., Krejtz, I.: Eye tracking cognitive load using pupil diameter and microsaccades with fixed gaze. PLoS ONE 13(9), e0203629 (2018)

    Article  Google Scholar 

  30. Lancaster, J.A., Casali, J.G.: Investigating pilot performance using mixed-modality simulated data link. Hum. Factors 50(2), 183–193 (2008)

    Article  Google Scholar 

  31. Latham, G.P., Seijts, G., Crim, D.: The effects of learning goal difficulty level and cognitive ability on performance. Can. J. Behav. Sci. 40(4), 220 (2008)

    Article  Google Scholar 

  32. LaViola, J.J.: A discussion of cybersickness in virtual environments. SIGCHI Bull. 32(1), 47–56 (2000). https://doi.org/10.1145/333329.333344

    Article  Google Scholar 

  33. Leube, A., Rifai, K., Rifai, K.: Sampling rate influences saccade detection in mobile eye tracking of a reading task. J. Eye Mov. Res. 10(3) (2017)

    Google Scholar 

  34. Lockheed Martin Corporation: Lockheed Martin Prepar3D (2022). https://www.prepar3d.com. Accessed 23 Feb 2022

  35. Lohani, M., Payne, B.R., Strayer, D.L.: A review of psychophysiological measures to assess cognitive states in real-world driving. Front. Hum. Neurosci. 13, 57 (2019)

    Article  Google Scholar 

  36. Mathur, A., Gehrmann, J., Atchison, D.A.: Pupil shape as viewed along the horizontal visual field. J. Vis. 13(6), 3–3 (2013)

    Article  Google Scholar 

  37. McCraty, R., Shaffer, F.: Heart rate variability: new perspectives on physiological mechanisms, assessment of self-regulatory capacity, and health risk. Glob. Adv. Health Med. 4(1), 46–61 (2015)

    Article  Google Scholar 

  38. Mehler, B., Reimer, B., Wang, Y.: A comparison of heart rate and heart rate variability indices in distinguishing single-task driving and driving under secondary cognitive workload. In: Proceedings of the Sixth International Driving Symposium on Human Factors in Driver Assessment, Training and Vehicle Design, pp. 590–597 (2011)

    Google Scholar 

  39. Mohanavelu, K., et al.: Cognitive workload analysis of fighter aircraft pilots in flight simulator environment. Defence Sci. J. 70(2), 131–139 (2020)

    Article  Google Scholar 

  40. Mulder, L.J.M.: Assessment of cardiovascular reactivity by means of spectral analysis. Ph.D. thesis, Rijksuniversiteit (1988)

    Google Scholar 

  41. Niewind, I.: Pilot gain and the workload buildup flight test technique: about the influence of natural pilot gain on the achievable pilot gain range. Institute report IB 111-2012/69. German Aerospace Centre (2012)

    Google Scholar 

  42. Olsen, A.: The Tobii I-VT fixation filter. Tobii Technology, p. 21 (2012)

    Google Scholar 

  43. Olsen, A., Matos, R.: Identifying parameter values for an I-VT fixation filter suitable for handling data sampled with various sampling frequencies. In: Proceedings of the Symposium on Eye Tracking Research and Applications, ETRA 2012, pp. 317–320. Association for Computing Machinery, New York (2012). https://doi.org/10.1145/2168556.2168625

  44. Prabhakar, G., Mukhopadhyay, A., Murthy, L., Modiksha, M., Sachin, D., Biswas, P.: Cognitive load estimation using ocular parameters in automotive. Transp. Eng. 2, 100008 (2020)

    Article  Google Scholar 

  45. Roscoe, A.H.: Assessing pilot workload. Why measure heart rate, HRV and respiration? Biol. Psychol. 34(2–3), 259–287 (1992)

    Article  Google Scholar 

  46. Roscoe, A.H., Ellis, G.A.: A subjective rating scale for assessing pilot workload in flight: a decade of practical use. Technical report, Royal Aerospace Establishment Farnborough, United Kingdom (1990)

    Google Scholar 

  47. Salvucci, D.D., Goldberg, J.H.: Identifying fixations and saccades in eye-tracking protocols. In: Proceedings of the 2000 Symposium on Eye Tracking Research & Applications, pp. 71–78 (2000)

    Google Scholar 

  48. Schmidt, E.A., Schrauf, M., Simon, M., Fritzsche, M., Buchner, A., Kincses, W.E.: Drivers’ misjudgement of vigilance state during prolonged monotonous daytime driving. Accid. Anal. Prev. 41(5), 1087–1093 (2009)

    Article  Google Scholar 

  49. Shaffer, F., Ginsberg, J.P.: An overview of heart rate variability metrics and norms. Front. Public Health 5, 258 (2017)

    Article  Google Scholar 

  50. Shaffer, F., McCraty, R., Zerr, C.L.: A healthy heart is not a metronome: an integrative review of the heart’s anatomy and heart rate variability. Front. Psychol. 5, 1040 (2014)

    Article  Google Scholar 

  51. Shakouri, M., Ikuma, L.H., Aghazadeh, F., Nahmens, I.: Analysis of the sensitivity of heart rate variability and subjective workload measures in a driving simulator: the case of highway work zones. Int. J. Ind. Ergon. 66, 136–145 (2018)

    Article  Google Scholar 

  52. Siegel, E., et al.: HP Omnicept cognitive load database (HPO-CLD)-developing a multimodal inference engine for detecting real-time mental workload in VR. Technical report, HP Labs, Palo Alto (2021). https://developers.hp.com/omnicept

  53. Spyker, D., Stackhouse, S., Khalafalla, A., McLane, R.: Development of techniques for measuring pilot workload. Technical report, NASA (1971)

    Google Scholar 

  54. Thayer, J.F., Lane, R.D.: Claude Bernard and the heart-brain connection: further elaboration of a model of neurovisceral integration. Neurosci. Biobehav. Rev. 33(2), 81–88 (2009)

    Article  Google Scholar 

  55. Thrustmaster: HOTAS WARTHOG (2022). https://www.thrustmaster.com/products/hotas-warthog/. Accessed 23 Feb 2022

  56. Thrustmaster: T.Flight Rudder Pedals (2022). https://www.thrustmaster.com/en-us/products/t-flight-rudder-pedals/. Accessed 23 Feb 2022

  57. Valve Corporation: SteamVR on Steam (2022). https://store.steampowered.com/app/250820/SteamVR/. Accessed 23 Feb 2022

  58. Villafaina, S., Fuentes-García, J.P., Gusi, N., Tornero-Aguilera, J.F., Clemente-Suárez, V.J.: Psychophysiological response of military pilots in different combat flight maneuvers in a flight simulator. Physiol. Behav. 238, 113483 (2021)

    Article  Google Scholar 

  59. Wang, Z., Yang, L., Ding, J.: Application of heart rate variability in evaluation of mental workload. Zhonghua lao dong wei sheng zhi ye bing za zhi= Zhonghua laodong weisheng zhiyebing zazhi= Chin. J. Ind. Hyg. Occup. Dis. 23(3), 182–184 (2005)

    Google Scholar 

  60. Wikipedia contributors: Aerobatic maneuver – Wikipedia, the free encyclopedia (2021). https://en.wikipedia.org/w/index.php?title=Aerobatic_maneuver&oldid=1044302151. Accessed 23 Feb 2022

  61. Wikipedia contributors: Microsaccade – Wikipedia, the free encyclopedia (2022). https://en.wikipedia.org/w/index.php?title=Microsaccade&oldid=1064617483. Accessed 23 Feb 2022

  62. Wong, L., et al.: Performance on the robotics on-board trainer (ROBoT-r) spaceflight simulation during acute sleep deprivation. Front. Neurosci. 14, 697 (2020)

    Article  Google Scholar 

Download references

Acknowledgements

We would like to acknowledge the U.S. Navy for supporting this research via contract number N68335-18-C0133.

Author information

Authors and Affiliations

  1. Institute of Cognitive Science, University of Colorado, Boulder, CO, USA

    G. S. Rajshekar Reddy, Cara A. Spencer, Gavin Zimmerman & Leanne Hirshfield

  2. Aptima, Inc., Dayton, OH, USA

    Kevin Durkee, Olivia Fox Cotton, Sheila Galbreath & Sarah Meyer

  3. U.S. Navy, Wright-Patterson Air Force Base, Dayton, OH, USA

    Brennan Cox, Michael Natali & Todd Seech

  4. Naval Air Warfare Center Training Systems Division, Orlando, FL, USA

    Gabriella Severe-Valsaint

Authors
  1. G. S. Rajshekar Reddy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cara A. Spencer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kevin Durkee
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Brennan Cox
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Olivia Fox Cotton
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sheila Galbreath
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Sarah Meyer
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Michael Natali
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Todd Seech
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Gabriella Severe-Valsaint
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Gavin Zimmerman
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Leanne Hirshfield
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to G. S. Rajshekar Reddy .

Editor information

Editors and Affiliations

  1. U.S. Army Research Laboratory, Aberdeen Proving Ground, MD, USA

    Jessie Y. C. Chen

  2. U.S. Army Combat Capabilities Development Command Soldier Center, Orlando, FL, USA

    Gino Fragomeni

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Reddy, G.S.R. et al. (2022). Estimating Cognitive Load and Cybersickness of Pilots in VR Simulations via Unobtrusive Physiological Sensors. In: Chen, J.Y.C., Fragomeni, G. (eds) Virtual, Augmented and Mixed Reality: Applications in Education, Aviation and Industry. HCII 2022. Lecture Notes in Computer Science, vol 13318. Springer, Cham. https://doi.org/10.1007/978-3-031-06015-1_18

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-06015-1_18

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-06014-4

  • Online ISBN: 978-3-031-06015-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation