Skip to main content
SpringerLink
Log in

Multimodal Analysis Using Neuroimaging and Eye Movements to Assess Cognitive Workload

  • Conference paper
  • First Online:
Augmented Cognition. Theoretical and Technological Approaches (HCII 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12196))

Included in the following conference series:

Abstract

Air Traffic Control (ATC) specialists work in an environment where the proficient interaction between humans and computer systems is crucial to provide a safe and efficient flow of traffic. The complexity of this system may increase due to planned changes in operator roles and a projected rise in traffic volume. This increase, over an already highly complex system, will exacerbate the mental workload placed on the operator. The emergence of wearable sensors that measure physiological signals enables us to monitor the mental workload in real time without interfering in operational activity. However, the use of a single sensor approach may not provide a comprehensive assessment of cognitive workload while executing a complex task. Therefore, this study implemented a multimodal approach by using two sensors, namely functional near infrared spectroscopy (fNIRS) and eye-tracking, to evaluate the cognitive workload changes experienced by an ATC specialist. Three retired tower controllers with over 20 years of experience, underwent three sessions of experimentation where each individual fulfilled one of the following roles - observer, a Local controller or a Ground controller. During each iteration, the fNIRS and eye tracking sensors were attached to the Local controller while they commanded aircraft through verbal clearances. The task difficulty and complexity were quantified by the number of aircraft and clearances given, respectively. The number of aircraft displayed on the screen increased across time and was positively correlated with oxygenation measures assessed by the fNIRS signals of both the right and left prefrontal cortex. On the other hand, the number of fixations was positively correlated with the number of clearances. These results suggest fNIRS and eye-tracking measures are sensitive to changes in cognitive workload, and indicates that they may be amenable to complement each other for the assessment of the multidimensionality of cognitive workload induced by task difficulty and complexity.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 74.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Lesiuk, T.: The effect of preferred music listening on stress levels of air traffic controllers. Arts Psychother. 35, 1–10 (2008)

    Article  Google Scholar 

  • Finkelman, J.M.: A large database study of the factors associated with work-induced fatigue. Hum. Factors 36, 232–243 (1994)

    Article  Google Scholar 

  • Federal Aviation Administration: Forecasts of IFR aircraft handled by FAA air route traffic control centers FY 2017-2040 (2018)

    Google Scholar 

  • ICAO: The World of Air Transport in 2017. https://www.icao.int/annual-report-2017/Pages/the-world-of-air-transport-in-2017.aspx. Accessed 09 Feb 2020

  • Vogt, J., Hagemann, T., Kastner, M.: The impact of workload on heart rate and blood pressure in en-route and tower air traffic control. J. Psychophysiol. 20, 297–314 (2006)

    Article  Google Scholar 

  • Ahlstrom, U., Friedman-Berg, F.J.: Using eye movement activity as a correlate of cognitive workload. Int. J. Ind. Ergon. 36(7), 623–636 (2006)

    Article  Google Scholar 

  • Di Nocera, F., Fabrizi, R., Terenzi, M., Ferlazzo, F.: Procedural errors in air traffic control: effects of traffic density, expertise, and automation. Aviat. Space Environ. Med. 77, 639–643 (2006)

    Google Scholar 

  • Hilburn, B., Jorna, P.G., Byrne, E.A., Parasuraman, R.: The effect of adaptive air traffic control (ATC) decision aiding on controller mental workload. In: Mouloua, M., Koonce, J. (eds.) Human-Automation Interaction: Research and Practice, pp. 84–91. Erlbaum, Mahwah (1997)

    Google Scholar 

  • Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Hancock, P.A., Meshkati, N. (eds.) Advances in Psychology, Human Mental Workload, pp. 139–183. Amsterdam, Elseiver (1988)

    Chapter  Google Scholar 

  • Langan-Fox, J., Sankey, M.J., Canty, J.M.: Human factors measurement for future air traffic control systems. Hum. Factors J. Hum. Factors Ergon. Soc. 51(5), 595–637 (2009)

    Article  Google Scholar 

  • Bhavsar, P., Srinivasan, B., Srinivasan, R.: Quantifying situation awareness of control room operators using eye-gaze behavior. Comput. Chem. Eng. 106, 191–201 (2017)

    Article  Google Scholar 

  • Bruder, C., Hasse, C.: Differences between experts and novices in the monitoring of automated systems. Int. J. Ind. Ergon. 72, 1–11 (2019)

    Article  Google Scholar 

  • Otero, S.C., Weekes, B.S., Hutton, S.B.: Pupil size changes during recognition memory. Psychophysiology 48(10), 1346–1353 (2011)

    Article  Google Scholar 

  • Causse, M., Lancelot, F., Maillant, J., Behrend, J., Cousy, M., Schneider, N.: Encoding decisions and expertise in the operator’s eyes: using eye-tracking as input for system adaptation. Int. J. Hum Comput Stud. 125, 55–65 (2019)

    Article  Google Scholar 

  • Rudi, D., Kiefer, P., Raubal, M.: The instructor assistant system (iASSYST) utilizing eye tracking for commercial aviation training purposes. Ergonomics 63(1), 61–78 (2019)

    Article  Google Scholar 

  • Edwards T.: Human performance in air traffic control. Dissertation, University of Nottingham (2013)

    Google Scholar 

  • Ball, M., Barnhart, C., Nemhauser, G., Odoni, A.: Air transportation: irregular operations and control. In: Barnhart, C., Laporte, G. (eds.) Handbooks in Operations Research and Management Science, vol. 14, pp. 1–67. Elsevier, Amsterdam (2007)

    Google Scholar 

  • Durso, F.T., Manning, C.A.: Air traffic control. Rev. Hum. Factors. Ergon. 4, 195–244 (2008)

    Article  Google Scholar 

  • Kostenko, A., Rauffet, P., Coppin, G.: A dynamic closed-looped and multidimensional model for mental workload evaluation. IFAC - Papers On Line 49(19), 549–554 (2016)

    Article  Google Scholar 

  • Ayaz, H., Shewokis, P.A., Bunce, S., Izzetoglu, K., Willems, B., Onaral, B.: Optical brain monitoring for operator training and mental workload assessment. NeuroImage 59(1), 36–47 (2012)

    Article  Google Scholar 

  • Evans, D.C., Fendley, M.: A multi-measure approach for connecting cognitive workload and automation. Int. J. Hum Comput Stud. 97, 182–189 (2017)

    Article  Google Scholar 

  • Friedrich, M., Biermann, M., Gontar, P., Biella, M., Bengler, K.: The influence of task load on situation awareness and control strategy in the ATC tower environment. Cogn. Technol. Work 20(2), 205–217 (2018)

    Article  Google Scholar 

  • İşbilir, E., Çakır, M.P., Acartürk, C., Tekerek, A.Ş.: Towards a multimodal model of cognitive workload through synchronous optical brain imaging and eye tracking measures. Front. Hum. Neurosci. 13, 1–13 (2019)

    Article  Google Scholar 

  • Truschzinski, M., Betella, A., Brunnett, G., Verschure, P.F.M.J.: Emotional and cognitive influences in air traffic controller tasks: an investigation using a virtual environment. Appl. Ergon. 69, 1–9 (2018)

    Article  Google Scholar 

  • Marchitto, M., Benedetto, S., Baccino, T., Cañas, J.J.: Air traffic control: ocular metrics reflect cognitive complexity. Int. J. Ind. Ergon. 54, 120–130 (2016)

    Article  Google Scholar 

  • Harrison, J., et al.: Cognitive workload and learning assessment during the implementation of a next-generation air traffic control technology using functional near-infrared spectroscopy. IEEE Trans. Hum.-Mach. Syst. 44(4), 429–440 (2014)

    Article  Google Scholar 

  • Tsai, M.-J., Hou, H.-T., Lai, M.-L., Liu, W.-Y., Yang, F.-Y.: Visual attention for solving multiple-choice science problem: an eye-tracking analysis. Comput. Educ. 58(1), 375–385 (2012)

    Article  Google Scholar 

  • van der Wel, P., van Steenbergen, H.: Pupil dilation as an index of effort in cognitive control tasks: a review. Psychon. Bull. Rev. 25(6), 2005–2015 (2018)

    Article  Google Scholar 

  • ICAO.: Doc 9328: Manual of Runway Visual Range Observing and Reporting Practices. International Civil Aviation Organization, 3rd edn. (2005)

    Google Scholar 

  • van Schaik, F.J., Roessingh, J.J.M., Lindqvist, G., Fält, K.: Assessment of visual cues by tower controllers, with implications for a remote tower control centre. IFAC Proc. Vol. 43(13), 123–128 (2010)

    Article  Google Scholar 

  • Ha, C.H., Kim, J.H., Lee, S.J., Seong, P.H.: Investigation on relationship between information flow rate and mental workload of accident diagnosis tasks in NPPs. IEEE Trans. Nucl. Sci. 53(3), 1450–1459 (2006)

    Article  Google Scholar 

  • Di Flumeri, G., et al.: Brain–computer interface-based adaptive automation to prevent out-of-the-loop phenomenon in air traffic controllers dealing with highly automated systems. Front. Hum. Neurosci. 13, 296 (2019)

    Article  Google Scholar 

  • Bernhardt, K.A., et al.: The effects of dynamic workload and experience on commercially available EEG cognitive state metrics in a high-fidelity air traffic control environment. Appl. Ergon. 77, 83–91 (2019)

    Article  Google Scholar 

  • Matthews, G., Reinerman-Jones, L.E., Barber, D.J., Abich, J.: The psychometrics of mental workload: multiple measures are sensitive but divergent. Hum. Factors 57(1), 125–143 (2015)

    Article  Google Scholar 

  • Hogervorst, M.A., Brouwer, A.M., van Erp, J.B.F.: Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload. Front. Neurosci. 8, 1–15 (2014)

    Article  Google Scholar 

  • Komogortsev, O.V., Karpov, A.: Automated classification and scoring of smooth pursuit eye movements in the presence of fixations and saccades. Behav. Res. Methods 45, 203–215 (2013)

    Article  Google Scholar 

  • Wei, T., Simko, V.: R package “corrplot”: visualization of a correlation matrix (version 0.84) (2017)

    Google Scholar 

  • Engelhardt, P.E., Ferreira, F., Patsenko, E.G.: Rapid communication pupillometry reveals processing load during spoken language comprehension. Q. J. Exp. Psychol. 63(4), 639–645 (2010)

    Article  Google Scholar 

  • Wickham, H.: R Package “ggplot2”: Elegant Graphics for Data Analysis. Springer, New York (2016). https://doi.org/10.1107/978-0-387-98141-3

    Book  MATH  Google Scholar 

  • Hansen, J.P., Hardenberg, D., Biermann, F., Bækgaard, P.: A gaze interactive assembly instruction with pupillometric recording. Behav. Res. Methods 50(4), 1723–1733 (2018)

    Article  Google Scholar 

  • Eckstein, M.K., Guerra-Carrillo, B., Miller Singley, A.T., Bunge, S.A.: Beyond eye gaze: what else can eyetracking reveal about cognition and cognitive development? Dev. Cogn. Neurosci. 25, 69–91 (2017)

    Article  Google Scholar 

  • Alemdag, E., Cagiltay, K.: A systematic review of eye tracking research on multimedia learning. Comput. Educ. 125, 413–428 (2018)

    Article  Google Scholar 

  • Athènes, S., Averty, P., Puechmorel, S., Delahaye, D., Collet, C.: ATC complexity and controller workload: trying to bridge the gap. www.aaai.org. Accessed 10 Feb 2020

  • Delpy, D.T., Cope, M., van der Zee, P.: Estimation of optical path length through tissue from direct time of flight measurement. Phys. Med. Biol. 33(12), 1433–1442 (1988)

    Article  Google Scholar 

  • Herrmann, M.J., et al.: D4 receptor gene variation modulates activation of prefrontal cortex during working memory. Eur. J. Neurosci. 26(10), 2713–2718 (2007)

    Article  Google Scholar 

  • Hilburn, B., Jorna, P.G.A.M.: Workload and air traffic control. In: Human Factors in Transportation. Stress, Workload, and Fatigue, pp. 384–394 (2001)

    Google Scholar 

  • Izzetoglu, M., Izzetoglu, K., Bunce, S., Ayaz, H., Devaraj, A., Onaral, B.: Functional Near-Infrared Neuroimaging. IEEE Trans. Neural Syst. Rehabil. Eng. 13(2), 153–159 (2005)

    Article  Google Scholar 

  • Jobsis, F.F.: Noninvasive, infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. Science 198(4323), 1264–1267 (1977)

    Article  Google Scholar 

  • Kirwan, B.: Validation of human reliability assessment of techniques: part 1 - Validation issues. Saf. Sci. 27(1), 25–41 (1997)

    Article  Google Scholar 

  • Kirwan, B., Scaife, R., Kennedy, R.: Investigating complexity factors in UK air traffic management. In: Engineering Psychology and Cognitive Ergonomics, pp. 189–195 (2017)

    Google Scholar 

  • Stein, E.S.: Air traffic controller workload: an examination of workload probe. Atlantic City International Airport: Federal Aviation Administration Technical Center. (DOT/FAA/CT-TN84/24) (1985)

    Google Scholar 

  • Koros, A., Rocco, P.S., Panjwani, G., Ingurgio, V., D’Arcy, J.F.: Complexity in air traffic control towers: a field study. Part 1: complexity factors (2003)

    Google Scholar 

  • Martijn Jansma, J., Ramsey, N.F., Coppola, R., Kahn, R.S.: Specific versus nonspecific brain activity in a parametric N-back task. Neuroimage 12, 688–697 (2000)

    Article  Google Scholar 

  • Matsuo, K., et al.: Prefrontal hyperactivation during working memory task in untreated individuals with major depressive disorder. Mol. Psychiatry 12(2), 158–166 (2007)

    Article  Google Scholar 

  • Milton, J.G., Small, S.S., Solodkin, A.: On the road to automatic: dynamic aspects in the development of expertise. J. Clin. Neurophysiol. 21(3), 134–143 (2004)

    Article  Google Scholar 

  • Veltman, D.J., Rombouts, S.A.R.B., Dolan, R.J.: Maintenance versus manipulation in verbal working memory revisited: an fMRI study. NeuroImage 18(2), 247–256 (2003)

    Article  Google Scholar 

  • Villringer, A., Chance, B.: Non invasive optical spectroscopy and imaging of human brain function. Trends Neurosci. 20(10), 435–442 (1997)

    Article  Google Scholar 

  • Wickens, C.D., Hollands, J.G.: Engineering Psychology and Human Performance, 3rd edn. Prentice Hall, Upper Saddle River (2000). N. Roberts & B. Webber

    Google Scholar 

  • Wickens, C.D.: Situation awareness and workload in aviation. Curr. Dir. Psychol. Sci. 11(4), 128–133 (2002)

    Article  Google Scholar 

  • Izzetoglu, K., Bunce, S., Onaral, B., Pourrezaei, K., Chance, B.: Functional optical brain imaging using near-infrared during cognitive tasks. Int. J. Hum.-Comput. Interact. 17(2), 211–227 (2004)

    Article  Google Scholar 

  • Reddy, P., Richards, D., Izzetoglu, K.: Cognitive performance assessment of UAS sensor operators via neurophysiological measures. Front. Hum. Neurosci. 12 (2018)

    Google Scholar 

  • Izzetoglu, K., Richards, D.: Human performance assessment: evaluation of wearable sensors for monitoring brain activity. In: Vidulich, M., Tsang, P. (eds.) Improving Aviation Performance through Applying Engineering Psychology: Advances in Aviation Psychology, 1st edn, pp. 163–180. CRC Press, Boca Raton (2019)

    Chapter  Google Scholar 

Download references

Acknowledgment

The authors would like to gratefully thank and recognize Dr. Jerry Crutchfield (FAA – Civil Aeronautical Medical Insitute. Oklahoma City, Oklahoma) for his continuous support through the experimental protocol, data acquisition phases of the experiment; and for facilitating access to the simulation environment. In addition, his comments throughout the writing process of this paper were invaluable.

Author information

Authors and Affiliations

  1. Department of Industrial and Systems Engineering, University of Oklahoma, Norman, USA

    Ricardo Palma Fraga & Ziho Kang

  2. Drexel University, School of Biomedical Engineering, Science and Health Systems, Philadelphia, USA

    Pratusha Reddy & Kurtulus Izzetoglu

Authors
  1. Ricardo Palma Fraga
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pratusha Reddy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ziho Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kurtulus Izzetoglu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ricardo Palma Fraga .

Editor information

Editors and Affiliations

  1. Soar Technology Inc., Orlando, FL, USA

    Dylan D. Schmorrow

  2. Design Interactive, Inc., Orlando, FL, USA

    Cali M. Fidopiastis

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Palma Fraga, R., Reddy, P., Kang, Z., Izzetoglu, K. (2020). Multimodal Analysis Using Neuroimaging and Eye Movements to Assess Cognitive Workload. In: Schmorrow, D.D., Fidopiastis, C.M. (eds) Augmented Cognition. Theoretical and Technological Approaches. HCII 2020. Lecture Notes in Computer Science(), vol 12196. Springer, Cham. https://doi.org/10.1007/978-3-030-50353-6_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-50353-6_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-50352-9

  • Online ISBN: 978-3-030-50353-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation