Skip to main content
SpringerLink
Log in

Saccade response testing during teleoperations with a head-mounted display

  • Original Article
  • Published:
Cognition, Technology & Work Aims and scope Submit manuscript

Abstract

Head-mounted displays (HMD) are increasingly used for teleoperating robots that stream video to the operator in real time. Eye-tracking sensors that can record saccadic eye movements non-intrusively are becoming a standard feature in HMDs. Saccade tests have been widely used in human factors research and can indicate operators’ mental task resources. However, previous work has not yet explored saccade tests in robot teleoperation using an HMD. We implemented a saccade response test (SRT) and the Situation Present Assessment Method (SPAM) and conducted an experiment with 32 operators navigating a telerobot via an HMD. The SRT was significantly correlated with the SPAM metrics (response time to the prompt test and correctness of answers to SPAM queries), the performance metrics (the number of collisions, and the task completion time), and the NASA Task Load Index ratings (Physical Demand). SRT was found to be more than 10 times faster than the SPAM prompt response test as an indicator of mental workload, that is, 291 ms vs. 3125 ms. SRT was highly correlated with the SPAM prompt test (r = 0.73) and moderately correlated with task completion time and the number of collisions. SRT was able to distinguish the eight best robot operators from the eight weakest performers. Our results suggest SRT be considered part of the assessments of mental task resources, for instance, when selecting operators for industrial teleoperations with HMDs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  • Allspaw J, Heinold L, Yanco HA (2019) Design of virtual reality for humanoid robots with inspiration from video games. In: International conference on human-computer interaction, pp 3–18

  • Almeida L, Menezes P, Dias J (2017) Improving robot teleoperation experience via immersive interfaces. In: 2017 4th Experiment@ international conference (exp. at’17), pp 87–92

  • Annie L, Basil MD (1998) Attention, resource allocation, and communication research: what do secondary task reaction times measure, anyway? Ann Int Commun Assoc 21(1):443–458

    Google Scholar 

  • Argyle EM, Gourley JJ, Kang Z, Shehab RL (2020) Investigating the relationship between eye movements and situation awareness in weather forecasting. Appl Ergono 85:103071

    Article  Google Scholar 

  • Bafna T, Hansen JP (2021) Mental fatigue measurement using eye metrics: a systematic literature review. Psychophysiology 58(6):e13828

    Article  PubMed  Google Scholar 

  • Bitkina OV, Park J, Kim HK (2021) The ability of eye-tracking metrics to classify and predict the perceived driving workload. Int J Indust Ergon 86:103193

    Article  Google Scholar 

  • Bracken B, Tobyne S, Winder A, Shamsi N, Endsley MR (2021) Can situation awareness be measured physiologically? In: International conference on applied human factors and ergonomics, pp 31–38

  • Brizzi F, Peppoloni L, Graziano A, Di Stefano E, Avizzano CA, Ruffaldi E (2017) Effects of augmented reality on the performance of teleoperated industrial assembly tasks in a robotic embodiment. IEEE Trans Human-Mach Sys 48(2):197–206

    Article  Google Scholar 

  • Cassin B, Solomon S, Rubin ML (1990) Dictionary of eye terminology. Triad Publishing Company, Gainsville, Florida

    Google Scholar 

  • Charles RL, Nixon J (2019) Measuring mental workload using physiological measures: a systematic review. Appl Ergon 74:221–232

    Article  PubMed  Google Scholar 

  • Charlton SG, O’Brien TG (2019) Handbook of human factors testing and evaluation. CRC Press

    Book  Google Scholar 

  • Chen S, Epps J, Ruiz N, Chen F (2011) Eye activity as a measure of human mental effort in HCI. In: Proceedings of the 16th international conference on Intelligent user interfaces, pp 315–318

  • Coral MP (2016) Analyzing cognitive workload through eye-related measurements: a meta-analysis. Doctoral dissertation, Wright State University

  • Das S, Maiti J, Krishna OB (2020) Assessing mental workload in virtual reality based EOT crane operations: a multi-measure approach. Int J Indust Ergon 80:103017

    Article  Google Scholar 

  • de Winter JCF, Eisma Yke B, Cabrall CDD, Hancock Peter A, Stanton NA (2019) Situation awareness based on eye movements in relation to the task environment. Cogn Technol Work 21(1):99–111

    Article  Google Scholar 

  • Di Stasi LL, Renner R, Staehr P, Helmert JR, Velichkovsky Boris M, Canas Jose J, Andres C, Sebastian P (2010) Saccadic peak velocity sensitivity to variations in mental workload. Aviation, Space, Environ Med 81(4):413–417

    Article  Google Scholar 

  • Di Stasi LL, Renner R, Catena A, Canas Jose J, Velichkovsky Boris M, Sebastian P (2012) Towards a driver fatigue test based on the saccadic main sequence: a partial validation by subjective report data. Trans Res Part C Emerg Technol 21(1):122–133

    Article  Google Scholar 

  • Di Stasi LL, McCamy MB, Macknik SL, Mankin JA, Hooft N, Catena A, Martinez-Conde S (2014) Saccadic eye movement metrics reflect surgical residents’ fatigue. Ann Surg 259(4):824–829

    Article  PubMed  Google Scholar 

  • Di Stasi LL, McCamy MB, Martinez-Conde S, Gayles E, Hoare C, Foster M, Catena A, Macknik SL (2016) Effects of long and short simulated flights on the saccadic eye movement velocity of aviators. Physiol Behav 153:91–96

    Article  PubMed  Google Scholar 

  • Diaz-Piedra C, Rieiro H, Suarez J, Francisco R-T, Andres C, Luigi DSL (2016) Fatigue in the military: towards a fatigue detection test based on the saccadic velocity. Physiol Measurement 37(9):62

    Article  Google Scholar 

  • Durso FT, Hackworth CA, Truitt TR, Jerry C, Danko N, Manning Carol A (1998a) Situation awareness as a predictor of performance for en route air traffic controllers. Air Traffic Control Quarterly. 6(1):1–20

    Article  Google Scholar 

  • Durso FT, Truitt TR, Hackworth CA, Crutchfield Jerry M, Manning CA (1998b) En route operational errors and situational awareness. Int J Aviat Psychol 8(2):177–194

    Article  Google Scholar 

  • Durso FT, Bleckley MK, Dattel AR (2006) Does situation awareness add to the validity of cognitive tests? Hum Factors 48(4):721–733

    Article  PubMed  Google Scholar 

  • Endsley MR (1993) Situation awareness and workload-flip sides of the same coin. In: International symposium on aviation psychology, 7th, Columbus, OH, pp 906–911

  • Endsley MR (1995) Measurement of situation awareness in dynamic systems. Hum Factors 37(1):65–84

    Article  Google Scholar 

  • Endsley MR (2000) Direct measurement of situation awareness: validity and use of SAGAT. Situation awareness analysis and measurement. Lawrence Erlbaum Associates, Mahwah NJ, pp 147–173

    Book  Google Scholar 

  • Endsley MR (2021) A systematic review and meta-analysis of direct objective measures of situation awareness: a comparison of SAGAT and SPAM. Hum Factors 63(1):124–150

    Article  PubMed  Google Scholar 

  • Ericsson KA, Smith J (1991) Toward a general theory of expertise: prospects and limits. Cambridge University Press

    Google Scholar 

  • Frazier S, Pitts BJ, McComb S (2022) Measuring cognitive workload in automated knowledge work environments: a systematic literature review. Cogn Technol Work 24(4):557–587

    Article  Google Scholar 

  • Greef TD, Lafeber H, Oostendorp H van, Lindenberg J (2009) Eye movement as indicators of mental workload to trigger adaptive automation. In: International conference on foundations of augmented cognition, pp 219–228

  • Green EA, Armstrong ME, Mantua J (2023) Scientific measurement of situation awareness in operational testing. ITEA J Test Eval 44:3

    Google Scholar 

  • Guidetti G, Guidetti R, Manfredi M, Manfredi M, Lucchetta A, Livio S (2019) Saccades and driving. Acta Otorhinolaryngol Ital 39(3):186

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Guo Y, Freer D, Deligianni F, Yang G (2022) Eye-tracking for performance evaluation and workload estimation in space telerobotic training. IEEE Trans Human-Mach Sys 52(1):1–11

    Article  Google Scholar 

  • Hart SG, Staveland LE (1988) Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv in Psychol 52(8):139–183

    Article  Google Scholar 

  • Hasanzadeh S, Esmaeili B, Dodd MD (2018) Examining the relationship between construction workers’ visual attention and situation awareness under fall and tripping hazard conditions: using mobile eye tracking. J Constr Eng Manag 144(7):04018060

    Article  Google Scholar 

  • Hirvonen K, Puttonen S, Gould K, Jussi K, Koefoed VF, Mueller K (2010) Improving the saccade peak velocity measurement for detecting fatigue. J Neurosci Methods 187(2):199–206

    Article  PubMed  Google Scholar 

  • Holland C, Komogortsev OV (2011) Biometric identification via eye movement scanpaths in reading. In: 2011 International joint conference on biometrics (IJCB), pp 1–8

  • IJsselsteijn W, Bierhoff I, Slangen-de KY (2001) Duration estimation and presence. In: Presence 2001 conference

  • Illing B, Westhoven M, Gaspers B, Nanja S, Bernd B, Tintu M (2020) Evaluation of immersive teleoperation systems using standardized tasks and measurements. In: 29th IEEE international conference on robot and human interactive communication, pp 278–285

  • Jankowski J, Grabowski A (2015) Usability evaluation of vr interface for mobile robot teleoperation. Int J Human-Comput Interact 31(12):882–889

    Article  Google Scholar 

  • Jerald J (2015) The VR book: human-centered design for virtual reality. Morgan & Claypoo

    Book  Google Scholar 

  • Jones DG, Endsley MR (1996) Sources of situation awareness errors in aviation. Aviation, Space, Environ Med 67(6):507–512

    CAS  Google Scholar 

  • Kot TS, Novak P (2018) Application of virtual reality in teleoperation of the military mobile robotic system TAROS. Int J Adv Robot Sys 15(1):1729881417751545

    Google Scholar 

  • Lai HY, Saavedra-Pena G, Sodini CG, Vivienne S, Thomas H (2020) Measuring saccade latency using smartphone cameras. IEEE J Biomed Health Inform 24(3):885–897

    Article  PubMed  Google Scholar 

  • Lai H-Y, Saavedra-Pena G, Sodini C, Heldt T, Sze V (2022) App-based saccade latency and directional error determination across the adult age spectrum. IEEE Trans Biomed Eng 69(2):1029–1039

    Article  PubMed  Google Scholar 

  • Li WC, Zhang J, Court S, Kearney P, Braithwaite G (2022) The influence of augmented reality interaction design on Pilot’s perceived workload and situation awareness. Int J Indust Ergon 92:103382

    Article  Google Scholar 

  • Liang N, Yang J, Zhang T, Pitts B, Yu D (2020) What can eye tracking reveal about situation awareness? A systematic review. Proc Human Factors Ergon Soc Ann Meeting 64(1):1631

    Article  Google Scholar 

  • Linkov V, Vanzura M (2021) Situation awareness measurement in remotely controlled cars. Front Psychol 12:762

    Article  Google Scholar 

  • Livatino S, De Paolis LT, D'Agostino M, Zocco A, Agrimi A, De Santis A, Bruno LV, Lapresa M (2014) Stereoscopic visualization and 3D technologies in medical endoscopic teleoperation. IEEE Trans Indust Electron 62(1):525–535

    Article  Google Scholar 

  • Longo L, Wickens CD, Hancock G, Hancock PA (2022) Human mental workload: a survey and a novel inclusive definition. Front Psychol 13:883321

    Article  PubMed  PubMed Central  Google Scholar 

  • Marchitto M, Benedetto S, Baccino T, Juan CJ (2016) Air traffic control: ocular metrics reflect cognitive complexity. Int J Indust Ergon 54:120–130

    Article  Google Scholar 

  • Mardanbegi D, Wilcockson T, Sawyer P, Hans G, Trevor G (2019) SaccadeMachine: software for analyzing saccade tests (anti-saccade and pro-saccade). In: Proceedings of the 11th ACM symposium on eye tracking research and applications, pp 1–8

  • Martins H, Oakley I, Ventura R (2015) Design and evaluation of a head-mounted display for immersive 3D teleoperation of field robots. Robotica 33(10):2166–2185

    Article  Google Scholar 

  • Mast M, Materna Z, Spanel M, Weisshardt F, Arbeiter G, Burmester M, Smrz P, Graf B (2015) Semi-autonomous domestic service robots: evaluation of a user interface for remote manipulation and navigation with focus on effects of stereoscopic display. Int J Soc Robot 7(2):183–202

    Article  Google Scholar 

  • Maurage P, Masson N, Bollen Z, D’Hondt F (2020) Eye tracking correlates of acute alcohol consumption: a systematic and critical review. Neurosci Biobehav Rev 108:400–422

    Article  PubMed  Google Scholar 

  • May JG, Kennedy RS, Williams MC, Dunlap William P, Brannan JR (1990) Eye movement indices of mental workload. Acta Physiol (oxf) 75(1):75–89

    CAS  Google Scholar 

  • Moore K, Gugerty L (2010) Development of a novel measure of situation awareness: the case for eye movement analysis. Proc Human Factors Ergon Soc Ann Meeting 54(19):1650–1654

    Article  Google Scholar 

  • Opiyo S, Zhou J, Mwangi E, Wang K, Idris S (2021) A review on teleoperation of mobile ground robots: architecture and situation awareness. Int J Control, Autom Sys 19:1384–1407

    Article  Google Scholar 

  • Pratt J, Trottier L (2005) Pro-saccades and anti-saccades to onset and offset targets. Vision Res 45(6):765–774

    Article  PubMed  Google Scholar 

  • Ragan ED, Kopper R, Schuchardt P, Bowman DA (2012) Studying the effects of stereo, head tracking, and field of regard on a small-scale spatial judgment task. IEEE Trans vis Comput Graph 19(5):886–896

    Article  Google Scholar 

  • Riley JM, Kaber DB, Draper JV (2004) Situation awareness and attention allocation measures for quantifying telepresence experiences in teleoperation. Human Factors Ergon Manuf 14:51–67

    Article  Google Scholar 

  • Ruff HA, Narayanan S, Draper MH (2002) Human interaction with levels of automation and decision-aid fidelity in the supervisory control of multiple simulated unmanned air vehicles. Presence-Teleoperators Virtual Environ 11:335–351

    Article  Google Scholar 

  • Salmon P, Stanton N, Walker G, Green D (2006) Situation awareness measurement: a review of applicability for C4i environments. Appl Ergon 37(2):225–238

    Article  PubMed  Google Scholar 

  • Schmitt KU, Seeger R, Fischer H, Lanz C (2015) Muser Markus, Walz Felix, Schwarz Urs. Saccadic eye movement performance as an indicator of driving ability in elderly drivers. Swiss Med Wkly 145:w14098

    PubMed  Google Scholar 

  • Schwind V, Knierim P, Haas N, Henze N (2019) Using presence questionnaires in virtual reality. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp 1–12

  • Sun D, Kiselev A, Liao Q, Stoyanov T, Loutfi A (2020) A new mixed-reality-based teleoperation system for telepresence and maneuverability enhancement. IEEE Trans Human-Mach Sys 5(1):55–67

    Article  Google Scholar 

  • Tall M, Alapetite A, San Agustin J, Skovsgaard HH, Hansen JP, Hansen DW, Møllenbach E (2009) Gaze-controlled driving. In: CHI'09 extended abstracts on human factors in computing systems, pp 4387–4392

  • Taylor RM (1989) Situational awareness rating technique (SART): the development of a tool for aircrew systems design. In: Proceedings of the AGARD AMP symposium on situational awareness in aerospace operations, pp CP478

  • Tokuda S, Obinata G, Palmer E, Chaparro A (2011) Estimation of mental workload using saccadic eye movements in a free-viewing task. In: 2011 annual international conference of the IEEE engineering in medicine and biology society, pp 4523–4529

  • van de Merwe K, van Dijk H, Zon R (2012) Eye movements as an indicator of situation awareness in a flight simulator experiment. The Int J Aviat Psychol 22(1):78–95

    Article  Google Scholar 

  • Vanderhaegen F, Wolff M, Mollard R (2023) Repeatable effects of synchronizing perceptual tasks with heartbeat on perception-driven situation awareness. Cogn Sys Res 81:80–92

    Article  Google Scholar 

  • Wierwille WW, Eggemeier FT (1993) Recommendations for mental workload measurement in a test and evaluation environment. Hum Factors 35(2):263–281

    Article  Google Scholar 

  • Wu C, Cha J, Sulek J, Zhou T, Sundaram CP, Wachs J, Yu D (2020) Eye-tracking metrics predict perceived workload in robotic surgical skills training. Hum Factors 62(8):1365–1386

    Article  PubMed  Google Scholar 

  • Zhang G, Hansen JP 2019 A virtual reality simulator for training gaze control of wheeled tele-robots. 25th ACM Symposium on Virtual Reality Software and Technology, 1–2

  • Zhang G, Hansen JP (2022) Telepresence robots for people with special needs: a systematic review. Int J Human–comput Interact 38:1–17

    Article  Google Scholar 

  • Zhang G, Hansen JP, Minakata K (2019) Hand- and gaze-control of telepresence robots. In: Proceedings of the 11th ACM symposium on eye tracking research & applications (ETRA '19). Association for Computing Machinery, New York, NY, Article 70, pp 1–8. https://doi.org/10.1145/3317956.3318149

  • Zhang T, Yang J, Liang N, Pitts BJ, Prakah-Asante K, Curry R, Duerstock B, Wachs JP, Yu D (2023) Physiological measurements of situation awareness: a systematic review. Hum Factors 65(5):737–758

    Article  PubMed  Google Scholar 

  • Zhao L, Nybacka M, Rothhamel M (2023) A survey of teleoperation: driving feedback. In: IEEE intelligent vehicles symposium, Anchorage, Alaska, USA, June 4–7, 2023

Download references

Acknowledgements

The research has been supported by the Horizon 2020 EU project ReHyb (grant agreement number 871767), the Bevica Foundation, the Shuimu Tsinghua Scholar Program, and the China Scholarship Council. The authors wish to thank Diako Mardanbegi for assistance with the SaccadeMachine software.

Author information

Authors and Affiliations

  1. BNRIST and Department of Automation, Tsinghua University, Beijing, 100084, China

    Guangtao Zhang

  2. Department of Technology, Management and Economics, Technical University of Denmark, 2800, Kgs.Lyngby, Denmark

    Sebastian Hedegaard Hansen, Oliver Repholtz Behrens & John Paulin Hansen

Authors
  1. Guangtao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sebastian Hedegaard Hansen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Oliver Repholtz Behrens
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. John Paulin Hansen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

G.Z., S.H.H. and O.R.B. conducted the experiment and analyzed the data. G.Z. and J.P.H. wrote the main manuscript.

Corresponding author

Correspondence to Guangtao Zhang.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, G., Hansen, S.H., Behrens, O.R. et al. Saccade response testing during teleoperations with a head-mounted display. Cogn Tech Work 26, 127–138 (2024). https://doi.org/10.1007/s10111-023-00750-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10111-023-00750-6

Keywords

Search

Navigation