Skip to main content
SpringerLink
Log in

Predictive aids can lead to sustained attention decrements in the detection of non-routine critical events in event monitoring

  • Original Article
  • Published:
Cognition, Technology & Work Aims and scope Submit manuscript

Abstract

Task performance in uncertain and complex environments is dependent on human effectiveness in dynamic prioritisation and attention allocation. Non-routine critical events place a high demand on cognitive resources in operators, and they require operators to dynamically re-prioritise sub-tasks calling upon the need for predictive aids. Predictive aids provide real-time prediction of system variables, and they can potentially facilitate the detection of non-routine critical events and dynamic re-prioritisation during such events. However, prior research has reported mixed findings in their effectiveness, and no prior research has been done to test the effectiveness of predictive aids during non-routine critical events that require operators to dynamically prioritise tasks. Our experimental study with 77 participants examined the effect of a predictive aid on prioritisation of non-routine critical events in a cyber security event monitoring task. The predictive aid resulted in decrements in sustained attention as seen in delayed detection of some non-routine critical events and errors in prioritising non-routine critical events. This effect is a result of miscalibration in the importance of looking ahead with the predictive aid. This miscalibration was possibly associated with the look-ahead time, complexity of the predictive aid, and the lack of automation transparency (i.e. mechanism underlying prediction) together affecting its perceived usefulness. The experimental results have the following implications on the design and testing of predictive aids: (1) in a task requiring dynamic prioritisation and detection of non-routine critical events, predictive aids with insufficient look-ahead times can result in decrements in sustained attention; (2) predictive aids should be evaluated with subjective measures of perceived usefulness and workload; (3) predictive aids should be designed to have sufficient look-ahead times and transparency of underlying prediction mechanisms as these would affect visual scanning effort and perceived usefulness; and (4) experimental tests of their effectiveness should involve scenarios long enough that would take sustained attention effects and any potential learning effects into consideration.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Andre AD, Heers S, Cashion P (1995) Effects of workload preview on task scheduling during simulated instrument flight. Int J Aviat Psychol 5(1):5–23

    Article  Google Scholar 

  • Chou CC, Madhavan D, Funk K (1996) Studies of cockpit task management errors. Int J Aviat Psychol 6(4):307–320

    Article  Google Scholar 

  • Colvin K, Funk K, Braune R (2005) Task prioritization factors: two part-task simulator studies. Int J Aviat Psychol 15(4):321–338

    Article  Google Scholar 

  • Craik FI, Lockhart RS (1972) Levels of processing: a framework for memory research. J Verbal Learn Verbal Behav 11(6):671–684

    Article  Google Scholar 

  • Cummings M, Mitchell P (2005) Managing multiple UAV’s through a timeline display. In: Proceedings of AIAA info tech, 2005

  • Cvach M (2012) Monitor alarm fatigue: an integrative review. Biomed Instrum Technol 46(4):268–277

    Article  Google Scholar 

  • D’Amico A, Whitley K, Tesone D, O’Brien B, Roth E (2005) Achieving cyber defense situational awareness: a cognitive task analysis of information assurance analysts. In: Proceedings of the human factors and ergonomics society annual meeting, vol 49, no 3, pp 229–233

  • Donald FM, Donald CH (2015) Task disengagement and implications for vigilance performance in CCTV surveillance. Cognit Technol Work 17(1):121–130

    Article  Google Scholar 

  • Endsley M, Garland D (2000) Situation awareness analysis and measurement. CRC Press, Taylor & Francis Group

  • Funk K (1991) Cockpit task management: preliminary definitions, normative theory, error taxonomy, and design recommendations. Int J Aviat Psychol 1(4):271–285

    Article  Google Scholar 

  • Ghazizadeh M, Lee JD, Boyle LN (2012) Extending the technology acceptance model to assess automation. Cognit Technol Work 14(1):39–49

    Article  Google Scholar 

  • Giacobe NA, McNeese MD, Mancuso VF, Minotra D (2013) Capturing human cognition in cyber-security simulations with NETS. In: IEEE Int Conf Intell Secur Inform (ISI), pp 284–288

  • Grier RA, Warm JS, Dember WN, Matthews G, Galinsky TL, Szalma JL, Parasuraman R (2003) The vigilance decrement reflects limitations in effortful attention, not mindlessness. Hum Factors J Hum Factors Ergon Soc 45(3):349–359

    Article  Google Scholar 

  • Hamilton K, Mancuso B, Minotra D, Hoult R, Mohammed S et al (2010) Using the NeoCITIES 3.1 simulation to study and measure team cognition. In: Proceedings of the human factors and ergonomics society annual meeting, vol 54, no 4, pp 433–437

  • Hart S, Staveland L (1988) Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Hancock P, Meshkati N (eds) Human mental workload, vol 52. North-Holland, Amsterdam, pp 139–183

    Chapter  Google Scholar 

  • Hellar B, McNeese MD (2010) NeoCITIES: a simulated command and control task environment for experimental research. In: Proceedings of the human factors and ergonomics society annual meeting, vol 54, no 13, pp 1027–1031

  • Helton WS, Russell PN (2011) Working memory load and the vigilance decrement. Exp Brain Res 212(3):429–437

    Article  Google Scholar 

  • Hoffman R, Johnson M, Bradshaw M, Underbrink A (2013) Trust in automation. IEEE Intell Syst 28(1):84–88

    Article  Google Scholar 

  • Iani C, Wickens CD (2007) Factors affecting task management in aviation. Hum Factors J Hum Factors Ergon Soc 49(1):16–24

    Article  Google Scholar 

  • Lau N, Jamieson GA, Skraaning G (2013) Distinguishing three accounts of situation awareness based on their domains of origin. In: Proceedings of the human factors and ergonomics society annual meeting, vol 57, no 1, pp 220–224

  • Lee JD, McGehee DV, Brown TL, Reyes ML (2002) Collision warning timing, driver distraction, and driver response to imminent rear-end collisions in a high-fidelity driving simulator. Hum Factors J Hum Factors Ergon Soc 44(2):314–334

    Article  Google Scholar 

  • Mancuso V, Minotra D, Giacobe N, McNeese MD, Tyworth M (2012) idsNETS: an experimental platform to study situation awareness for intrusion detection analysts. In: Proceedings of the IEEE CogSIMA conference, New Orleans, pp 73–79

  • Metzger U, Parasuraman R (2005) Automation in future air traffic management: effects of decision aid reliability on controller performance and mental workload. Hum Factors J Hum Factors Ergon Soc 47(1):35

    Article  Google Scholar 

  • Minotra D (2012) The effect of a workload-preview on task-prioritization and task-performance. Dissertation, The Pennsylvania State University, University Park

  • Morris C, Leung Y (2006) Pilot mental workload: How well do pilots really perform? Ergonomics 49(15):1581–1596

    Article  Google Scholar 

  • Nunes A (2003) The impact of automation use on the mental model: findings from the air traffic control domain. In: Proceedings of the human factors and ergonomics society annual meeting, vol 47, no 1, pp 66–70

  • Nunes A, St-Cyr O (2003) Predictive aids and mental models under free flight: proceed with caution. In: Proceedings of the 12th international symposium of aviation psychology, Dayton

  • Parasuraman R (2005) Workload modeling and workload management: recent theoretical developments. Department of Psychology, George Mason University, Fairfax

    Google Scholar 

  • Prinet JC, Terhune A, Sarter NB (2012) Supporting dynamic re-planning in multiple UAV control: a comparison of 3 levels of automation. In: Proceedings of the human factors and ergonomics society annual meeting, vol 56, no 1, pp 423–427

  • Raby M, Wickens CD (1994) Strategic workload management and decision biases in aviation. Int J Aviat Psychol 4(3):211–240

    Article  Google Scholar 

  • Roberts R, Flin R, Cleland J (2014) Staying in the zone offshore drillers’ situation awareness. Hum Factors J Hum Factors Ergon Soc. doi:10.1177/0018720814562643

    Google Scholar 

  • Rummel J, Meiser T (2013) The role of metacognition in prospective memory: anticipated task demands influence attention allocation strategies. Conscious Cognit 22(3):931–943

    Article  Google Scholar 

  • Smilek D, Eastwood D, Reynolds M, Kingstone A (2007) Metacognitive errors in change detection: missing the gap between lab and life. Conscious Cognit 16(1):52–57

    Article  Google Scholar 

  • Solet JM, Barach PR (2012) Managing alarm fatigue in cardiac care. Progress Pediatr Cardiol 33(1):85–90

    Article  Google Scholar 

  • Szafir D, Mutlu B (2012) Pay attention!: designing adaptive agents that monitor and improve user engagement. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 11–20

  • Vicente KJ (1999) Cognitive work analysis: toward safe, productive, and healthy computer-based work. CRC Press, Taylor & Francis Group

  • Wellens R, Ergener D (1988) The CITIES game. Simul Gaming 19(3):304–327

    Article  Google Scholar 

  • Wickens CD (2001) Attention to safety and the psychology of surprise. In: Proceedings of the 2001 symposium on aviation psychology

  • Wickens CD (2015) Situation awareness its applications value and its fuzzy dichotomies. J Cognit Eng Decis Mak 9(1):90–94

    Article  Google Scholar 

  • Wickens CD, Pizzaro D, Bell B (1991) Overconfidence, preview, and probability in strategic planning. In: Proceedings of the 35th human factors and ergonomics society meeting, vol 35, no 20, pp 1556–1560

  • Wickens CD, Heather D, Merlot J (1999) Integration of information sources of varying weights: the effect of display features and attention cueing. Savoy Aviation Research Lab, University of Illinois, Urbana-Champaign

    Google Scholar 

  • Yeh M, Wickens CD, Seagull F (1999) Target cuing in visual search: the effects of conformality and display location on the allocation of visual attention. Hum Factors J Hum Factors Ergon Soc 41(4):524–542

    Article  Google Scholar 

  • Zhang G, Feyen RG (2005) A conceptual framework for dynamic prioritization in multiple-task scenarios. In: Proceedings of the human factors and ergonomics society annual meeting, vol 49, no 12, pp 1172–1175

  • Zhang G, Feyen RG (2007a) Hierarchical task prioritization behavior in two-and four-task scenarios. In: Proceedings of the human factors and ergonomics society annual meeting, vol 51, no 4, pp 191–195

  • Zhang G, Feyen RG (2007b) A queueing network model of task prioritization using a general hierarchy of prioritization rules. In: Proceedings of the human factors and ergonomics society annual meeting, vol 51, no 12, pp 807–811

Download references

Acknowledgements

This work was partially supported by US Army Research Office (ARO) MURI Grant ‘Computer Aided Human Centric Cyber Situation Awareness’ W911-NF-09-1-0525. We would like to thank members of the MINDS Lab at Penn State for their support. We would like to thank the reviewers for their insightful comments. Additionally, the support and input from Dr. Catherine Burns, Dr. Xiaolong Zhang, and Dr. Ling Rothrock are also highly appreciated.

Author information

Authors and Affiliations

  1. Department of Systems Design Engineering, University of Waterloo, 200 University Ave. West, Waterloo, ON, N2L 3G1, Canada

    Dev Minotra

  2. College of Information Sciences and Technology, The Pennsylvania State University, 316 Information Sciences and Technology Building University Park, State College, PA, 16802, USA

    Michael D. McNeese

Authors
  1. Dev Minotra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael D. McNeese
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dev Minotra.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Minotra, D., McNeese, M.D. Predictive aids can lead to sustained attention decrements in the detection of non-routine critical events in event monitoring. Cogn Tech Work 19, 161–177 (2017). https://doi.org/10.1007/s10111-017-0402-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10111-017-0402-x

Keywords

Search

Navigation