Jeronimo G Grandi
Zekun Cao
ABSTRACT
Virtual reality (VR) is a powerful tool for investigating user experience in interactive systems by simulating experiences that are challenging to obtain in real life. However, it is crucial to design VR tasks appropriately, especially when used for training first responders who face high cognitive workloads in extreme conditions. Poorly designed VR tasks can strain cognitive resources and hinder human performance and training transfer. Therefore, careful attention must be given to developing VR tasks that enhance cognitive processing in high-cognitive workload scenarios. This study aims to assess the impact of different types and intensities of alerts on reaction time and comprehension of notifications, and task performance in a VR environment designed to induce various levels of cognitive load. Two experiments were conducted, including a remote experiment, where participants had to complete a task in which they identified matching figures while reacting to and interpreting alerts delivered through auditory, visual, and haptic channels. Results showed that, while task performance was not affected by alert type or intensity, audio alerts were the slowest to react to, indicating higher cognitive processing in alerts of that modality.
Supplemental Material
- Jessica S Ancker, Alison Edwards, Sarah Nosal, Diane Hauser, Elizabeth Mauer, Rainu Kaushal, and With the HITEC Investigators. 2017. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC medical informatics and decision making 17 (2017), 1–9.Google Scholar
- Carryl L. Baldwin and Bridget A. Lewis. 2014. Perceived urgency mapping across modalities within a driving context. Applied Ergonomics 45, 5 (2014), 1270–1277. https://doi.org/10.1016/j.apergo.2013.05.002Google ScholarCross Ref
- Katharina Barcatta, Elisabeth Holl, Layla Battistutta, Marian van der Meulen, and Katharina M Rischer. 2022. When less is more: Investigating factors influencing the distraction effect of virtual reality from pain. Frontiers in Pain Research 2 (2022), 800258.Google ScholarCross Ref
- Stanley J Bolanowski Jr, George A Gescheider, Ronald T Verrillo, and Christin M Checkosky. 1988. Four channels mediate the mechanical aspects of touch. The Journal of the Acoustical society of America 84, 5 (1988), 1680–1694.Google ScholarCross Ref
- John M. Carroll, Dennis C. Neale, Philip L. Isenhour, Mary Beth Rosson, and D. Scott McCrickard. 2003. Notification and awareness: Synchronizing task-oriented collaborative activity. International Journal of Human Computer Studies 58, 5 (2003), 605–632. https://doi.org/10.1016/S1071-5819(03)00024-7Google ScholarDigital Library
- Alan H.S. Chan and Annie W.Y. Ng. 2012. Finger response times to visual, auditory and tactile modality stimuli. Lecture Notes in Engineering and Computer Science 2196 (2012), 1449–1454.Google Scholar
- Kuan-Wen Chen, Yung-Ju Chang, and Liwei Chan. 2022. Predicting opportune moments to deliver notifications in virtual reality. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–18.Google ScholarDigital Library
- Kait Clark, Matthew S. Cain, and Stephen R. Mitroff. 2015. Perception and Human Information Processing in Visual Search. Cambridge University Press, 199–217. https://doi.org/10.1017/CBO9780511973017.016Google ScholarCross Ref
- N. Demeter, D. Pud, and N. Josman. 2018. Cognitive Components Predict Virtual Reality-induced Analgesia: Repeated Measures In Healthy Subjects. Front. Robot. AI 4 (2018). https://doi.org/10.3389/frobt.2017.00070Google ScholarCross Ref
- Arindam Dey, Alex Chatourn, and Mark Billinghurst. 2019. Exploration of an EEG-based cognitively adaptive training system in virtual reality. 26th IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2019 - Proceedings (2019), 220–226. https://doi.org/10.1109/VR.2019.8797840Google ScholarCross Ref
- Ajoy S Fernandes and Steven K Feiner. 2016. Combating VR sickness through subtle dynamic field-of-view modification. In Proc. 3DUI. IEEE, 201–210.Google ScholarCross Ref
- Jérémy Frey, Maxime Daniel, Julien Castet, Martin Hachet, and Fabien Lotte. 2016. Framework for electroencephalography-based evaluation of user experience. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 2283–2294.Google ScholarDigital Library
- Sarthak Ghosh, Lauren Winston, Nishant Panchal, Philippe Kimura-Thollander, Jeff Hotnog, Douglas Cheong, Gabriel Reyes, and Gregory D. Abowd. 2018. NotifiVR: Exploring Interruptions and Notifications in Virtual Reality. IEEE Transactions on Visualization and Computer Graphics 24, 4 (2018), 1447–1456. https://doi.org/10.1109/TVCG.2018.2793698Google ScholarDigital Library
- Jerônimo G Grandi, Zekun Cao, Mark Ogren, and Regis Kopper. 2020. Simulating next-generation user interfaces for law enforcement traffic stops. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE, 826–827.Google ScholarCross Ref
- Jerônimo G Grandi, Zekun Cao, Mark Ogren, and Regis Kopper. 2021. Design and Simulation of Next-Generation Augmented Reality User Interfaces in Virtual Reality. In 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW).Google Scholar
- David Hecht and Miriam Reiner. 2009. Sensory dominance in combinations of audio, visual and haptic stimuli. Experimental Brain Research 193, 2 (2009), 307–314. https://doi.org/10.1007/s00221-008-1626-zGoogle ScholarCross Ref
- Shamsi T. Iqbal and Eric Horvitz. 2010. Notifications and awareness: A field study of alert usage and preferences. Proceedings of the ACM Conference on Computer Supported Cooperative Work, CSCW (2010), 27–30. https://doi.org/10.1145/1718918.1718926Google ScholarDigital Library
- Julia M Juliano, Nicolas Schweighofer, and Sook-Lei Liew. 2022. Increased cognitive load in immersive virtual reality during visuomotor adaptation is associated with decreased long-term retention and context transfer. Journal of NeuroEngineering and Rehabilitation 19, 1 (2022), 1–14.Google ScholarCross Ref
- Alexandra D Kaplan, Jessica Cruit, Mica Endsley, Suzanne M Beers, Ben D Sawyer, and Peter A Hancock. 2021. The effects of virtual reality, augmented reality, and mixed reality as training enhancement methods: A meta-analysis. Human factors 63, 4 (2021), 706–726.Google ScholarCross Ref
- Benjamin Keinert, Matthias Innmann, Michael Sänger, and Marc Stamminger. 2015. Spherical Fibonacci Mapping. ACM Trans. Graph. 34, 6, Article 193 (oct 2015), 7 pages. https://doi.org/10.1145/2816795.2818131Google ScholarDigital Library
- Negar Khojasteh and Andrea Stevenson Won. 2021. Working together on diverse tasks: A longitudinal study on individual workload, presence and emotional recognition in collaborative virtual environments. Frontiers in Virtual Reality 2 (2021), 643331.Google ScholarCross Ref
- Hyungil Kim, Xuefang Wu, Joseph L. Gabbard, and Nicholas F. Polys. 2013. Exploring head-up augmented reality interfaces for crash warning systems. Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI 2013 (2013), 224–227. https://doi.org/10.1145/2516540.2516566Google ScholarDigital Library
- Inge M Knudson and Jennifer R Melcher. 2016. Elevated acoustic startle responses in humans: relationship to reduced loudness discomfort level, but not self-report of hyperacusis. Journal of the Association for Research in Otolaryngology 17, 3 (2016), 223–235.Google ScholarCross Ref
- Stephanie J Lackey, JN Salcedo, James L Szalma, and Peter A Hancock. 2016. The stress and workload of virtual reality training: the effects of presence, immersion and flow. Ergonomics 59, 8 (2016), 1060–1072.Google ScholarCross Ref
- Luca Longo, Christopher D Wickens, Gabriella Hancock, and Peter A Hancock. 2022. Human Mental Workload: A Survey and a Novel Inclusive Definition. Frontiers in psychology 13 (2022), 883321.Google Scholar
- Tiffany Luong, Ferran Argelaguet, Nicolas Martin, and Anatole Lécuyer. 2020. Introducing mental workload assessment for the design of virtual reality training scenarios. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 662–671.Google ScholarCross Ref
- Horia A Maior, Max L Wilson, and Sarah Sharples. 2018. Workload alerts—using physiological measures of mental workload to provide feedback during tasks. ACM Transactions on Computer-Human Interaction 25, 2 (2018).Google ScholarDigital Library
- Denys J.C. Matthies, Laura Milena Daza Parra, and Bodo Urban. 2018. Scaling notifications beyond alerts: From subtly drawing attention up to forcing the user to take action. UIST 2018 Adjunct - Adjunct Publication of the 31st Annual ACM Symposium on User Interface Software and Technology (2018), 45–47. https://doi.org/10.1145/3266037.3266096Google ScholarDigital Library
- D. Scott McCrickard, Richard Catrambone, C. M. Chewar, and John T. Stasko. 2003. Establishing tradeoffs that leverage attention for utility: Empirically evaluating information display in notification systems. Vol. 58. 547–582 pages. https://doi.org/10.1016/S1071-5819(03)00022-3Google ScholarDigital Library
- D. Scott McCrickard, C. M. Chewar, Jacob P. Somervell, and All Ndiwalana. 2003. A model for notification systems evaluation-assessing user goals for multitasking activity. ACM Transactions on Computer-Human Interaction 10, 4 (2003), 312–338. https://doi.org/10.1145/966930.966933Google ScholarDigital Library
- D. Scott McCrickard, Mary Czerwinski, and Lyn Bartram. 2003. Introduction: Design and evaluation of notification user interfaces. International Journal of Human Computer Studies 58, 5 (2003), 509–514. https://doi.org/10.1016/S1071-5819(03)00025-9Google ScholarDigital Library
- N O T Measurement. 1999. Department of defense design criteria standard fuze design safety criteria for. July (1999), 25–28.Google Scholar
- Luis Mejia-Puig and Tilanka Chandrasekera. 2023. The Presentation of Self in Virtual Reality: A Cognitive Load Study. Journal of Interior Design 48, 1 (2023), 29–46.Google ScholarCross Ref
- Mark R Mine. 1995. Virtual environment interaction techniques. UNC Chapel Hill CS Dept (1995).Google ScholarDigital Library
- Neville Moray. 2013. Mental workload: Its theory and measurement. Vol. 8. Springer Science & Business Media.Google Scholar
- Miyuki Morioka and Michael J. Griffin. 2008. Absolute thresholds for the perception of fore-and-aft, lateral, and vertical vibration at the hand, the seat, and the foot. Journal of Sound and Vibration 314, 1 (2008), 357–370. https://doi.org/10.1016/j.jsv.2007.12.036Google ScholarCross Ref
- S Pohja. 1996. Survey of studies on tactile sensing: R96: 02. Kista, Sweden, RWCP Neuro SICS Laboratory (1996), 1–13.Google Scholar
- Unnikrishnan Radhakrishnan, Konstantinos Koumaditis, and Francesco Chinello. 2021. A systematic review of immersive virtual reality for industrial skills training. Behaviour & Information Technology 40, 12 (2021), 1310–1339.Google ScholarCross Ref
- A. K. Rao, S. Chandra, and V. Dutt. 2022. Learning From Feedback: Evaluation Of Dynamic Decision-making In Virtual Reality Under Various Repetitive Training Frameworks. Front. Psychol. 13 (2022). https://doi.org/10.3389/fpsyg.2022.872061Google ScholarCross Ref
- Maximilian Rettinger, Christoph Schmaderer, and Gerhard Rigoll. 2022. Do you notice me? how bystanders affect the cognitive load in virtual reality. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 77–82.Google ScholarCross Ref
- Sadia Rubab, Lingyun Yu, Junxiu Tang, and Yingcai Wu. 2023. Exploring Effective Relationships Between Visual-Audio Channels in Data Visualization. Journal of Visualization (2023), 1–20.Google Scholar
- Rufat Rzayev, Sven Mayer, Christian Krauter, and Niels Henze. 2019. Notification in VR: The effect of notification placement, task, and environment. CHI PLAY 2019 - Proceedings of the Annual Symposium on Computer-Human Interaction in Play (2019), 199–211. https://doi.org/10.1145/3311350.3347190Google ScholarDigital Library
- Bahador Saket, Chrisnawan Prasojo, Yongfeng Huang, and Shengdong Zhao. 2013. Designing an effective vibration-based notification interface for mobile phones. In Proceedings of the 2013 conference on Computer supported cooperative work. 149–1504.Google ScholarDigital Library
- Hans Jochen Scholl, Karyn Hubbell, and Jeffrey Leonard. 2019. Information sharing and situational awareness: insights from the Cascadia rising exercise of June 2016. In 52nd Hawaii International Conference on System Sciences (HICSS), Grand Wailea, Maui, Hawaii, USA (January 8-11, 2019). 3035–3045.Google ScholarCross Ref
- Ben Shneiderman, Catherine Plaisant, Maxine S Cohen, Steven Jacobs, Niklas Elmqvist, and Nicholas Diakopoulos. 2016. Designing the user interface: strategies for effective human-computer interaction. Pearson.Google Scholar
- Alexis D Souchet, Stéphanie Philippe, Domitile Lourdeaux, and Laure Leroy. 2022. Measuring visual fatigue and cognitive load via eye tracking while learning with virtual reality head-mounted displays: A review. International Journal of Human–Computer Interaction 38, 9 (2022), 801–824.Google ScholarCross Ref
- Gabrielle Almeida De Souza, Laura Amaya Torres, Vinicius Stein Dani, David Steeven Villa, Abel Ticona Larico, Anderson MacIel, and Luciana Nedel. 2018. Evaluation of visual, auditory and vibro-tactile alerts in supervised interfaces. Proceedings - 2018 20th Symposium on Virtual and Augmented Reality, SVR 2018 (2018), 163–169. https://doi.org/10.1109/SVR.2018.00033Google ScholarCross Ref
- Anthony Steed, Ye Pan, Fiona Zisch, and William Steptoe. 2016. The impact of a self-avatar on cognitive load in immersive virtual reality. In 2016 IEEE virtual reality (VR). IEEE, 67–76.Google Scholar
- C. Stothart, A. Mitchum, and C. Yehnert. 2015. The Attentional Cost Of Receiving a Cell Phone Notification.Journal of Experimental Psychology: Human Perception and Perfor 41 (2015), 893–897. Issue 4. https://doi.org/10.1037/xhp0000100Google ScholarCross Ref
- Ewald Strasser, Astrid Weiss, Roland Buchner, and Manfred Tscheligi. 2013. Alert in the Cleanroom: Testing Alerting Modalities for a Task Guiding Interface. Conference on Human Factors in Computing Systems - Proceedings 2013-April (2013), 661–666. https://doi.org/10.1145/2468356.2468473Google ScholarDigital Library
- Zachary O. Toups and Andruid Kerne. 2007. Implicit Coordination in Firefighting Practice: Design Implications for Teaching Fire Emergency Responders. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’07). Association for Computing Machinery, New York, NY, USA, 707–716. https://doi.org/10.1145/1240624.1240734Google ScholarDigital Library
- Christoph Tremmel, Christian Herff, Tetsuya Sato, Krzysztof Rechowicz, Yusuke Yamani, and Dean J Krusienski. 2019. Estimating cognitive workload in an interactive virtual reality environment using EEG. Frontiers in human neuroscience 13 (2019), 401.Google Scholar
- Remo MA van der Heiden, Christian P Janssen, Stella F Donker, and J Leon Kenemans. 2020. The influence of cognitive load on susceptibility to audio. Acta psychologica 205 (2020), 103058.Google Scholar
- Nannan Xi, Juan Chen, Filipe Gama, Marc Riar, and Juho Hamari. 2023. The challenges of entering the metaverse: An experiment on the effect of extended reality on workload. Information Systems Frontiers 25, 2 (2023), 659–680.Google ScholarDigital Library
- André Zenner, Marco Speicher, Sören Klingner, Donald Degraen, Florian Daiber, and Antonio Krüger. 2018. Immersive notification framework: Adaptive & plausible notifications in virtual reality. Conference on Human Factors in Computing Systems - Proceedings 2018-April (2018), 1–6. https://doi.org/10.1145/3170427.3188505Google ScholarDigital Library
Index Terms
- Assessing the Impact of Alert Modalities on User Performance and Comprehension in Controlled Workload Conditions
Recommendations
Presence, workload and performance effects of synthetic environment design factors
There remains a limited understanding of factors in presence and its relation to performance. This research examined a range of synthetic environment (SE) design features (viewpoint, auditory cue type and visual background) suspected to influence ...
The challenges of entering the metaverse: An experiment on the effect of extended reality on workload
AbstractInformation technologies exist to enable us to either do things we have not done before or do familiar things more efficiently. Metaverse (i.e. extended reality: XR) enables novel forms of engrossing telepresence, but it also may make mundate ...
Evaluating User Performance, Workload, and Presence of Virtual Reality Questionnaires Using Joystick and Raycasting Selection Techniques
ICVARS '23: Proceedings of the 2023 7th International Conference on Virtual and Augmented Reality SimulationsUnderstanding users’ subjective feelings is vital for Virtual Reality (VR) research, and questionnaire is one of the common used approaches to obtain subjective feedback. Embedding questionnaires into VR systems has been shown effective in reducing the ...
Comments