skip to main content
10.1145/3656650.3656657acmotherconferencesArticle/Chapter ViewAbstractPublication PagesaviConference Proceedingsconference-collections
research-article

Optimizing Visual Complexity for Physiologically-Adaptive VR Systems: Evaluating a Multimodal Dataset using EDA, ECG and EEG Features

Published: 03 June 2024 Publication History

Abstract

Physiologically-adaptive Virtual Reality systems dynamically adjust virtual content based on users’ physiological signals to enhance interaction and achieve specific goals. However, as different users’ cognitive states may underlie multivariate physiological patterns, adaptive systems necessitate a multimodal evaluation to investigate the relationship between input physiological features and target states for efficient user modeling. Here, we investigated a multimodal dataset (EEG, ECG, and EDA) while interacting with two different adaptive systems adjusting the environmental visual complexity based on EDA. Increased visual complexity led to increased alpha power and alpha-theta ratio, reflecting increased mental fatigue and workload. At the same time, EDA exhibited distinct dynamics with increased tonic and phasic components. Integrating multimodal physiological measures for adaptation evaluation enlarges our understanding of the impact of system adaptation on users’ physiology and allows us to account for it and improve adaptive system design and optimization algorithms.

Supplemental Material

ZIP File
Github repository with data and preprocessing pipeline

References

[1]
Laurence Alison, Claudia Van Den Heuvel, Sara Waring, Nicola Power, Amy Long, Terence O’Hara, and Jonathan Crego. 2013. Immersive simulated learning environments for researching critical incidents: A knowledge synthesis of the literature and experiences of studying high-risk strategic decision making. Journal of Cognitive Engineering and decision making.
[2]
Richard W Backs, John K Lenneman, and Jamie L Sicard. 1999. The Use of Autonomic Components to Improve Cardiovascular Assessment of Mental Workload in Flight.The Int. J. of Aviation Psychology.
[3]
Mathias Benedek and Christian Kaernbach. 2010. Decomposition of skin conductance data by means of nonnegative deconvolution. Psychophysiology, 647–658.
[4]
Kyle A. Bernhardt, Dmitri Poltavski, Thomas Petros, and F. Richard Ferraro. 2019. Differentiating Active and Passive Fatigue with the use of Electroencephalography. Proc. of HFES.
[5]
Nicolas Bourdillon, Laurent Schmitt, Sasan Yazdani, Jean-Marc Vesin, and Grégoire P Millet. 2017. Minimal window duration for accurate HRV recording in athletes. Front. in neuroscience.
[6]
Guillaume Chanel, Julien Kronegg, Didier Grandjean, and Thierry Pun. 2006. Emotion assessment: Arousal evaluation using EEG’s and peripheral physiological signals. In Int. workshop on multimedia content representation. Springer.
[7]
Francesco Chiossi, Changkun Ou, Carolina Gerhardt, Felix Putze, and Sven Mayer. 2023. Designing and Evaluating an Adaptive Virtual Reality System using EEG Frequencies to Balance Internal and External Attention States. arXiv preprint arXiv:2311.10447.
[8]
Francesco Chiossi, Changkun Ou, and Sven Mayer. 2023. Exploring Physiological Correlates of Visual Complexity Adaptation: Insights from EDA, ECG, and EEG Data for Adaptation Evaluation in VR Adaptive Systems. In Proc. CHI EA. ACM.
[9]
Francesco Chiossi, Yagiz Turgut, Robin Welsch, and Sven Mayer. 2023. Adapting Visual Complexity Based on Electrodermal Activity Improves Working Memory Performance in Virtual Reality. Proc. ACM Hum.-Comput. Interact.
[10]
Francesco Chiossi, Robin Welsch, Steeven Villa, Lewis Chuang, and Sven Mayer. 2022. Virtual Reality Adaptation Using Electrodermal Activity to Support the User Experience. Big Data and Cognitive Computing.
[11]
Francesco Chiossi, Johannes Zagermann, Jakob Karolus, Nils Rodrigues, Priscilla Balestrucci, Daniel Weiskopf, Benedikt Ehinger, Tiare Feuchtner, Harald Reiterer, Lewis L. Chuang, Marc Ernst, Andreas Bulling, Sven Mayer, and Albrecht Schmidt. 2022. Adapting visualizations and Interfaces to the User. it-Information Technology.
[12]
Örjan De Manzano, Töres Theorell, László Harmat, and Fredrik Ullén. 2010. The psychophysiology of flow during piano playing.Emotion.
[13]
John Duncan and Glyn W Humphreys. 1989. Visual search and stimulus similarity.Psychological review.
[14]
Caroline Dussault, Jean-Claude Jouanin, Matthieu Philippe, and Charles-Yannick Guezennec. 2005. EEG and ECG changes during simulator operation reflect mental workload and vigilance. Aviation, space, and environmental medicine.
[15]
Stephen H. Fairclough. 2009. Fundamentals of physiological computing. Interacting with Computers.
[16]
Stephen H Fairclough and Louise Venables. 2006. Prediction of subjective states from psychophysiology: A multivariate approach. Biological psychology.
[17]
Alexandre Gramfort, Martin Luessi, Eric Larson, Denis A Engemann, Daniel Strohmeier, Christian Brodbeck, Roman Goj, Mainak Jas, Teon Brooks, and Lauri Parkkonen. 2013. MEG and EEG data analysis with MNE-Python. Front. in neuroscience.
[18]
Hayrettin Gürkök and Anton Nijholt. 2012. Brain–computer interfaces for multimodal interaction: a survey and principles. Int. J. of HCI.
[19]
Linda Hirsch, Florian Müller, Francesco Chiossi, Theodor Benga, and Andreas Martin Butz. 2023. My Heart Will Go On: Implicitly Increasing Social Connectedness by Visualizing Asynchronous Players’ Heartbeats in VR Games. Proceedings of the ACM on Human-Computer Interaction, 976–1001.
[20]
Hans-Christian Jetter, Jan-Henrik Schröder, Jan Gugenheimer, Mark Billinghurst, Christoph Anthes, Mohamed Khamis, and Tiare Feuchtner. 2021. Transitional Interfaces in Mixed and Cross-Reality: A New Frontier?. In Proc. of ISS(ISS ’21). ACM.
[21]
Lik-Hang Lee, Tristan Braud, Pengyuan Zhou, Lin Wang, Dianlei Xu, Zijun Lin, Abhishek Kumar, Carlos Bermejo, and Pan Hui. 2021. All one needs to know about metaverse: A complete survey on technological singularity, virtual ecosystem, and research agenda.
[22]
Robert Leeb, Hesam Sagha, Ricardo Chavarriaga, and José del R Millán. 2011. A hybrid brain–computer interface based on the fusion of electroencephalographic and electromyographic activities. Journal of neural engineering.
[23]
Adam Li, Jacob Feitelberg, Anand Prakash Saini, Richard Höchenberger, and Mathieu Scheltienne. 2022. MNE-ICALabel: Automatically annotating ICA components with ICLabel in Python. Journal of Open Source Software.
[24]
Christopher R Madan, Janine Bayer, Matthias Gamer, Tina B Lonsdorf, and Tobias Sommer. 2018. Visual complexity and affect: Ratings reflect more than meets the eye. Front. in psychology.
[25]
Elisa Magosso, Francesca De Crescenzio, Giulia Ricci, Sergio Piastra, and Mauro Ursino. 2019. EEG alpha power is modulated by attentional changes during cognitive tasks and virtual reality immersion. Computational intelligence and neuroscience.
[26]
Dominique Makowski, Tam Pham, Zen J. Lau, Jan C. Brammer, François Lespinasse, Hung Pham, Christopher Schölzel, and SH Chen. 2021. NeuroKit2: A Python toolbox for neurophysiological signal processing. Behav. research methods.
[27]
Matteo Marucci, Gianluca Di Flumeri, Gianluca Borghini, Nicolina Sciaraffa, Michele Scandola, Enea Francesco Pavone, Fabio Babiloni, Viviana Betti, and Pietro Aricò. 2021. The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence. Scientific Reports.
[28]
Kathryn M McMillan, Angela R Laird, Suzanne T Witt, and M Elizabeth Meyerand. 2007. Self-paced working memory: Validation of verbal variations of the n-back paradigm. Brain research.
[29]
Anna C Merzagora, Meltem Izzetoglu, Robi Polikar, Valerie Weisser, Banu Onaral, and Maria T Schultheis. 2009. Functional near-infrared spectroscopy and electroencephalography: a multimodal imaging approach. In HCI International. Springer.
[30]
John E. Muñoz, M. Cameirão, S. Bermúdez i Badia, and E. Rubio Gouveia. 2018. Closing the Loop in Exergaming - Health Benefits of Biocybernetic Adaptation in Senior Adults. In Proc. of CHI Play. ACM.
[31]
Aude Olivia, Michael L Mack, Mochan Shrestha, and Angela Peeper. 2004. Identifying the perceptual dimensions of visual complexity of scenes. In Proc. of CCC.
[32]
Mark Parent, Vsevolod Peysakhovich, Kevin Mandrick, Sébastien Tremblay, and Mickaël Causse. 2019. The diagnosticity of psychophysiological signatures: Can we disentangle mental workload from acute stress with ECG and fNIRS?Int. J. of Psychophysiology.
[33]
Corinna Peifer, André Schulz, Hartmut Schächinger, Nicola Baumann, and Conny H Antoni. 2014. The relation of flow-experience and physiological arousal under stress—can u shape it?Journal of Experimental Social Psychology.
[34]
Gert Pfurtscheller, Brendan Z Allison, Günther Bauernfeind, Clemens Brunner, Teodoro Solis Escalante, Reinhold Scherer, Thorsten O Zander, Gernot Mueller-Putz, Christa Neuper, and Niels Birbaumer. 2010. The hybrid BCI. Front. in neuroscience.
[35]
Andrea C Pierno, Andrea Caria, Scott Glover, and Umberto Castiello. 2005. Effects of increasing visual load on aurally and visually guided target acquisition in a virtual environment. Applied ergonomics.
[36]
Alan T Pope, Edward H Bogart, and Debbie S Bartolome. 1995. Biocybernetic system evaluates indices of operator engagement in automated task. Biological psychology.
[37]
Eric D Ragan, Doug A Bowman, Regis Kopper, Cheryl Stinson, Siroberto Scerbo, and Ryan P McMahan. 2015. Effects of field of view and visual complexity on virtual reality training effectiveness for a visual scanning task. IEEE TVCG.
[38]
Rahul Rajan, Ted Selker, and Ian Lane. 2016. Task Load Estimation and Mediation Using Psycho-Physiological Measures. In Proc. of IUI ’16. ACM.
[39]
Bujar Raufi and Luca Longo. 2022. An Evaluation of the EEG alpha-to-theta and theta-to-alpha band Ratios as Indexes of Mental Workload. Front. in Neuroinformatics.
[40]
M. Richter, G.H.E. Gendolla, and R.A. Wright. 2016. Chapter Five - Three Decades of Research on Motivational Intensity Theory: What We Have Learned About Effort and What We Still Don’t Know. Advances in Motivation Science.
[41]
Hartmut Schächinger, Johannes Port, Stuart Brody, Lilly Linder, Frank H Wilhelm, Peter R Huber, Daniel Cox, and Ulrich Keller. 2004. Increased high-frequency heart rate variability during insulin-induced hypoglycaemia in healthy humans. Clinical Science.
[42]
Reinhold Scherer, GR Müller-Putz, and Gert Pfurtscheller. 2007. Self-initiation of EEG-based brain–computer communication using the heart rate response. Journal of Neural Engineering.
[43]
Pradeep Shenoy, Matthias Krauledat, Benjamin Blankertz, Rajesh PN Rao, and Klaus-Robert Müller. 2006. Towards adaptive classification for BCI. Journal of neural engineering.
[44]
Yvonne Tran, Ashley Craig, Rachel Craig, Rifai Chai, and Hung Nguyen. 2020. The influence of mental fatigue on brain activity: Evidence from a systematic review with meta-analyses. Psychophysiology, e13554.
[45]
Leonard J Trejo, Karla Kubitz, Roman Rosipal, Rebekah L Kochavi, Leslie D Montgomery, 2015. EEG-based estimation and classification of mental fatigue. Psychology.
[46]
Edmund Wascher, Björn Rasch, Jessica Sänger, Sven Hoffmann, Daniel Schneider, Gerhard Rinkenauer, Herbert Heuer, and Ingmar Gutberlet. 2014. Frontal theta activity reflects distinct aspects of mental fatigue. Biological psychology.
[47]
David Zhang, Fengxi Song, Yong Xu, and Zhizhen Liang. 2009. Decision level fusion. In Advanced pattern recognition technologies with applications to biometrics. IGI Global.

Cited By

View all
  • (2025)Cognitive Assessment and Training in Extended Reality: Multimodal Systems, Clinical Utility, and Current ChallengesEncyclopedia10.3390/encyclopedia50100085:1(8)Online publication date: 13-Jan-2025
  • (2025)Designing and Evaluating an Adaptive Virtual Reality System using EEG Frequencies to Balance Internal and External Attention StatesInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103433196:COnline publication date: 1-Feb-2025
  • (2024)Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Using Fixation-Related Potentials and Eye Tracking FeaturesProceedings of the ACM on Human-Computer Interaction10.1145/36765288:MHCI(1-33)Online publication date: 24-Sep-2024
  • Show More Cited By

Index Terms

  1. Optimizing Visual Complexity for Physiologically-Adaptive VR Systems: Evaluating a Multimodal Dataset using EDA, ECG and EEG Features

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    AVI '24: Proceedings of the 2024 International Conference on Advanced Visual Interfaces
    June 2024
    578 pages
    ISBN:9798400717642
    DOI:10.1145/3656650
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 03 June 2024

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Adaptive Systems
    2. Physiological Computing
    3. Virtual Reality
    4. Visual Complexity

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Data Availability

    Github repository with data and preprocessing pipeline https://dl.acm.org/doi/10.1145/3656650.3656657#avi24-adaptation-dataset-main.zip

    Funding Sources

    • DFG

    Conference

    AVI 2024

    Acceptance Rates

    AVI '24 Paper Acceptance Rate 21 of 82 submissions, 26%;
    Overall Acceptance Rate 128 of 490 submissions, 26%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)110
    • Downloads (Last 6 weeks)18
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Cognitive Assessment and Training in Extended Reality: Multimodal Systems, Clinical Utility, and Current ChallengesEncyclopedia10.3390/encyclopedia50100085:1(8)Online publication date: 13-Jan-2025
    • (2025)Designing and Evaluating an Adaptive Virtual Reality System using EEG Frequencies to Balance Internal and External Attention StatesInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103433196:COnline publication date: 1-Feb-2025
    • (2024)Understanding the Impact of the Reality-Virtuality Continuum on Visual Search Using Fixation-Related Potentials and Eye Tracking FeaturesProceedings of the ACM on Human-Computer Interaction10.1145/36765288:MHCI(1-33)Online publication date: 24-Sep-2024
    • (2024)Multimodal Detection of External and Internal Attention in Virtual Reality using EEG and Eye Tracking FeaturesProceedings of Mensch und Computer 202410.1145/3670653.3670657(29-43)Online publication date: 1-Sep-2024
    • (2024)Searching Across Realities: Investigating ERPs and Eye-Tracking Correlates of Visual Search in Mixed RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.345617230:11(6997-7007)Online publication date: Nov-2024
    • (2024)PsyQoE: Improving Quality-of-Experience Assessment with Psychological Effects in Video StreamingIEEE Transactions on Services Computing10.1109/TSC.2024.3451215(1-14)Online publication date: 2024
    • (2024)Comparative Analysis of Teleportation and Joystick Locomotion in Virtual Reality Navigation with Different Postures: A Comprehensive Examination of Mental WorkloadInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2429891(1-12)Online publication date: 22-Nov-2024

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media