skip to main content
10.1145/3573381.3596150acmconferencesArticle/Chapter ViewAbstractPublication PagesimxConference Proceedingsconference-collections
research-article

An Integrated Framework for Understanding Multimodal Embodied Experiences in Interactive Virtual Reality

Published: 29 August 2023 Publication History

Abstract

Virtual Reality (VR) technology enables “embodied interactions” in realistic environments where users can freely move and interact, with deep physical and emotional states. However, a comprehensive understanding of the embodied user experience is currently limited by the extent to which one can make relevant observations, and the accuracy at which observations can be interpreted.
Paul Dourish proposed a way forward through the characterisation of embodied interactions in three senses: ontology, intersubjectivity, and intentionality. In a joint effort between computer and neuro-scientists, we built a framework to design studies that investigate multimodal embodied experiences in VR, and apply it to study the impact of simulated low-vision on user navigation. Our methodology involves the design of 3D scenarios annotated with an ontology, modelling intersubjective tasks, and correlating multimodal metrics such as gaze and physiology to derive intentions. We show how this framework enables a more fine-grained understanding of embodied interactions in behavioural research.

Supplementary Material

Presentation video with subtitle file (video.zip)

References

[1]
Orientada al Ser Humano. 2009. A Neuroscience Approach to Virtual Reality Experience Using Transcranial Doppler Monitoring. (2009).
[2]
Mariano Alcañiz, Beatriz Rey, Jose Tembl, and Vera Parkhutik. 2009. A neuroscience approach to virtual reality experience using transcranial Doppler monitoring. Presence: Teleoperators and Virtual Environments 18, 2 (2009), 97–111.
[3]
Sahar Aseeri and Victoria Interrante. 2021. The Influence of Avatar Representation on Interpersonal Communication in Virtual Social Environments. IEEE Transactions on Visualization and Computer Graphics 27, 5 (May 2021), 2608–2617. https://doi.org/10.1109/TVCG.2021.3067783 Conference Name: IEEE Transactions on Visualization and Computer Graphics.
[4]
Domna Banakou, Parasuram D Hanumanthu, and Mel Slater. 2016. Virtual embodiment of white people in a black virtual body leads to a sustained reduction in their implicit racial bias. Frontiers in human neuroscience 10 (2016), 601.
[5]
Stuart M. Bender and Billy Sung. 2021. Fright, attention, and joy while killing zombies in Virtual Reality: A psychophysiological analysis of VR user experience. Psychology & Marketing 38, 6 (2021), 937–947. https://doi.org/10.1002/mar.21444 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/mar.21444.
[6]
Mette Boldt, Michael Bonfert, Inga Lehne, Melina Cahnbley, Kim Korschinq, Loannis Bikas, Stefan Finke, Martin Hanci, Valentin Kraft, Boxuan Liu, Tram Nguyen, Alina Panova, Ramneek Singh, Alexander Steenbergen, Rainer Malaka, and Jan Smeddinck. 2018. You Shall Not Pass: Non-Intrusive Feedback for Virtual Walls in VR Environments with Room-Scale Mapping. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, Tuebingen/Reutlingen, Germany, 143–150. https://doi.org/10.1109/VR.2018.8446177
[7]
Rozenn Bouville, Valerie Gouranton, Thomas Boggini, Florian Nouviale, and Bruno Arnaldi. 2015. #FIVE : High-level components for developing collaborative and interactive virtual environments. In 2015 IEEE 8th Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS). IEEE, Arles, France, 33–40. https://doi.org/10.1109/SEARIS.2015.7854099
[8]
Lung-Pan Cheng, Eyal Ofek, Christian Holz, and Andrew D. Wilson. 2019. VRoamer: Generating On-The-Fly VR Experiences While Walking inside Large, Unknown Real-World Building Environments. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, Osaka, Japan, 359–366. https://doi.org/10.1109/VR.2019.8798074
[9]
Rory M.S. Clifford, Sungchul Jung, Simon Hoermann, Mark Billinghurst, and Robert W. Lindeman. 2019. Creating a Stressful Decision Making Environment for Aerial Firefighter Training in Virtual Reality. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, Osaka, Japan, 181–189. https://doi.org/10.1109/VR.2019.8797889 ISSN: 2642-5254.
[10]
Rossana De Beni, Francesca Pazzaglia, and Simona Gardini. 2006. The role of mental rotation and age in spatial perspective-taking tasks: when age does not impair perspective-taking performance. Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition 20, 6 (2006), 807–821.
[11]
Nonny De la Peña, Peggy Weil, Joan Llobera, Bernhard Spanlang, Doron Friedman, Maria V Sanchez-Vives, and Mel Slater. 2010. Immersive journalism: Immersive virtual reality for the first-person experience of news. Presence 19, 4 (2010), 291–301.
[12]
Alexandre Delaux, Jean-Baptiste de Saint Aubert, Stephen Ramanoël, Marcia Bécu, Lukas Gehrke, Marius Klug, Ricardo Chavarriaga, José-Alain Sahel, Klaus Gramann, and Angelo Arleo. 2021. Mobile brain/body imaging of landmark-based navigation with high-density EEG. European Journal of Neuroscience 54, 12 (2021), 8256–8282.
[13]
Diane Dewez, Ludovic Hoyet, Anatole Lécuyer, and Ferran Argelaguet Sanz. 2021. Towards “Avatar-Friendly” 3D Manipulation Techniques: Bridging the Gap Between Sense of Embodiment and Interaction in Virtual Reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems(CHI ’21). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3411764.3445379
[14]
Patrick Dickinson, Andrew Cardwell, Adrian Parke, Kathrin Gerling, and John Murray. 2021. Diegetic Tool Management in a Virtual Reality Training Simulation. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). IEEE, Lisboa, Portugal, 131–139. https://doi.org/10.1109/VR50410.2021.00034 ISSN: 2642-5254.
[15]
Paul Dourish. 2004. Where the action is: the foundations of embodied interaction. MIT press, USA.
[16]
Pawel Grabarczyk and Marek Pokropski. 2016. Perception of affordances and experience of presence in virtual reality. AVANT. Trends in Interdisciplinary Studies (2016).
[17]
Quentin Guimard, Florent Robert, Camille Bauce, Aldric Ducreux, Lucile Sassatelli, Hui-Yin Wu, Marco Winckler, and Auriane Gros. 2022. PEM360: A dataset of 360 videos with continuous Physiological measurements, subjective Emotional ratings and Motion traces. In Proceedings of the 13th ACM Multimedia Systems Conference. 252–258.
[18]
Joshua Peter Handali, Johannes Schneider, Michael Gau, Valentin Holzwarth, and Jan vom Brocke. 2021. Visual Complexity and Scene Recognition: How Low Can You Go?. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). 286–295. https://doi.org/10.1109/VR50410.2021.00051 ISSN: 2642-5254.
[19]
Dan Hawes and Ali Arya. 2021. VR-based Student Priming to Reduce Anxiety and Increase Cognitive Bandwidth. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). 245–254. https://doi.org/10.1109/VR50410.2021.00046 ISSN: 2642-5254.
[20]
Meggy Hayotte, Pierre Thérouanne, Laura Gray, Karine Corrion, and Fabienne d’Arripe Longueville. 2020. The French eHealth acceptability scale using the unified theory of acceptance and use of technology 2 model: instrument validation study. Journal of medical Internet research 22, 4 (2020), e16520.
[21]
Dongjin Huang, Xianglong Wang, Jinhua Liu, Jinyao Li, and Wen Tang. 2022. Virtual reality safety training using deep EEG-net and physiology data. The visual computer 38, 4 (2022), 1195–1207.
[22]
Crescent Jicol, Chun Hin Wan, Benjamin Doling, Caitlin H Illingworth, Jinha Yoon, Charlotte Headey, Christof Lutteroth, Michael J Proulx, Karin Petrini, and Eamonn O’Neill. 2021. Effects of Emotion and Agency on Presence in Virtual Reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–13. https://doi.org/10.1145/3411764.3445588
[23]
Conor Keighrey, Ronan Flynn, Siobhan Murray, Sean Brennan, and Niall Murray. 2017. Comparing User QoE via Physiological and Interaction Measurements of Immersive AR and VR Speech and Language Therapy Applications. In Proceedings of the on Thematic Workshops of ACM Multimedia 2017 - Thematic Workshops ’17. ACM Press, Mountain View, California, USA, 485–492. https://doi.org/10.1145/3126686.3126747
[24]
Robert S Kennedy, Norman E Lane, Kevin S Berbaum, and Michael G Lilienthal. 1993. Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness. The international journal of aviation psychology 3, 3 (1993), 203–220.
[25]
Ue-Hwan Kim, Jin-Man Park, Taek-jin Song, and Jong-Hwan Kim. 2020. 3-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agents. IEEE Transactions on Cybernetics 50, 12 (2020), 4921–4933. https://doi.org/10.1109/TCYB.2019.2931042
[26]
Meghan Kumar, Connor Delaney, and Dean Krusienski. 2022. Estimation of Affective States in Virtual Reality Environments using EEG. In Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments. 396–401.
[27]
Peter Lang. 1980. Behavioral treatment and bio-behavioral assessment: Computer applications. Technology in mental health care delivery systems (1980), 119–137.
[28]
Yining Lang, Liang Wei, Fang Xu, Yibiao Zhao, and Lap-Fai Yu. 2018. Synthesizing Personalized Training Programs for Improving Driving Habits via Virtual Reality. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, Reutlingen, 297–304. https://doi.org/10.1109/VR.2018.8448290
[29]
Sarah Lopez, Yi Yang, Kevin Beltran, Soo Jung Kim, Jennifer Cruz Hernandez, Chelsy Simran, Bingkun Yang, and Beste F Yuksel. 2019. Investigating implicit gender bias and embodiment of white males in virtual reality with full body visuomotor synchrony. In Proceedings of the 2019 CHI Conference on human factors in computing systems. 1–12.
[30]
Dominique Makowski, Tam Pham, Zen J. Lau, Jan C. Brammer, François Lespinasse, Hung Pham, Christopher Schölzel, and S. H. Annabel Chen. 2021. NeuroKit2: A Python toolbox for neurophysiological signal processing. Behavior Research Methods 53, 4 (feb 2021), 1689–1696. https://doi.org/10.3758/s13428-020-01516-y
[31]
Marieke AG Martens, Angus Antley, Daniel Freeman, Mel Slater, Paul J Harrison, and Elizabeth M Tunbridge. 2019. It feels real: physiological responses to a stressful virtual reality environment and its impact on working memory. Journal of Psychopharmacology 33, 10 (2019), 1264–1273.
[32]
Maryam Memar and Amin Mokaribolhassan. 2021. Stress level classification using statistical analysis of skin conductance signal while driving. SN Applied Sciences 3, 1 (Jan. 2021), 64. https://doi.org/10.1007/s42452-020-04134-7
[33]
Deniz Mevlevioğlu, David Murphy, and Sabin Tabirca. 2021. Real-time Anxiety Prediction in Virtual Reality Exposure Therapy. New York City (2021), 5.
[34]
Christos Mousas, Dominic Kao, Alexandros Koilias, and Banafsheh Rekabdar. 2020. Real and virtual environment mismatching induces arousal and alters movement behavior. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 626–635.
[35]
T. C. Peck, M. Doan, K. A. Bourne, and J. J. Good. 2018. The Effect of Gender Body-Swap Illusions on Working Memory and Stereotype Threat. IEEE Transactions on Visualization & Computer Graphics 24, 04 (apr 2018), 1604–1612. https://doi.org/10.1109/TVCG.2018.2793598
[36]
Mary Pidgeon, Nadia Kanwal, and Niall Murray. 2022. Federated Learning to Understand Human Emotions via Smart Clothing: Research Proposal. In Proceedings of the 13th ACM Multimedia Systems Conference (Athlone, Ireland) (MMSys ’22). Association for Computing Machinery, New York, NY, USA, 408–411. https://doi.org/10.1145/3524273.3533936
[37]
Aylen Ricca, Amine Chellali, and Samir Otrnane. 2021. The influence of hand visualization in tool-based motor-skills training, a longitudinal study. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). 103–112. https://doi.org/10.1109/VR50410.2021.00031 ISSN: 2642-5254.
[38]
Stephanie Schulze, Toni Pence, Ned Irvine, and Curry Guinn. 2019. The effects of embodiment in virtual reality on implicit gender bias. In Virtual, Augmented and Mixed Reality. Multimodal Interaction: 11th International Conference, VAMR 2019, Held as Part of the 21st HCI International Conference, HCII 2019, Orlando, FL, USA, July 26–31, 2019, Proceedings, Part I 21. Springer, 361–374.
[39]
Sofia Seinfeld, Tiare Feuchtner, Johannes Pinzek, and Jörg Müller. 2020. Impact of Information Placement and User Representations in VR on Performance and Embodiment. Vol. PP. https://doi.org/10.1109/TVCG.2020.3021342 Journal Abbreviation: IEEE transactions on visualization and computer graphics Publication Title: IEEE transactions on visualization and computer graphics.
[40]
Pavel Ugwitz, Alžběta Šašinková, Čeněk Šašinka, Zdeněk Stachoň, and Vojtěch Juřík. 2021. Toggle toolkit: A tool for conducting experiments in unity virtual environments. Behavior research methods (2021), 1–11.
[41]
Viswanath Venkatesh, James YL Thong, and Xin Xu. 2012. Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology. MIS quarterly (2012), 157–178.
[42]
Jhon Bueno Vesga, Xinhao Xu, and Hao He. 2021. The Effects of Cognitive Load on Engagement in a Virtual Reality Learning Environment. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). 645–652. https://doi.org/10.1109/VR50410.2021.00090 ISSN: 2642-5254.
[43]
Sophie Villenave, Jonathan Cabezas, Patrick Baert, Florent Dupont, and Guillaume Lavoué. 2022. XREcho: A Unity plug-in to record and visualize user behavior during XR sessions. In Proceedings of the 13th ACM Multimedia Systems Conference. 341–346.
[44]
Bob G Witmer and Michael J Singer. 1998. Measuring presence in virtual environments: A presence questionnaire. Presence 7, 3 (1998), 225–240.
[45]
Erwin Wu, Mitski Piekenbrock, Takuto Nakumura, and Hideki Koike. 2021. SPinPong - Virtual Reality Table Tennis Skill Acquisition using Visual, Haptic and Temporal Cues. IEEE Transactions on Visualization and Computer Graphics 27, 5 (May 2021), 2566–2576. https://doi.org/10.1109/TVCG.2021.3067761 Conference Name: IEEE Transactions on Visualization and Computer Graphics.
[46]
Hui-Yin Wu, Florent Robert, Théo Fafet, Brice Graulier, Barthélemy Passin-Cauneau, Lucile Sassatelli, and Marco Winckler. 2022. Designing Guided User Tasks in VR Embodied Experiences. Proceedings of the ACM on Human-Computer Interaction 6, EICS (2022), 1–24.
[47]
Tong Xue, Abdallah El Ali, Tianyi Zhang, Gangyi Ding, and Pablo Cesar. 2021. CEAP-360VR: A Continuous Physiological and Behavioral Emotion Annotation Dataset for 360 VR Videos. IEEE Transactions on Multimedia (2021), 1–1. https://doi.org/10.1109/TMM.2021.3124080
[48]
Tong Xue, Abdallah El Ali, Tianyi Zhang, Gangyi Ding, and Pablo Cesar. 2021. CEAP-360VR: A Continuous Physiological and Behavioral Emotion Annotation Dataset for 360 VR Videos. IEEE Transactions on Multimedia (2021), 1–1. https://doi.org/10.1109/TMM.2021.3124080

Cited By

View all
  • (2024)AMD Journee: A Patient Co-designed VR Experience to Raise Awareness Towards the Impact of AMD on Social InteractionsProceedings of the 2024 ACM International Conference on Interactive Media Experiences10.1145/3639701.3656314(17-29)Online publication date: 7-Jun-2024
  • (2024)Task-based methodology to characterise immersive user experience with multivariate data2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00092(722-731)Online publication date: 16-Mar-2024
  • (2024)PLUME: Record, Replay, Analyze and Share User Behavior in 6DoF XR ExperiencesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337210730:5(2087-2097)Online publication date: 4-Mar-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
IMX '23: Proceedings of the 2023 ACM International Conference on Interactive Media Experiences
June 2023
465 pages
ISBN:9798400700286
DOI:10.1145/3573381
Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 August 2023

Permissions

Request permissions for this article.

Check for updates

Badges

  • Best Paper

Author Tags

  1. 3D environments
  2. Embodied experiences
  3. immersion
  4. interaction
  5. navigation
  6. scene ontology
  7. task modeling
  8. user experience analysis

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • UCA JEDI Investissements d?Avenir
  • ANR

Conference

IMX '23

Acceptance Rates

Overall Acceptance Rate 69 of 245 submissions, 28%

Upcoming Conference

IMX '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)156
  • Downloads (Last 6 weeks)15
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)AMD Journee: A Patient Co-designed VR Experience to Raise Awareness Towards the Impact of AMD on Social InteractionsProceedings of the 2024 ACM International Conference on Interactive Media Experiences10.1145/3639701.3656314(17-29)Online publication date: 7-Jun-2024
  • (2024)Task-based methodology to characterise immersive user experience with multivariate data2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00092(722-731)Online publication date: 16-Mar-2024
  • (2024)PLUME: Record, Replay, Analyze and Share User Behavior in 6DoF XR ExperiencesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337210730:5(2087-2097)Online publication date: 4-Mar-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media