skip to main content
10.1145/3526113.3545682acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article
Public Access

Integrating Real-World Distractions into Virtual Reality

Published: 28 October 2022 Publication History

Abstract

With the proliferation of consumer-level virtual reality (VR) devices, users started experiencing VR in less controlled environments, such as in social gatherings and public areas. While the current VR hardware provides an increasingly immersive experience, it ignores stimuli originating from the physical surroundings that distract users from the VR experience. To block distractions from the outside world, many users wear noise-canceling headphones. However, this is insufficient to block loud or transient sounds (e.g., drilling or hammering) and, especially, multi-modal distractions (e.g., air drafts, temperature shifts from an A/C, construction vibrations, or food smells). To tackle this, we explore a new concept, where we directly integrate the distracting stimuli from the user's physical surroundings into their virtual reality experience to enhance presence. Using our approach, an otherwise distracting wind gust can be directly mapped to the sway of trees in a VR experience that already contains trees. Using our novel approach, we demonstrate how to integrate a range of distractive stimuli into the VR experience, such as haptics (temperature, vibrations, touch), sounds, and smells. To validate our approach, we conducted three user studies and a technical evaluation. First, to validate our key principle, we conducted a controlled study where participants were exposed to distractions while playing a VR game. We found that our approach improved users’ sense of presence, compared to wearing noise-canceling headphones. From these results, we engineered a sensing module that detects a set of simple distractive signals (e.g., sounds, winds, and temperature shifts). We validated our hardware in a technical evaluation and in an out-of-lab study where participants played VR games in an uncontrolled environment. Moreover, to gather the perspective of VR content creators that might one day utilize a system inspired by our findings, we invited game designers to use our approach and collected their feedback and VR designs. Finally, we present design considerations for mapping distracting external stimuli and discuss ethical considerations of integrating real-world stimuli into virtual reality.

References

[1]
[1] Christopher C. Berger, Mar Gonzalez-Franco, Eyal Ofek, and Ken Hinckley. 2018. The uncanny valley of haptics. Sci. Robot. 3, 17 (April 2018), eaar7010.
[2]
[2] John M. Carroll, Robert L. Mack, and Wendy A. Kellogg. 1988. Chapter 3 - Interface Metaphors and User Interface Design. In Handbook of Human-Computer Interaction, MARTIN Helander (ed.). North-Holland, Amsterdam, 67–85.
[3]
[3] Lung-Pan Cheng, Eyal Ofek, Christian Holz, Hrvoje Benko, and Andrew D. Wilson. 2017. Sparse Haptic Proxy: Touch Feedback in Virtual Environments Using a General Passive Prop. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, ACM, Denver Colorado USA, 3718–3728.
[4]
[4] Lung-Pan Cheng, Eyal Ofek, Christian Holz, and Andrew D. Wilson. 2019. VRoamer: Generating On-The-Fly VR Experiences While Walking inside Large, Unknown Real-World Building Environments. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), IEEE, Osaka, Japan, 359–366.
[5]
[5] Isamu Endo, Kazuki Takashima, Maakito Inoue, Kazuyuki Fujita, Kiyoshi Kiyokawa, and Yoshifumi Kitamura. 2021. A Reconfigurable Mobile Head-Mounted Display Supporting Real World Interactions. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–7. Retrieved March 30, 2022 from https://doi.org/10.1145/3411763.3451765
[6]
[6] Isamu Endo, Kazuki Takashima, Maakito Inoue, Kazuyuki Fujita, Kiyoshi Kiyokawa, and Yoshifumi Kitamura. 2021. ModularHMD: A Reconfigurable Mobile Head-Mounted Display Enabling Ad-hoc Peripheral Interactions with the Real World. In The 34th Annual ACM Symposium on User Interface Software and Technology, ACM, Virtual Event USA, 100–117.
[7]
[7] Maia Garau, Doron Friedman, Hila Ritter Widenfeld, Angus Antley, Andrea Brogni, and Mel Slater. 2008. Temporal and Spatial Variations in Presence: Qualitative Analysis of Interviews from an Experiment on Breaks in Presence. Presence: Teleoperators and Virtual Environments 17, 3 (June 2008), 293–309.
[8]
[8] Ceenu George, Philipp Janssen, David Heuss, and Florian Alt. 2019. Should I Interrupt or Not? Understanding Interruptions in Head-Mounted Display Settings. In Proceedings of the 2019 on Designing Interactive Systems Conference (DIS ’19), Association for Computing Machinery, New York, NY, USA, 497–510.
[9]
[9] Sarthak Ghosh, Lauren Winston, Nishant Panchal, Philippe Kimura-Thollander, Jeff Hotnog, Douglas Cheong, Gabriel Reyes, and Gregory D. Abowd. 2018. NotifiVR: Exploring Interruptions and Notifications in Virtual Reality. IEEE Trans. Visual. Comput. Graphics 24, 4 (April 2018), 1447–1456.
[10]
[10] Matt Gottsacker, Nahal Norouzi, Kangsoo Kim, Gerd Bruder, and Greg Welch. 2021. Diegetic Representations for Seamless Cross-Reality Interruptions. In 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE, Bari, Italy, 310–319.
[11]
[11] Yizheng Gu, Chun Yu, Zhipeng Li, Weiqi Li, Shuchang Xu, Xiaoying Wei, and Yuanchun Shi. 2019. Accurate and Low-Latency Sensing of Touch Contact on Any Surface with Finger-Worn IMU Sensor. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, ACM, New Orleans LA USA, 1059–1070.
[12]
[12] Daniel Harley, Alexander Verni, Mackenzie Willis, Ashley Ng, Lucas Bozzo, and Ali Mazalek. 2018. Sensory VR: Smelling, Touching, and Eating Virtual Reality. In Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction, ACM, Stockholm Sweden, 386–397.
[13]
[13] Jeremy Hartmann, Christian Holz, Eyal Ofek, and Andrew D. Wilson. 2019. RealityCheck: Blending Virtual Environments with Situated Physical Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, ACM, Glasgow Scotland Uk, 1–12.
[14]
[14] Carrie Heeter. 1992. Being There: The Subjective Experience of Presence. Presence: Teleoperators & Virtual Environments 1, 2 (January 1992), 262–271.
[15]
[15] K. Hinckley, R. Pausch, J. Goble, and N. Kassell. 1994. Passive real-world interface props for neurosurgical visualization. In Proceedings of the SIGCHI conference on Human factors in computing systems, 452–458.
[16]
[16] H.G. Hoffman. 1998. Physically touching virtual objects using tactile augmentation enhances the realism of virtual environments. In Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No.98CB36180), 59–63.
[17]
[17] Ching-Yu Hsieh, Yi-Shyuan Chiang, Hung-Yu Chiu, and Yung-Ju Chang. 2020. Bridging the Virtual and Real Worlds: A Preliminary Study of Messaging Notifications in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–14. Retrieved March 29, 2022 from https://doi.org/10.1145/3313831.3376228
[18]
[18] Gierad Laput, Karan Ahuja, Mayank Goel, and Chris Harrison. 2018. Ubicoustics: Plug-and-Play Acoustic Activity Recognition. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, ACM, Berlin Germany, 213–224.
[19]
[19] Bernhard Lehner, Gerhard Widmer, and Sebastian Bock. 2015. A low-latency, real-time-capable singing voice detection method with LSTM recurrent neural networks. In 2015 23rd European Signal Processing Conference (EUSIPCO), 21–25.
[20]
[20] Huimin Liu, Zhiquan Wang, Angshuman Mazumdar, and Christos Mousas. 2021. Virtual reality game level layout design for real environment constraints. Graphics and Visual Computing 4, (June 2021), 200020.
[21]
[21] Sebastian Marwecki and Patrick Baudisch. 2018. Scenograph: Fitting Real-Walking VR Experiences into Various Tracking Volumes. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, ACM, Berlin Germany, 511–520.
[22]
[22] Brett R. C. Molesworth, Marion Burgess, and Daniel Kwon. 2013. The use of noise cancelling headphones to improve concurrent task performance in a noisy environment. Applied Acoustics 74, 1 (January 2013), 110–115.
[23]
[23] Yves Moser and Martin A. M. Gijs. 2007. Miniaturized Flexible Temperature Sensor. Journal of Microelectromechanical Systems 16, 6 (December 2007), 1349–1354.
[24]
[24] Adiyan Mujibiya, Xiang Cao, Desney S. Tan, Dan Morris, Shwetak N. Patel, and Jun Rekimoto. 2013. The sound of touch: on-body touch and gesture sensing based on transdermal ultrasound propagation. In Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces, ACM, St. Andrews Scotland, United Kingdom, 189–198.
[25]
[25] Donald A. Norman. 1999. Affordance, conventions, and design. interactions 6, 3 (May 1999), 38–43.
[26]
[26] Catherine Oh, Fernanda Herrera, and Jeremy Bailenson. 2019. The Effects of Immersion and Real-World Distractions on Virtual Social Interactions. Cyberpsychology, Behavior, and Social Networking 22, 6 (June 2019), 365–372.
[27]
[27] Joseph O'Hagan, Julie R. Williamson, and Mohamed Khamis. 2020. Bystander interruption of VR users. In Proceedings of the 9TH ACM International Symposium on Pervasive Displays, ACM, Manchester United Kingdom, 19–27.
[28]
[28] Sigeru Omatu and Mitsuaki Yano. 2016. E-nose system by using neural networks. Neurocomputing 172, (January 2016), 394–398.
[29]
[29] Craig Phillips. 2018. Biomimicry in UX. Medium. Retrieved April 5, 2022 from https://medium.com/@craig5446/biomimicry-in-ux-design-249cc6d8649
[30]
[30] Mirco Rossi, Sebastian Feese, Oliver Amft, Nils Braune, Sandro Martis, and Gerhard Tröster. 2013. AmbientSense: A real-time ambient sound recognition system for smartphones. In 2013 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops), 230–235.
[31]
[31] Rufat Rzayev, Sven Mayer, Christian Krauter, and Niels Henze. 2019. Notification in VR: The Effect of Notification Placement, Task and Environment. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (CHI PLAY ’19), Association for Computing Machinery, New York, NY, USA, 199–211.
[32]
[32] Munehiko Sato, Ivan Poupyrev, and Chris Harrison. 2012. Touché: enhancing touch interaction on humans, screens, liquids, and everyday objects. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, Austin Texas USA, 483–492.
[33]
[33] Lior Shapira and Daniel Freedman. 2016. Reality Skins: Creating Immersive and Tactile Virtual Environments. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE, Merida, Yucatan, Mexico, 115–124.
[34]
[34] Adalberto L. Simeone, Eduardo Velloso, and Hans Gellersen. 2015. Substitutional Reality: Using the Physical Environment to Design Virtual Reality Experiences. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, ACM, Seoul Republic of Korea, 3307–3316.
[35]
[35] Mel Slater. A note on presence terminology. Presence connect 3, 3, 1–5.
[36]
[36] Mel Slater, Andrea Brogni, and Anthony Steed. 2003. Physiological Responses to Breaks in Presence: A Pilot Study. In In The 6th Annual International Workshop on Presence, 3–23.
[37]
[37] Mel Slater and Anthony Steed. 2000. A Virtual Presence Counter. Presence: Teleoperators & Virtual Environments 9, 5 (October 2000), 413–434.
[38]
[38] Mel Slater and Sylvia Wilbur. 1997. A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence: Teleoperators & Virtual Environments 6, 6 (December 1997), 603–616.
[39]
[39] Misha Sra, Sergio Garrido-Jurado, Chris Schmandt, and Pattie Maes. 2016. Procedurally generated virtual reality from 3D reconstructed physical space. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology, ACM, Munich Germany, 191–200.
[40]
[40] Li Su, Hailu Wang, Zhen Tian, Haojie Wang, Qian Cheng, and Wei Yu. 2019. Low Detection Limit and High Sensitivity Wind Speed Sensor Based on Triboelectrification-Induced Electroluminescence. Advanced Science 6, 23 (2019), 1901980.
[41]
[41] Qi Sun, Li-Yi Wei, and Arie Kaufman. 2016. Mapping virtual and physical reality. ACM Trans. Graph. 35, 4 (July 2016), 1–12.
[42]
[42] Kien T.P. Tran, Sungchul Jung, Simon Hoermann, and Robert W. Lindeman. 2019. MDI: A Multi-channel Dynamic Immersion Headset for Seamless Switching between Virtual and Real World Activities. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 350–358.
[43]
[43] Martin Usoh, Ernest Catena, Sima Arman, and Mel Slater. 2000. Using Presence Questionnaires in Reality. Presence: Teleoperators & Virtual Environments 9, 5 (October 2000), 497–503.
[44]
[44] Eva L. Waterworth and John A. Waterworth. 2001. Focus, Locus, and Sensus: The Three Dimensions of Virtual Experience. CyberPsychology & Behavior 4, 2 (April 2001), 203–213.
[45]
[45] Danli Wu, Yu Cheng, Dehan Luo, Kin-Yeung Wong, Kevin Hung, and Zhijing Yang. 2019. POP-CNN: Predicting Odor's Pleasantness with Convolutional Neural Network. arXiv:1903.07821 [cs, stat] (March 2019). Retrieved April 1, 2022 from http://arxiv.org/abs/1903.07821
[46]
[46] Jackie (Junrui) Yang, Christian Holz, Eyal Ofek, and Andrew D. Wilson. 2019. DreamWalker: Substituting Real-World Walking Experiences with a Virtual Reality. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, ACM, New Orleans LA USA, 1093–1107.
[47]
[47] What is Skeuomorphism? - Definition from Techopedia. Techopedia.com. Retrieved April 5, 2022 from http://www.techopedia.com/definition/28955/skeuomorphism
[48]
[48] Towards Balancing VR Immersion and Bystander Awareness | Proceedings of the ACM on Human-Computer Interaction. Retrieved from https://dl.acm.org/doi/10.1145/3486950

Cited By

View all
  • (2024)Understanding Interaction and Breakouts of Safety Boundaries in Virtual Reality Through Mixed-Method Studies2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00069(482-492)Online publication date: 16-Mar-2024
  • (2024)RedirectedDoors+: Door-Opening Redirection with Dynamic Haptics in Room-Scale VRIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337210530:5(2276-2286)Online publication date: 11-Mar-2024
  • (2024)PetPresence: Investigating the Integration of Real-World Pet Activities in Virtual RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337209530:5(2559-2569)Online publication date: May-2024
  • Show More Cited By

Index Terms

  1. Integrating Real-World Distractions into Virtual Reality

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    UIST '22: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology
    October 2022
    1363 pages
    ISBN:9781450393201
    DOI:10.1145/3526113
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 28 October 2022

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Distractions
    2. Haptics
    3. Presence
    4. Virtual Reality

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Funding Sources

    Conference

    UIST '22

    Acceptance Rates

    Overall Acceptance Rate 561 of 2,567 submissions, 22%

    Upcoming Conference

    UIST '25
    The 38th Annual ACM Symposium on User Interface Software and Technology
    September 28 - October 1, 2025
    Busan , Republic of Korea

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)906
    • Downloads (Last 6 weeks)93
    Reflects downloads up to 17 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Understanding Interaction and Breakouts of Safety Boundaries in Virtual Reality Through Mixed-Method Studies2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00069(482-492)Online publication date: 16-Mar-2024
    • (2024)RedirectedDoors+: Door-Opening Redirection with Dynamic Haptics in Room-Scale VRIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337210530:5(2276-2286)Online publication date: 11-Mar-2024
    • (2024)PetPresence: Investigating the Integration of Real-World Pet Activities in Virtual RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337209530:5(2559-2569)Online publication date: May-2024
    • (2024)Hold Tight: Identifying Behavioral Patterns During Prolonged Work in VR Through Video AnalysisIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337204830:5(2796-2806)Online publication date: 5-Mar-2024
    • (2024)A novel quantitative assessment of engagement in virtual realityComputers & Education10.1016/j.compedu.2023.104959209:COnline publication date: 4-Mar-2024
    • (2024)Swarm manipulation: An efficient and accurate technique for multi-object manipulation in virtual realityComputers & Graphics10.1016/j.cag.2024.104113125(104113)Online publication date: Dec-2024
    • (2023)Location-Aware Virtual Reality for Situational Awareness On the RoadProceedings of the 2023 ACM Symposium on Spatial User Interaction10.1145/3607822.3614530(1-12)Online publication date: 13-Oct-2023
    • (2023)Reality Anchors: Bringing Cues from Reality to Increase Acceptance of Immersive Technologies in TransitProceedings of the ACM on Human-Computer Interaction10.1145/36042667:MHCI(1-28)Online publication date: 13-Sep-2023
    • (2023)Neighbor-Environment Observer: An Intelligent Agent for Immersive Working CompanionshipProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606728(1-14)Online publication date: 29-Oct-2023
    • (2023)Extended VR: Exploring the Integration of VR Experiences and Real-world EngagementCompanion Publication of the 2023 ACM Designing Interactive Systems Conference10.1145/3563703.3596636(184-187)Online publication date: 10-Jul-2023
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Login options

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media