ABSTRACT
With the proliferation of consumer-level virtual reality (VR) devices, users started experiencing VR in less controlled environments, such as in social gatherings and public areas. While the current VR hardware provides an increasingly immersive experience, it ignores stimuli originating from the physical surroundings that distract users from the VR experience. To block distractions from the outside world, many users wear noise-canceling headphones. However, this is insufficient to block loud or transient sounds (e.g., drilling or hammering) and, especially, multi-modal distractions (e.g., air drafts, temperature shifts from an A/C, construction vibrations, or food smells). To tackle this, we explore a new concept, where we directly integrate the distracting stimuli from the user's physical surroundings into their virtual reality experience to enhance presence. Using our approach, an otherwise distracting wind gust can be directly mapped to the sway of trees in a VR experience that already contains trees. Using our novel approach, we demonstrate how to integrate a range of distractive stimuli into the VR experience, such as haptics (temperature, vibrations, touch), sounds, and smells. To validate our approach, we conducted three user studies and a technical evaluation. First, to validate our key principle, we conducted a controlled study where participants were exposed to distractions while playing a VR game. We found that our approach improved users’ sense of presence, compared to wearing noise-canceling headphones. From these results, we engineered a sensing module that detects a set of simple distractive signals (e.g., sounds, winds, and temperature shifts). We validated our hardware in a technical evaluation and in an out-of-lab study where participants played VR games in an uncontrolled environment. Moreover, to gather the perspective of VR content creators that might one day utilize a system inspired by our findings, we invited game designers to use our approach and collected their feedback and VR designs. Finally, we present design considerations for mapping distracting external stimuli and discuss ethical considerations of integrating real-world stimuli into virtual reality.
- [1] Christopher C. Berger, Mar Gonzalez-Franco, Eyal Ofek, and Ken Hinckley. 2018. The uncanny valley of haptics. Sci. Robot. 3, 17 (April 2018), eaar7010. DOI:https://doi.org/10.1126/scirobotics.aar7010Google Scholar
- [2] John M. Carroll, Robert L. Mack, and Wendy A. Kellogg. 1988. Chapter 3 - Interface Metaphors and User Interface Design. In Handbook of Human-Computer Interaction, MARTIN Helander (ed.). North-Holland, Amsterdam, 67–85. DOI:https://doi.org/10.1016/B978-0-444-70536-5.50008-7Google Scholar
- [3] Lung-Pan Cheng, Eyal Ofek, Christian Holz, Hrvoje Benko, and Andrew D. Wilson. 2017. Sparse Haptic Proxy: Touch Feedback in Virtual Environments Using a General Passive Prop. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, ACM, Denver Colorado USA, 3718–3728. DOI:https://doi.org/10.1145/3025453.3025753Google ScholarDigital Library
- [4] Lung-Pan Cheng, Eyal Ofek, Christian Holz, and Andrew D. Wilson. 2019. VRoamer: Generating On-The-Fly VR Experiences While Walking inside Large, Unknown Real-World Building Environments. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), IEEE, Osaka, Japan, 359–366. DOI:https://doi.org/10.1109/VR.2019.8798074Google Scholar
- [5] Isamu Endo, Kazuki Takashima, Maakito Inoue, Kazuyuki Fujita, Kiyoshi Kiyokawa, and Yoshifumi Kitamura. 2021. A Reconfigurable Mobile Head-Mounted Display Supporting Real World Interactions. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–7. Retrieved March 30, 2022 from https://doi.org/10.1145/3411763.3451765Google ScholarDigital Library
- [6] Isamu Endo, Kazuki Takashima, Maakito Inoue, Kazuyuki Fujita, Kiyoshi Kiyokawa, and Yoshifumi Kitamura. 2021. ModularHMD: A Reconfigurable Mobile Head-Mounted Display Enabling Ad-hoc Peripheral Interactions with the Real World. In The 34th Annual ACM Symposium on User Interface Software and Technology, ACM, Virtual Event USA, 100–117. DOI:https://doi.org/10.1145/3472749.3474738Google ScholarDigital Library
- [7] Maia Garau, Doron Friedman, Hila Ritter Widenfeld, Angus Antley, Andrea Brogni, and Mel Slater. 2008. Temporal and Spatial Variations in Presence: Qualitative Analysis of Interviews from an Experiment on Breaks in Presence. Presence: Teleoperators and Virtual Environments 17, 3 (June 2008), 293–309. DOI:https://doi.org/10.1162/pres.17.3.293Google ScholarDigital Library
- [8] Ceenu George, Philipp Janssen, David Heuss, and Florian Alt. 2019. Should I Interrupt or Not? Understanding Interruptions in Head-Mounted Display Settings. In Proceedings of the 2019 on Designing Interactive Systems Conference (DIS ’19), Association for Computing Machinery, New York, NY, USA, 497–510. DOI:https://doi.org/10.1145/3322276.3322363Google ScholarDigital Library
- [9] Sarthak Ghosh, Lauren Winston, Nishant Panchal, Philippe Kimura-Thollander, Jeff Hotnog, Douglas Cheong, Gabriel Reyes, and Gregory D. Abowd. 2018. NotifiVR: Exploring Interruptions and Notifications in Virtual Reality. IEEE Trans. Visual. Comput. Graphics 24, 4 (April 2018), 1447–1456. DOI:https://doi.org/10.1109/TVCG.2018.2793698Google ScholarDigital Library
- [10] Matt Gottsacker, Nahal Norouzi, Kangsoo Kim, Gerd Bruder, and Greg Welch. 2021. Diegetic Representations for Seamless Cross-Reality Interruptions. In 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE, Bari, Italy, 310–319. DOI:https://doi.org/10.1109/ISMAR52148.2021.00047Google Scholar
- [11] Yizheng Gu, Chun Yu, Zhipeng Li, Weiqi Li, Shuchang Xu, Xiaoying Wei, and Yuanchun Shi. 2019. Accurate and Low-Latency Sensing of Touch Contact on Any Surface with Finger-Worn IMU Sensor. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, ACM, New Orleans LA USA, 1059–1070. DOI:https://doi.org/10.1145/3332165.3347947Google ScholarDigital Library
- [12] Daniel Harley, Alexander Verni, Mackenzie Willis, Ashley Ng, Lucas Bozzo, and Ali Mazalek. 2018. Sensory VR: Smelling, Touching, and Eating Virtual Reality. In Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction, ACM, Stockholm Sweden, 386–397. DOI:https://doi.org/10.1145/3173225.3173241Google ScholarDigital Library
- [13] Jeremy Hartmann, Christian Holz, Eyal Ofek, and Andrew D. Wilson. 2019. RealityCheck: Blending Virtual Environments with Situated Physical Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, ACM, Glasgow Scotland Uk, 1–12. DOI:https://doi.org/10.1145/3290605.3300577Google ScholarDigital Library
- [14] Carrie Heeter. 1992. Being There: The Subjective Experience of Presence. Presence: Teleoperators & Virtual Environments 1, 2 (January 1992), 262–271. DOI:https://doi.org/10.1162/pres.1992.1.2.262Google ScholarCross Ref
- [15] K. Hinckley, R. Pausch, J. Goble, and N. Kassell. 1994. Passive real-world interface props for neurosurgical visualization. In Proceedings of the SIGCHI conference on Human factors in computing systems, 452–458. DOI:https://doi.org/10.1145/191666.191821Google ScholarDigital Library
- [16] H.G. Hoffman. 1998. Physically touching virtual objects using tactile augmentation enhances the realism of virtual environments. In Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No.98CB36180), 59–63. DOI:https://doi.org/10.1109/VRAIS.1998.658423Google ScholarCross Ref
- [17] Ching-Yu Hsieh, Yi-Shyuan Chiang, Hung-Yu Chiu, and Yung-Ju Chang. 2020. Bridging the Virtual and Real Worlds: A Preliminary Study of Messaging Notifications in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–14. Retrieved March 29, 2022 from https://doi.org/10.1145/3313831.3376228Google ScholarDigital Library
- [18] Gierad Laput, Karan Ahuja, Mayank Goel, and Chris Harrison. 2018. Ubicoustics: Plug-and-Play Acoustic Activity Recognition. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, ACM, Berlin Germany, 213–224. DOI:https://doi.org/10.1145/3242587.3242609Google ScholarDigital Library
- [19] Bernhard Lehner, Gerhard Widmer, and Sebastian Bock. 2015. A low-latency, real-time-capable singing voice detection method with LSTM recurrent neural networks. In 2015 23rd European Signal Processing Conference (EUSIPCO), 21–25. DOI:https://doi.org/10.1109/EUSIPCO.2015.7362337Google ScholarCross Ref
- [20] Huimin Liu, Zhiquan Wang, Angshuman Mazumdar, and Christos Mousas. 2021. Virtual reality game level layout design for real environment constraints. Graphics and Visual Computing 4, (June 2021), 200020. DOI:https://doi.org/10.1016/j.gvc.2021.200020Google Scholar
- [21] Sebastian Marwecki and Patrick Baudisch. 2018. Scenograph: Fitting Real-Walking VR Experiences into Various Tracking Volumes. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology, ACM, Berlin Germany, 511–520. DOI:https://doi.org/10.1145/3242587.3242648Google ScholarDigital Library
- [22] Brett R. C. Molesworth, Marion Burgess, and Daniel Kwon. 2013. The use of noise cancelling headphones to improve concurrent task performance in a noisy environment. Applied Acoustics 74, 1 (January 2013), 110–115. DOI:https://doi.org/10.1016/j.apacoust.2012.06.015Google ScholarCross Ref
- [23] Yves Moser and Martin A. M. Gijs. 2007. Miniaturized Flexible Temperature Sensor. Journal of Microelectromechanical Systems 16, 6 (December 2007), 1349–1354. DOI:https://doi.org/10.1109/JMEMS.2007.908437Google ScholarCross Ref
- [24] Adiyan Mujibiya, Xiang Cao, Desney S. Tan, Dan Morris, Shwetak N. Patel, and Jun Rekimoto. 2013. The sound of touch: on-body touch and gesture sensing based on transdermal ultrasound propagation. In Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces, ACM, St. Andrews Scotland, United Kingdom, 189–198. DOI:https://doi.org/10.1145/2512349.2512821Google ScholarDigital Library
- [25] Donald A. Norman. 1999. Affordance, conventions, and design. interactions 6, 3 (May 1999), 38–43. DOI:https://doi.org/10.1145/301153.301168Google ScholarDigital Library
- [26] Catherine Oh, Fernanda Herrera, and Jeremy Bailenson. 2019. The Effects of Immersion and Real-World Distractions on Virtual Social Interactions. Cyberpsychology, Behavior, and Social Networking 22, 6 (June 2019), 365–372. DOI:https://doi.org/10.1089/cyber.2018.0404Google ScholarCross Ref
- [27] Joseph O'Hagan, Julie R. Williamson, and Mohamed Khamis. 2020. Bystander interruption of VR users. In Proceedings of the 9TH ACM International Symposium on Pervasive Displays, ACM, Manchester United Kingdom, 19–27. DOI:https://doi.org/10.1145/3393712.3395339Google ScholarDigital Library
- [28] Sigeru Omatu and Mitsuaki Yano. 2016. E-nose system by using neural networks. Neurocomputing 172, (January 2016), 394–398. DOI:https://doi.org/10.1016/j.neucom.2015.03.101Google ScholarDigital Library
- [29] Craig Phillips. 2018. Biomimicry in UX. Medium. Retrieved April 5, 2022 from https://medium.com/@craig5446/biomimicry-in-ux-design-249cc6d8649Google Scholar
- [30] Mirco Rossi, Sebastian Feese, Oliver Amft, Nils Braune, Sandro Martis, and Gerhard Tröster. 2013. AmbientSense: A real-time ambient sound recognition system for smartphones. In 2013 IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops), 230–235. DOI:https://doi.org/10.1109/PerComW.2013.6529487Google ScholarCross Ref
- [31] Rufat Rzayev, Sven Mayer, Christian Krauter, and Niels Henze. 2019. Notification in VR: The Effect of Notification Placement, Task and Environment. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (CHI PLAY ’19), Association for Computing Machinery, New York, NY, USA, 199–211. DOI:https://doi.org/10.1145/3311350.3347190Google ScholarDigital Library
- [32] Munehiko Sato, Ivan Poupyrev, and Chris Harrison. 2012. Touché: enhancing touch interaction on humans, screens, liquids, and everyday objects. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, Austin Texas USA, 483–492. DOI:https://doi.org/10.1145/2207676.2207743Google ScholarDigital Library
- [33] Lior Shapira and Daniel Freedman. 2016. Reality Skins: Creating Immersive and Tactile Virtual Environments. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE, Merida, Yucatan, Mexico, 115–124. DOI:https://doi.org/10.1109/ISMAR.2016.23Google Scholar
- [34] Adalberto L. Simeone, Eduardo Velloso, and Hans Gellersen. 2015. Substitutional Reality: Using the Physical Environment to Design Virtual Reality Experiences. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, ACM, Seoul Republic of Korea, 3307–3316. DOI:https://doi.org/10.1145/2702123.2702389Google ScholarDigital Library
- [35] Mel Slater. A note on presence terminology. Presence connect 3, 3 , 1–5.Google Scholar
- [36] Mel Slater, Andrea Brogni, and Anthony Steed. 2003. Physiological Responses to Breaks in Presence: A Pilot Study. In In The 6th Annual International Workshop on Presence, 3–23.Google Scholar
- [37] Mel Slater and Anthony Steed. 2000. A Virtual Presence Counter. Presence: Teleoperators & Virtual Environments 9, 5 (October 2000), 413–434. DOI:https://doi.org/10.1162/105474600566925Google ScholarDigital Library
- [38] Mel Slater and Sylvia Wilbur. 1997. A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence: Teleoperators & Virtual Environments 6, 6 (December 1997), 603–616. DOI:https://doi.org/10.1162/pres.1997.6.6.603Google ScholarDigital Library
- [39] Misha Sra, Sergio Garrido-Jurado, Chris Schmandt, and Pattie Maes. 2016. Procedurally generated virtual reality from 3D reconstructed physical space. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology, ACM, Munich Germany, 191–200. DOI:https://doi.org/10.1145/2993369.2993372Google ScholarDigital Library
- [40] Li Su, Hailu Wang, Zhen Tian, Haojie Wang, Qian Cheng, and Wei Yu. 2019. Low Detection Limit and High Sensitivity Wind Speed Sensor Based on Triboelectrification-Induced Electroluminescence. Advanced Science 6, 23 (2019), 1901980. DOI:https://doi.org/10.1002/advs.201901980Google ScholarCross Ref
- [41] Qi Sun, Li-Yi Wei, and Arie Kaufman. 2016. Mapping virtual and physical reality. ACM Trans. Graph. 35, 4 (July 2016), 1–12. DOI:https://doi.org/10.1145/2897824.2925883Google ScholarDigital Library
- [42] Kien T.P. Tran, Sungchul Jung, Simon Hoermann, and Robert W. Lindeman. 2019. MDI: A Multi-channel Dynamic Immersion Headset for Seamless Switching between Virtual and Real World Activities. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 350–358. DOI:https://doi.org/10.1109/VR.2019.8798240Google ScholarCross Ref
- [43] Martin Usoh, Ernest Catena, Sima Arman, and Mel Slater. 2000. Using Presence Questionnaires in Reality. Presence: Teleoperators & Virtual Environments 9, 5 (October 2000), 497–503. DOI:https://doi.org/10.1162/105474600566989Google ScholarDigital Library
- [44] Eva L. Waterworth and John A. Waterworth. 2001. Focus, Locus, and Sensus: The Three Dimensions of Virtual Experience. CyberPsychology & Behavior 4, 2 (April 2001), 203–213. DOI:https://doi.org/10.1089/109493101300117893Google ScholarCross Ref
- [45] Danli Wu, Yu Cheng, Dehan Luo, Kin-Yeung Wong, Kevin Hung, and Zhijing Yang. 2019. POP-CNN: Predicting Odor's Pleasantness with Convolutional Neural Network. arXiv:1903.07821 [cs, stat] (March 2019). Retrieved April 1, 2022 from http://arxiv.org/abs/1903.07821Google Scholar
- [46] Jackie (Junrui) Yang, Christian Holz, Eyal Ofek, and Andrew D. Wilson. 2019. DreamWalker: Substituting Real-World Walking Experiences with a Virtual Reality. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, ACM, New Orleans LA USA, 1093–1107. DOI:https://doi.org/10.1145/3332165.3347875Google ScholarDigital Library
- [47] What is Skeuomorphism? - Definition from Techopedia. Techopedia.com. Retrieved April 5, 2022 from http://www.techopedia.com/definition/28955/skeuomorphismGoogle Scholar
- [48] Towards Balancing VR Immersion and Bystander Awareness | Proceedings of the ACM on Human-Computer Interaction. Retrieved from https://dl.acm.org/doi/10.1145/3486950Google Scholar
Index Terms
- Integrating Real-World Distractions into Virtual Reality
Recommendations
Demonstrating the Integration of Real-World Distractions in Virtual Reality
UIST '22 Adjunct: Adjunct Proceedings of the 35th Annual ACM Symposium on User Interface Software and TechnologyAbstract. With the proliferation of consumer-level virtual reality (VR) devices, users started experiencing VR in less controlled environments, such as in social gatherings and public areas. While the current VR hardware provides an increasingly ...
Can Haptic Feedback on One Virtual Object Increase the Presence of Another Virtual Object?
VRST '22: Proceedings of the 28th ACM Symposium on Virtual Reality Software and TechnologyThis paper investigated whether increased presence from experiencing haptic feedback on one virtual object can transfer to another virtual object. Two similar studies were run in different environments: an immersive virtual environment and a mixed ...
Extending Virtual Reality Display Wall Environments Using Augmented Reality
SUI '19: Symposium on Spatial User InteractionTwo major form factors for virtual reality are head-mounted displays and large display environments such as CAVE®and the LCD-based successor CAVE2®. Each of these has distinct advantages and limitations based on how they’re used. This work explores ...
Comments