skip to main content
10.1145/3290605.3300577acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

RealityCheck: Blending Virtual Environments with Situated Physical Reality

Published: 02 May 2019 Publication History

Abstract

Today's virtual reality (VR) systems offer chaperone rendering techniques that prevent the user from colliding with physical objects. Without a detailed geometric model of the physical world, these techniques offer limited possibility for more advanced compositing between the real world and the virtual. We explore this using a realtime 3D reconstruction of the real world that can be combined with a virtual environment. RealityCheck allows users to freely move, manipulate, observe, and communicate with people and objects situated in their physical space without losing the sense of immersion or presence inside their virtual world. We demonstrate RealityCheck with seven existing VR titles, and describe compositing approaches that address the potential conflicts when rendering the real world and a virtual environment together. A study with frequent VR users demonstrate the affordances provided by our system and how it can be used to enhance current VR experiences.

Supplementary Material

MP4 File (paper347.mp4)
Supplemental video
MP4 File (paper347p.mp4)
Preview video

References

[1]
Mahdi Azmandian, Mark Hancock, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson. 2016. Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI '16. ACM Press, New York, New York, USA, 1968--1979.
[2]
Yoav Benjamini and Yosef Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological) 57, 1 (1995), 289--300. http://www.jstor.org/stable/2346101
[3]
Lung-Pan Cheng, Thijs Roumen, Hannes Rantzsch, Sven Köhler, Patrick Schmidt, Robert Kovacs, Johannes Jasper, Jonas Kemper, and Patrick Baudisch. 2015. TurkDeck: Physical Virtual Reality Based on People. Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology - UIST '15 (2015), 417--426.
[4]
Mingsong Dou, Jonathan Taylor, Pushmeet Kohli, Vladimir Tankovich, Shahram Izadi, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, and David Kim. 2016. Fusion4D: Real-time Performance Capture of Challenging Scenes. ACM Transactions on Graphics 35, 4 (jul 2016), 1--13.
[5]
Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney. 2018. Montage4D: Interactive Seamless Fusion of Multiview Video Textures. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games - I3D '18, Vol. 11. ACM Press, New York, New York, USA, 1--11.
[6]
Keinosuke Fukunaga and L. Hostetler. 1975. The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Transactions on Information Theory 21, 1 (jan 1975), 32--40.
[7]
Ran Gal, Lior Shapira, Eyal Ofek, and Pushmeet Kohli. 2014. FLARE: Fast layout for augmented reality applications. In 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 207--212.
[8]
Jan Gugenheimer, Evgeny Stemasov, Julian Frommel, and Enrico Rukzio. 2017. ShareVR: Enabling Co-Located Experiences for Virtual Reality between HMD and Non-HMD Users. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems CHI '17. ACM Press, New York, New York, USA, 4021--4033.
[9]
Anuruddha Hettiarachchi and Daniel Wigdor. 2016. Annexing Reality: Enabling Opportunistic Use of Everyday Objects as Tangible Proxies in Augmented Reality. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI '16. ACM Press, New York, New York, USA, 1957--1967.
[10]
Galen Hunt and Doug Brubacher. 1999. Detours: Binary Interception of Win32 Functions. In Proceedings of the 3rd Conference on USENIX Windows NT Symposium - Volume 3 (WINSYM'99). USENIX Association, Berkeley, CA, USA, 14--14. http://dl.acm.org/citation.cfm?id=1268427.
[11]
Seokhee Jeon and Seungmoon Choi. 2009. Haptic Augmented Reality: Taxonomy and an Example of Stiffness Modulation. Presence: Teleoperators and Virtual Environments 18, 5 (oct 2009), 387--408.
[12]
Brett Jones, Lior Shapira, Rajinder Sodhi, Michael Murdock, Ravish Mehra, Hrvoje Benko, Andrew Wilson, Eyal Ofek, Blair MacIntyre, and Nikunj Raghuvanshi. 2014. RoomAlive: Magical Experiences Enabled by Scalable, Adaptive Projector-camera Units. Proceedings of the 27th annual ACM symposium on User interface software and technology UIST '14 (2014), 637--644.
[13]
Brett R. Jones, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson. 2013. IllumiRoom. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI '13. ACM Press, New York, New York, USA, 869.
[14]
Takeo Kanade, Peter Rander, and PJ Narayanan. 1997. Virtualized reality: Constructing virtual worlds from real scenes. IEEE multimedia 4, 1 (1997), 34--47.
[15]
Pascal Knierim, Valentin Schwind, Anna Maria Feit, Florian Nieuwenhuizen, and Niels Henze. 2018. Physical Keyboards in Virtual Reality: Analysis of Typing Performance and Effects of Avatar Hands. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. ACM Press, New York, New York, USA, 1--9.
[16]
Ryohei Komiyama, Takashi Miyaki, and Jun Rekimoto. 2017. JackIn Space: Designing a Seamless Transition Between First and Third Person View for Effective Telepresence Collaborations Ryohei. In Proceedings of the 8th Augmented Human International Conference on - AH '17. ACM Press, New York, New York, USA, 1--9.
[17]
Jérémy Lacoche, Nico Pallamin, Thomas Boggini, and Jérôme Royan. 2017. Improved Redirection with Distractors: A large-scale-realwalking locomotion interface and its effect on navigation in virtual environments. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology - VRST '17. ACM Press, New York, New York, USA, 1--9.
[18]
David Lindlbauer and Andy D Wilson. 2018. Remixed Reality: Manipulating Space and Time in Augmented Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. ACM Press, New York, New York, USA, 1--13.
[19]
Christian Mai, Mariam Hassib, and Ceenu George. 2017. Like Elephants Do: Sensing Bystanders During HMD Usage. Technical Report. https://pdfs.semanticscholar.org/5513/ 8687c17ad46747c95ab33fa9c4851ee2d26e.pdf?
[20]
Steve Mann. 1999. Mediated Reality. Linux J. 1999, 59es, Article 5 (March 1999). http://dl.acm.org/citation.cfm?id=327697.327702
[21]
Sebastian Marwecki, Maximilian Brehm, Lukas Wagner, Lung-Pan Cheng, Florian 'Floyd' Mueller, and Patrick Baudisch. 2018. VirtualSpace - Overloading Physical Space with Multiple Virtual Reality Users. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. ACM Press, New York, New York, USA, 1--10.
[22]
John C. McClelland, Robert J Teather, and Audrey Girouard. 2017. Haptobend: Shape-Changing Passive Haptic Feedback in Virtual Reality. In Proceedings of the 5th Symposium on Spatial User Interaction - SUI '17. ACM Press, New York, New York, USA, 82--90.
[23]
Mark McGill, Daniel Boland, Roderick Murray-Smith, and Stephen Brewster. 2015. A dose of reality: overcoming usability challenges in vr head-mounted displays. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI '15. ACM Press, New York, New York, USA, 2143--2152.
[24]
Paul Milgram and Herman Colquhoun. 1999. A taxonomy of real and virtual world display integration. Mixed reality: Merging real and virtual worlds 1 (1999), 1--26.
[25]
Paul Milgram and Fumio Kishino. 1994. A Taxonomy of Mixed Reality Visual Displays Augmented Reality. Technical Report 12. http://vered. rose.utoronto.ca/people/paul
[26]
Raul Mur-Artal and Juan D. Tardos. 2017. ORB-SLAM2: An OpenSource SLAM System for Monocular, Stereo, and RGB-D Cameras. IEEE Transactions on Robotics 33, 5 (oct 2017), 1255--1262. arXiv:1610.06475
[27]
Tabitha C. Peck, Henry Fuchs, and Mary C. Whitton. 2010. Improved Redirection with Distractors: A large-scale-real-walking locomotion interface and its effect on navigation in virtual environments. In 2010 IEEE Virtual Reality Conference (VR). IEEE, 35--38.
[28]
Julian Petford, Miguel A Nacenta, Carl Gutwin, Joseph Eremondi, and Cody Ede. 2016. The ASPECTA Toolkit: Affordable Full Coverage Displays. In Proceedings of the 5th ACM International Symposium on Pervasive Displays - PerDis '16. ACM Press, New York, New York, USA, 87--105.
[29]
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 779--788. arXiv:1506.02640
[30]
Erik Reinhard, M. Adhikhmin, Bruce Gooch, and Peter Shirley. 2001. Color transfer between images. IEEE Computer Graphics and Applications 21, 4 (2001), 34--41.
[31]
Jeffrey M. Richter. 1999. Programming Applications for Microsoft Windows with Cdrom (4th ed.). Microsoft Press, Redmond, WA, USA.
[32]
Joan Sol Roo and Martin Hachet. 2017. One Reality. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology - UIST '17. ACM Press, New York, New York, USA, 787--795.
[33]
Lior Shapira and Daniel Freedman. 2016. Reality Skins: Creating Immersive and Tactile Virtual Environments. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 115--124.
[34]
Adalberto L Simeone. 2015. Substitutional reality: Towards a research agenda. In 2015 IEEE 1st Workshop on Everyday Virtual Reality (WEVR). IEEE, 19--22.
[35]
Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva, and Thomas Funkhouser. 2017. Semantic Scene Completion from a Single Depth Image. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 190--198. arXiv:1611.08974
[36]
Misha Sra, Sergio Garrido-Jurado, and Chris Schmandt. 2016. Procedurally generated virtual reality from 3D reconstructed physical space. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology - VRST '16. ACM Press, New York, New York, USA, 191--200.
[37]
William Steptoe, Simon Julier, and Anthony Steed. 2014. Presence and discernability in conventional and non-photorealistic immersive augmented reality. In 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 213--218.
[38]
Qi Sun, Arie Kaufman, Anjul Patney, Li-Yi Wei, Omer Shapira, Jingwan Lu, Paul Asente, Suwen Zhu, Morgan Mcguire, and David Luebke. 2018. Towards Virtual Reality Infinite Walking: Dynamic Saccadic Redirection. ACM Transactions on Graphics 37, 4 (jul 2018), 1--13.
[39]
Qi Sun, Li-Yi Wei, and Arie Kaufman. 2016. Mapping virtual and physical reality. ACM Transactions on Graphics 35, 4 (jul 2016), 1--12.
[40]
Sungjoon Choi, Qian-Yi Zhou, and Vladlen Koltun. 2015. Robust reconstruction of indoor scenes. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 5556--5565.
[41]
Martin Usoh, Ernest Catena, Sima Arman, and Mel Slater. 2000. Using Presence Questionnaires in Reality. Presence: Teleoperators and Virtual Environments 9, 5 (oct 2000), 497--503.
[42]
Andrew Wilson. 2017. Fast Lossless Depth Image Compression. In Proceedings of the Interactive Surfaces and Spaces on ZZZ - ISS '17. ACM Press, New York, New York, USA, 100--105.
[43]
Andrew D Wilson and Hrvoje Benko. 2016. Projected Augmented Reality with the RoomAlive Toolkit. In Proceedings of the 2016 ACM on Interactive Surfaces and Spaces. ACM, 517--520.
[44]
Jacob O Wobbrock, Leah Findlater, Darren Gergle, and James J Higgins. 2011. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the 2011 annual conference on Human factors in computing systems - CHI '11. ACM Press, New York, New York, USA, 143.
[45]
Yiwei Zhao and Sean Follmer. 2018. A Functional Optimization Based Approach for Continuous 3D Retargeted Touch of Arbitrary, Complex Boundaries in Haptic Virtual Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. ACM Press, New York, New York, USA, 1--12.

Cited By

View all
  • (2024)VR Headsets In-The-Wild: Qualitative Insights on Safety, Awareness, and Social Challenges from Real Train JourneysProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701576(82-94)Online publication date: 1-Dec-2024
  • (2024)Space Renovation in Virtual RealityCompanion Publication of the 2024 ACM Designing Interactive Systems Conference10.1145/3656156.3663711(130-134)Online publication date: 1-Jul-2024
  • (2024)SituationAdapt: Contextual UI Optimization in Mixed Reality with Situation Awareness via LLM ReasoningProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676470(1-13)Online publication date: 13-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
May 2019
9077 pages
ISBN:9781450359702
DOI:10.1145/3290605
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. 3d compositing
  2. depth cameras
  3. virtual reality

Qualifiers

  • Research-article

Conference

CHI '19
Sponsor:

Acceptance Rates

CHI '19 Paper Acceptance Rate 703 of 2,958 submissions, 24%;
Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)352
  • Downloads (Last 6 weeks)29
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)VR Headsets In-The-Wild: Qualitative Insights on Safety, Awareness, and Social Challenges from Real Train JourneysProceedings of the International Conference on Mobile and Ubiquitous Multimedia10.1145/3701571.3701576(82-94)Online publication date: 1-Dec-2024
  • (2024)Space Renovation in Virtual RealityCompanion Publication of the 2024 ACM Designing Interactive Systems Conference10.1145/3656156.3663711(130-134)Online publication date: 1-Jul-2024
  • (2024)SituationAdapt: Contextual UI Optimization in Mixed Reality with Situation Awareness via LLM ReasoningProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676470(1-13)Online publication date: 13-Oct-2024
  • (2024)GradualReality: Enhancing Physical Object Interaction in Virtual Reality via Interaction State-Aware BlendingProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676463(1-14)Online publication date: 13-Oct-2024
  • (2024)Augmented Object Intelligence with XR-ObjectsProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676379(1-15)Online publication date: 13-Oct-2024
  • (2024)RealityEffects: Augmenting 3D Volumetric Videos with Object-Centric Annotation and Dynamic Visual EffectsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661631(1248-1261)Online publication date: 1-Jul-2024
  • (2024)SoundShift: Exploring Sound Manipulations for Accessible Mixed-Reality AwarenessProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661556(116-132)Online publication date: 1-Jul-2024
  • (2024)Influence of Rotation Gains on Unintended Positional Drift during Virtual Steering Navigation in Virtual RealityProceedings of the 30th ACM Symposium on Virtual Reality Software and Technology10.1145/3641825.3687734(1-10)Online publication date: 9-Oct-2024
  • (2024)Virtual Games, Real Interactions: A Look at Cross-reality Asymmetrical Co-located Social GamesExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650824(1-9)Online publication date: 11-May-2024
  • (2024)SwitchSpace: Understanding Context-Aware Peeking Between VR and Desktop InterfacesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642358(1-16)Online publication date: 11-May-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media