Jeremy Hartmann
Christian Holz
Eyal Ofek
ABSTRACT
Today's virtual reality (VR) systems offer chaperone rendering techniques that prevent the user from colliding with physical objects. Without a detailed geometric model of the physical world, these techniques offer limited possibility for more advanced compositing between the real world and the virtual. We explore this using a realtime 3D reconstruction of the real world that can be combined with a virtual environment. RealityCheck allows users to freely move, manipulate, observe, and communicate with people and objects situated in their physical space without losing the sense of immersion or presence inside their virtual world. We demonstrate RealityCheck with seven existing VR titles, and describe compositing approaches that address the potential conflicts when rendering the real world and a virtual environment together. A study with frequent VR users demonstrate the affordances provided by our system and how it can be used to enhance current VR experiences.
Supplemental Material
- Mahdi Azmandian, Mark Hancock, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson. 2016. Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI '16. ACM Press, New York, New York, USA, 1968--1979. Google Scholar
Digital Library
- Yoav Benjamini and Yosef Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological) 57, 1 (1995), 289--300. http://www.jstor.org/stable/2346101Google ScholarCross Ref
- Lung-Pan Cheng, Thijs Roumen, Hannes Rantzsch, Sven Köhler, Patrick Schmidt, Robert Kovacs, Johannes Jasper, Jonas Kemper, and Patrick Baudisch. 2015. TurkDeck: Physical Virtual Reality Based on People. Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology - UIST '15 (2015), 417--426. Google ScholarDigital Library
- Mingsong Dou, Jonathan Taylor, Pushmeet Kohli, Vladimir Tankovich, Shahram Izadi, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, and David Kim. 2016. Fusion4D: Real-time Performance Capture of Challenging Scenes. ACM Transactions on Graphics 35, 4 (jul 2016), 1--13. Google ScholarDigital Library
- Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney. 2018. Montage4D: Interactive Seamless Fusion of Multiview Video Textures. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games - I3D '18, Vol. 11. ACM Press, New York, New York, USA, 1--11. Google ScholarDigital Library
- Keinosuke Fukunaga and L. Hostetler. 1975. The estimation of the gradient of a density function, with applications in pattern recognition. IEEE Transactions on Information Theory 21, 1 (jan 1975), 32--40. Google ScholarDigital Library
- Ran Gal, Lior Shapira, Eyal Ofek, and Pushmeet Kohli. 2014. FLARE: Fast layout for augmented reality applications. In 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 207--212.Google ScholarCross Ref
- Jan Gugenheimer, Evgeny Stemasov, Julian Frommel, and Enrico Rukzio. 2017. ShareVR: Enabling Co-Located Experiences for Virtual Reality between HMD and Non-HMD Users. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems CHI '17. ACM Press, New York, New York, USA, 4021--4033. Google ScholarDigital Library
- Anuruddha Hettiarachchi and Daniel Wigdor. 2016. Annexing Reality: Enabling Opportunistic Use of Everyday Objects as Tangible Proxies in Augmented Reality. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI '16. ACM Press, New York, New York, USA, 1957--1967. Google ScholarDigital Library
- Galen Hunt and Doug Brubacher. 1999. Detours: Binary Interception of Win32 Functions. In Proceedings of the 3rd Conference on USENIX Windows NT Symposium - Volume 3 (WINSYM'99). USENIX Association, Berkeley, CA, USA, 14--14. http://dl.acm.org/citation.cfm?id=1268427. Google ScholarDigital Library
- Seokhee Jeon and Seungmoon Choi. 2009. Haptic Augmented Reality: Taxonomy and an Example of Stiffness Modulation. Presence: Teleoperators and Virtual Environments 18, 5 (oct 2009), 387--408. Google ScholarDigital Library
- Brett Jones, Lior Shapira, Rajinder Sodhi, Michael Murdock, Ravish Mehra, Hrvoje Benko, Andrew Wilson, Eyal Ofek, Blair MacIntyre, and Nikunj Raghuvanshi. 2014. RoomAlive: Magical Experiences Enabled by Scalable, Adaptive Projector-camera Units. Proceedings of the 27th annual ACM symposium on User interface software and technology UIST '14 (2014), 637--644. Google ScholarDigital Library
- Brett R. Jones, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson. 2013. IllumiRoom. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI '13. ACM Press, New York, New York, USA, 869.Google Scholar
- Takeo Kanade, Peter Rander, and PJ Narayanan. 1997. Virtualized reality: Constructing virtual worlds from real scenes. IEEE multimedia 4, 1 (1997), 34--47. Google ScholarDigital Library
- Pascal Knierim, Valentin Schwind, Anna Maria Feit, Florian Nieuwenhuizen, and Niels Henze. 2018. Physical Keyboards in Virtual Reality: Analysis of Typing Performance and Effects of Avatar Hands. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. ACM Press, New York, New York, USA, 1--9. Google ScholarDigital Library
- Ryohei Komiyama, Takashi Miyaki, and Jun Rekimoto. 2017. JackIn Space: Designing a Seamless Transition Between First and Third Person View for Effective Telepresence Collaborations Ryohei. In Proceedings of the 8th Augmented Human International Conference on - AH '17. ACM Press, New York, New York, USA, 1--9. Google ScholarDigital Library
- Jérémy Lacoche, Nico Pallamin, Thomas Boggini, and Jérôme Royan. 2017. Improved Redirection with Distractors: A large-scale-realwalking locomotion interface and its effect on navigation in virtual environments. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology - VRST '17. ACM Press, New York, New York, USA, 1--9.Google Scholar
- David Lindlbauer and Andy D Wilson. 2018. Remixed Reality: Manipulating Space and Time in Augmented Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. ACM Press, New York, New York, USA, 1--13. Google ScholarDigital Library
- Christian Mai, Mariam Hassib, and Ceenu George. 2017. Like Elephants Do: Sensing Bystanders During HMD Usage. Technical Report. https://pdfs.semanticscholar.org/5513/ 8687c17ad46747c95ab33fa9c4851ee2d26e.pdf?Google Scholar
- Steve Mann. 1999. Mediated Reality. Linux J. 1999, 59es, Article 5 (March 1999). http://dl.acm.org/citation.cfm?id=327697.327702 Google ScholarDigital Library
- Sebastian Marwecki, Maximilian Brehm, Lukas Wagner, Lung-Pan Cheng, Florian 'Floyd' Mueller, and Patrick Baudisch. 2018. VirtualSpace - Overloading Physical Space with Multiple Virtual Reality Users. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. ACM Press, New York, New York, USA, 1--10. Google ScholarDigital Library
- John C. McClelland, Robert J Teather, and Audrey Girouard. 2017. Haptobend: Shape-Changing Passive Haptic Feedback in Virtual Reality. In Proceedings of the 5th Symposium on Spatial User Interaction - SUI '17. ACM Press, New York, New York, USA, 82--90. Google ScholarDigital Library
- Mark McGill, Daniel Boland, Roderick Murray-Smith, and Stephen Brewster. 2015. A dose of reality: overcoming usability challenges in vr head-mounted displays. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI '15. ACM Press, New York, New York, USA, 2143--2152. Google ScholarDigital Library
- Paul Milgram and Herman Colquhoun. 1999. A taxonomy of real and virtual world display integration. Mixed reality: Merging real and virtual worlds 1 (1999), 1--26.Google Scholar
- Paul Milgram and Fumio Kishino. 1994. A Taxonomy of Mixed Reality Visual Displays Augmented Reality. Technical Report 12. http://vered. rose.utoronto.ca/people/paulGoogle Scholar
- Raul Mur-Artal and Juan D. Tardos. 2017. ORB-SLAM2: An OpenSource SLAM System for Monocular, Stereo, and RGB-D Cameras. IEEE Transactions on Robotics 33, 5 (oct 2017), 1255--1262. arXiv:1610.06475Google ScholarDigital Library
- Tabitha C. Peck, Henry Fuchs, and Mary C. Whitton. 2010. Improved Redirection with Distractors: A large-scale-real-walking locomotion interface and its effect on navigation in virtual environments. In 2010 IEEE Virtual Reality Conference (VR). IEEE, 35--38. Google ScholarDigital Library
- Julian Petford, Miguel A Nacenta, Carl Gutwin, Joseph Eremondi, and Cody Ede. 2016. The ASPECTA Toolkit: Affordable Full Coverage Displays. In Proceedings of the 5th ACM International Symposium on Pervasive Displays - PerDis '16. ACM Press, New York, New York, USA, 87--105. Google ScholarDigital Library
- Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 779--788. arXiv:1506.02640Google ScholarCross Ref
- Erik Reinhard, M. Adhikhmin, Bruce Gooch, and Peter Shirley. 2001. Color transfer between images. IEEE Computer Graphics and Applications 21, 4 (2001), 34--41. Google ScholarDigital Library
- Jeffrey M. Richter. 1999. Programming Applications for Microsoft Windows with Cdrom (4th ed.). Microsoft Press, Redmond, WA, USA. Google ScholarDigital Library
- Joan Sol Roo and Martin Hachet. 2017. One Reality. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology - UIST '17. ACM Press, New York, New York, USA, 787--795.Google Scholar
- Lior Shapira and Daniel Freedman. 2016. Reality Skins: Creating Immersive and Tactile Virtual Environments. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 115--124.Google Scholar
- Adalberto L Simeone. 2015. Substitutional reality: Towards a research agenda. In 2015 IEEE 1st Workshop on Everyday Virtual Reality (WEVR). IEEE, 19--22.Google ScholarCross Ref
- Shuran Song, Fisher Yu, Andy Zeng, Angel X. Chang, Manolis Savva, and Thomas Funkhouser. 2017. Semantic Scene Completion from a Single Depth Image. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 190--198. arXiv:1611.08974Google ScholarCross Ref
- Misha Sra, Sergio Garrido-Jurado, and Chris Schmandt. 2016. Procedurally generated virtual reality from 3D reconstructed physical space. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology - VRST '16. ACM Press, New York, New York, USA, 191--200. Google ScholarDigital Library
- William Steptoe, Simon Julier, and Anthony Steed. 2014. Presence and discernability in conventional and non-photorealistic immersive augmented reality. In 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 213--218.Google ScholarCross Ref
- Qi Sun, Arie Kaufman, Anjul Patney, Li-Yi Wei, Omer Shapira, Jingwan Lu, Paul Asente, Suwen Zhu, Morgan Mcguire, and David Luebke. 2018. Towards Virtual Reality Infinite Walking: Dynamic Saccadic Redirection. ACM Transactions on Graphics 37, 4 (jul 2018), 1--13. Google ScholarDigital Library
- Qi Sun, Li-Yi Wei, and Arie Kaufman. 2016. Mapping virtual and physical reality. ACM Transactions on Graphics 35, 4 (jul 2016), 1--12. Google ScholarDigital Library
- Sungjoon Choi, Qian-Yi Zhou, and Vladlen Koltun. 2015. Robust reconstruction of indoor scenes. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 5556--5565.Google ScholarCross Ref
- Martin Usoh, Ernest Catena, Sima Arman, and Mel Slater. 2000. Using Presence Questionnaires in Reality. Presence: Teleoperators and Virtual Environments 9, 5 (oct 2000), 497--503. Google ScholarDigital Library
- Andrew Wilson. 2017. Fast Lossless Depth Image Compression. In Proceedings of the Interactive Surfaces and Spaces on ZZZ - ISS '17. ACM Press, New York, New York, USA, 100--105. Google ScholarDigital Library
- Andrew D Wilson and Hrvoje Benko. 2016. Projected Augmented Reality with the RoomAlive Toolkit. In Proceedings of the 2016 ACM on Interactive Surfaces and Spaces. ACM, 517--520. Google ScholarDigital Library
- Jacob O Wobbrock, Leah Findlater, Darren Gergle, and James J Higgins. 2011. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the 2011 annual conference on Human factors in computing systems - CHI '11. ACM Press, New York, New York, USA, 143. Google ScholarDigital Library
- Yiwei Zhao and Sean Follmer. 2018. A Functional Optimization Based Approach for Continuous 3D Retargeted Touch of Arbitrary, Complex Boundaries in Haptic Virtual Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18. ACM Press, New York, New York, USA, 1--12. Google ScholarDigital Library
Index Terms
- RealityCheck: Blending Virtual Environments with Situated Physical Reality
Recommendations
Procedurally generated virtual reality from 3D reconstructed physical space
VRST '16: Proceedings of the 22nd ACM Conference on Virtual Reality Software and TechnologyWe present a novel system for automatically generating immersive and interactive virtual reality (VR) environments using the real world as a template. The system captures indoor scenes in 3D, detects obstacles like furniture and walls, and maps walkable ...
SpaceWalk: Movement and Interaction in Virtual Space with Commodity Hardware
IE2014: Proceedings of the 2014 Conference on Interactive EntertainmentWe introduce SpaceWalk, an experimental approach to allow a person to move around a small room while wearing a Head Mounted Display. The goal is to get immersed users in everyday settings out of their seats and naturally experiencing movement within the ...
Haptic around: multiple tactile sensations for immersive environment and interaction in virtual reality
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and TechnologyIn this paper, we present Haptic Around, a hybrid-haptic feedback system, which utilizes fan, hot air blower, mist creator and heat light to recreate multiple tactile sensations in virtual reality for enhancing the immersive environment and interaction. ...
Comments