Qi Feng
Hubert P. H. Shum
ABSTRACT
Due to the need to interact with virtual objects, the hand-object interaction has become an important element in mixed reality (MR) applications. In this paper, we propose a novel approach to handle the occlusion of augmented 3D object manipulation with hands by exploiting the nature of hand poses combined with tracking-based and model-based methods, to achieve a complete mixed reality experience without necessities of heavy computations, complex manual segmentation processes or wearing special gloves. The experimental results show a frame rate faster than real-time and a great accuracy of rendered virtual appearances, and a user study verifies a more immersive experience compared to past approaches. We believe that the proposed method can improve a wide range of mixed reality applications that involve hand-object interactions.
Supplemental Material
- Ronald T. Azuma. 2017. Making Augmented Reality a Reality, In Imaging and Applied Optics 2017. Imaging and Applied Optics 2017, JTu1F.1. http://www.osapublishing.org/abstract.cfm?URI=AIO-2017-JTu1F.1Google Scholar
- Benjamin Bach, Ronell Sicat, Johanna Beyer, Maxime Cordeil, and Hanspeter Pfister. 2018. The Hologram in My Hand: How Effective is Interactive Exploration of 3D Visualizations in Immersive Tangible Augmented Reality? IEEE Transactions on Visualization & Computer Graphics 1 (2018), 1--1.Google Scholar
- Max Krichenbauer, Goshiro Yamamoto, Takafumi Taketom, Christian Sandor, and Hirokazu Kato. 2018. Augmented Reality versus Virtual Reality for 3D Object Manipulation. IEEE transactions on visualization and computer graphics 24, 2 (2018), 1038--1048.Google Scholar
- Hui Liang, Junsong Yuan, Daniel Thalmann, and Nadia Magnenat Thalmann. 2015. Ar in hand: Egocentric palm pose tracking and gesture recognition for augmented reality applications. In Proceedings of the 23rd ACM international conference on Multimedia. ACM, 743--744. Google ScholarDigital Library
- Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In Mixed and augmented reality (ISMAR), 2011 10th IEEE international symposium on. IEEE, 127--136. Google ScholarDigital Library
- Yuan Tian, Tao Guan, and Cheng Wang. 2010. Real-time occlusion handling in augmented reality based on an object tracking approach. Sensors 10, 4 (2010), 2885--2900.Google ScholarCross Ref
- Yuan Tian, Yan Long, Dan Xia, Huang Yao, and Jincheng Zhang. 2015. Handling occlusions in augmented reality based on 3D reconstruction method. Neurocomputing 156 (2015), 96--104. Google ScholarDigital Library
- Frank Weichert, Daniel Bachmann, Bartholomäus Rudak, and Denis Fisseler. 2013. Analysis of the accuracy and robustness of the leap motion controller. Sensors 13, 5 (2013), 6380--6393.Google ScholarCross Ref
Index Terms
- Resolving occlusion for 3D object manipulation with hands in mixed reality
Recommendations
Resolving occlusion in augmented reality
I3D '95: Proceedings of the 1995 symposium on Interactive 3D graphicsCurrent state-of-the-art augmented reality systems simply overlay computer-generated visuals on the real-world imagery, for example via video or optical see-through displays. However, overlays are not effective when displaying data in three dimensions, ...
Mixed Reality MIDI Keyboard Demonstration
AM '17: Proceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music ExperiencesThe Mixed Reality MIDI Keyboard is a prototype designed to augment virtual reality experiences through the inclusion of a physical interface which aligns the user's senses with the virtual environment. It also serves as a platform on which the uses of ...
Real-time 3D video avatar in mixed reality: an implementation for immersive telecommunication
Symposium: virtual reality simulationThis article presents an implementation of a real-time dynamic 3D avatar from multiview cameras for immersive telecommunication. Immersive telecommunication is a new challenging field that enables a user to share a virtual space with remote participants,...
Comments