Kit Yung Lam
ABSTRACT
3D objects are the primary media in virtual reality environments in immersive cyberspace, also known as the Metaverse. Users, through editing such objects, can communicate with other individuals on mobile headsets. Knowing that the tangible controllers cause the burden to carry such addendum devices, the body-centric interaction techniques, such as hand gestures, get rid of such burdens. However, object editing with hand gestures is usually overlooked. Accordingly, we propose and implement a palm-based virtual embodiment for hand gestural model editing, namely 3DeformR. We employ three optimized hand gestures on bi-harmonic deformation algorithms that enable selecting and editing 3D models in fine granularity. Our evaluation with nine participants considers three interaction techniques (two-handed tangible controller (OMC), a naive implementation of hand gestures (SH), and 3DeformR. Two experimental tasks of planar and spherical objects imply that 3DeformR outperforms SH, in terms of task completion time (~51%) and required actions (~17%). Also, our participants with 3DeformR make significantly better performance than the commercial standard (OMC) - saved task time (~43%) and actions (~3%). Remarkably, the edited objects by 3DeformR show no discernible difference from those with tangible controllers characterised by accurate and responsive detection.
Supplemental Material
- Ferran Argelaguet and Carlos Andujar. 2013. A survey of 3D object selection techniques for virtual environments. Computers Graphics 37, 3 (2013), 121--136. Google Scholar
Digital Library
- Carlos Bermejo, Lik Hang Lee, Paul Chojecki, David Przewozny, and Pan Hui. 2021. Exploring Button Designs for Mid-Air Interaction in Virtual Reality: A Hexa-Metric Evaluation of Key Representations and Multi-Modal Cues. Proc. ACM Hum.-Comput. Interact. 5, EICS, Article 194 (may 2021), 26 pages. Google ScholarDigital Library
- Tara Chittenden. 2018. Tilt Brush painting: Chronotopic adventures in a physicalvirtual threshold. Journal of contemporary painting 4, 2 (2018), 381--403.Google ScholarCross Ref
- Sabine Coquillart. 1990. Extended free-form deformation: A sculpturing tool for 3D geometric modeling. In Proceedings of the 17th annual conference on Computer graphics and interactive techniques. 187--196.Google ScholarDigital Library
- Fabrizio Corda, Jean-Marc Thiery, Marco Livesu, Enrico Puppo, Tamy Boubekeur, and Riccardo Scateni. 2020. Real-Time Deformation with Coupled Cages and Skeletons. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 19--32.Google Scholar
- Jian Cui, Dieter W Fellner, Arjan Kuijper, and Alexei Sourin. 2016. Mid-air gestures for virtual modeling with leap motion. In International Conference on Distributed, Ambient, and Pervasive Interactions. Springer, 221--230.Google ScholarCross Ref
- Henrique Galvan Debarba, Ronan Boulic, Roy Salomon, Olaf Blanke, and Bruno Herbelin. 2018. Self-attribution of distorted reaching movements in immersive virtual reality. Computers & Graphics 76 (2018), 142--152.Google ScholarCross Ref
- Facebook. 2018. Quill VR illustration and animation tool built to empower artists and creators. https://quill.fb.com/Google Scholar
- Scott Frees, G. Drew Kessler, and Edwin Kay. 2007. PRISM Interaction for Enhancing Control in Immersive Virtual Environments. ACM Trans. Comput.-Hum. Interact. 14, 1 (May 2007), 2--es. Google ScholarDigital Library
- Igor Guskov, Wim Sweldens, and Peter Schröder. 1999. Multiresolution signal processing for meshes. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques. 325--334.Google ScholarDigital Library
- Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. In Human Mental Workload, Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, 139--183. Google ScholarCross Ref
- Ammar Hattab and Gabriel Taubin. 2019. Rough Carving of 3D Models with Spatial Augmented Reality. In Proceedings of the ACM Symposium on Computational Fabrication (Pittsburgh, Pennsylvania) (SCF '19). Association for Computing Machinery, New York, NY, USA, Article 4, 10 pages. Google ScholarDigital Library
- Zhanpeng Huang, Weikai Li, and Pan Hui. 2015. Ubii: Towards Seamless Interaction between Digital and Physical Worlds. In Proceedings of the 23rd ACM International Conference on Multimedia (Brisbane, Australia) (MM '15). Association for Computing Machinery, New York, NY, USA, 341--350. Google ScholarDigital Library
- Alec Jacobson, Ilya Baran, Jovan Popovic, and Olga Sorkine. 2011. Bounded biharmonic weights for real-time deformation. ACM Trans. Graph. 30, 4 (2011), 78.Google ScholarDigital Library
- Alec Jacobson and Daniele Panozzo. 2017. libigl: prototyping geometry processing research in C++. In SIGGRAPH Asia 2017 courses. 1--172.Google Scholar
- Ying Jiang, Congyi Zhang, Hongbo Fu, Alberto Cannavò, Fabrizio Lamberti, Henry Y K Lau, and Wenping Wang. 2021. HandPainter - 3D Sketching in VR with Hand-Based Physical Proxy. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 412, 13 pages. Google ScholarDigital Library
- Pushkar Joshi, Mark Meyer, Tony DeRose, Brian Green, and Tom Sanocki. 2007. Harmonic coordinates for character articulation. ACM Transactions on Graphics (TOG) 26, 3 (2007), 71--es.Google ScholarDigital Library
- Jamil Joundi, Yannick Christiaens, Jelle Saldien, Peter Conradie, and Lien De Marez. 2020. An explorative study towards using VR sketching as a tool for ideation and prototyping in product design. In Proceedings of the Design Society: DESIGN Conference, Vol. 1. Cambridge University Press, 225--234.Google ScholarCross Ref
- Tao Ju, Scott Schaefer, and Joe Warren. 2005. Mean value coordinates for closed triangular meshes. In ACM Siggraph 2005 Papers. 561--566.Google Scholar
- Leif Kobbelt, Swen Campagna, Jens Vorsatz, and Hans-Peter Seidel. 1998. Interactive multi-resolution modeling on arbitrary meshes. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques. 105--114.Google ScholarDigital Library
- Kit-Yung Lam, Lik-Hang Lee, and Pan Hui. 2021. A2W: Context-Aware Recommendation System for Mobile Augmented Reality Web Browser. Proceedings of the 29th ACM International Conference on Multimedia (2021).Google ScholarDigital Library
- Lik Hang Lee, Tristan Braud, Farshid Hassani Bijarbooneh, and Pan Hui. 2019. TiPoint: Detecting Fingertip for Mid-Air Interaction on Computational Resource Constrained Smartglasses. In Proceedings of the 23rd International Symposium on Wearable Computers (London, United Kingdom) (ISWC '19). Association for Computing Machinery, New York, NY, USA, 118--122. Google ScholarDigital Library
- Lik Hang Lee, Tristan Braud, Farshid Hassani Bijarbooneh, and Pan Hui. 2020. UbiPoint: Towards Non-Intrusive Mid-Air Interaction for Hardware Constrained Smart Glasses. In Proceedings of the 11th ACM Multimedia Systems Conference (Istanbul, Turkey) (MMSys '20). Association for Computing Machinery, New York, NY, USA, 190--201. Google ScholarDigital Library
- Lik-Hang Lee, Tristan Braud, Simo Hosio, and Pan Hui. 2021. Towards Augmented Reality Driven Human-City Interaction: Current Research on Mobile Headsets and Future Challenges. ACM Comput. Surv. 54, 8, Article 165 (oct 2021), 38 pages. Google ScholarDigital Library
- Lik-Hang Lee, Zijun Lin, Rui Hu, Zhengya Gong, Abhishek Kumar, Tangyao Li, Sijia Li, and Pan Hui. 2021. When Creators Meet the Metaverse: A Survey on Computational Arts. ArXiv abs/2111.13486 (2021).Google Scholar
- Taehee Lee and Tobias Hollerer. 2007. Handy AR: Markerless inspection of augmented reality objects using fingertip tracking. In 2007 11th IEEE International Symposium on Wearable Computers. IEEE, 83--90.Google ScholarDigital Library
- John P Lewis, Matt Cordner, and Nickson Fong. 2000. Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 165--172.Google ScholarDigital Library
- Ke Lu, Qian Wang, Jian Xue, and Weiguo Pan. 2014. 3D model retrieval and classification by semi-supervised learning with content-based similarity. Information Sciences 281 (2014), 703--713.Google ScholarDigital Library
- D. Mendes, F. M. Caputo, A. Giachetti, A. Ferreira, and J. Jorge. 2019. A Survey on 3D Virtual Object Manipulation: From the Desktop to Immersive Virtual Environments. Computer Graphics Forum 38, 1 (2019), 21--45. arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/cgf.13390 Google ScholarCross Ref
- Frol Periverzov and Horea Ilieş. 2015. IDS: The intent driven selection method for natural user interfaces. In 2015 IEEE Symposium on 3D User Interfaces (3DUI). 121--128. Google ScholarCross Ref
- Jing Qian, Jiaju Ma, Xiangyu Li, Benjamin Attal, Haoming Lai, James Tompkin, John F Hughes, and Jeff Huang. 2019. Portal-ble: Intuitive free-hand manipulation in unbounded smartphone-based augmented reality. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 133--145.Google ScholarDigital Library
- Johanna Schmidt, Reinhold Preiner, Thomas Auzinger, Michael Wimmer, M Eduard Gröller, and Stefan Bruckner. 2014. YMCA---Your mesh comparison application. In 2014 IEEE Conference on Visual Analytics Science and Technology (VAST). IEEE, 153--162.Google ScholarCross Ref
- Thomas W Sederberg and Scott R Parry. 1986. Free-form deformation of solid geometric models. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques. 151--160.Google ScholarDigital Library
- Leia C Shum, Bulmaro A Valdés, and HF Machiel Van der Loos. 2019. Determining the accuracy of oculus touch controllers for motor rehabilitation applications using quantifiable upper limb kinematics: Validation study. JMIR Biomedical Engineering 4, 1 (2019), e12291.Google ScholarCross Ref
- Olga Sorkine, Daniel Cohen-Or, Yaron Lipman, Marc Alexa, Christian Rössl, and H-P Seidel. 2004. Laplacian surface editing. In Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing. 175--184.Google ScholarDigital Library
- Jean-Marc Thiery, Pooran Memari, and Tamy Boubekeur. 2018. Mean value coordinates for quad cages in 3D. ACM Transactions on Graphics (Proc. SIGGRAPH 2018) (2018).Google ScholarDigital Library
- Jose Abel Ticona, Steeven Villa, Rafael Torchelsen, Anderson Maciel, and Luciana Nedel. 2019. Phys-sketch: Sketching 3D dynamic objects in immersive virtual reality. In Computer Graphics International Conference. Springer, 119--130.Google ScholarCross Ref
- Pier Paolo Valentini. 2018. Natural interface for interactive virtual assembly in augmented reality using leap motion controller. International Journal on Interactive Design and Manufacturing (IJIDeM) 12, 4 (2018), 1157--1165.Google ScholarCross Ref
- Pier Paolo Valentini and Eugenio Pezzuti. 2017. Accuracy in fingertip tracking using leap motion controller for interactive virtual applications. International Journal on Interactive Design and Manufacturing (IJIDeM) 11, 3 (2017), 641--650.Google ScholarCross Ref
- Aleš Vysockỳ, Stefan Grushko, Petr Oščádal, Tomáš Kot, Jan Babjak, Rudolf Jánoš, Marek Sukop, and Zdenko Bobovskỳ. 2020. Analysis of Precision and Stability of Hand Tracking with Leap Motion Sensor. Sensors 20, 15 (2020), 4088.Google ScholarCross Ref
- Rustin Webster and Joseph F Dues. 2017. System Usability Scale (SUS): Oculus Rift® DK2 and Samsung Gear VR®. In 2017 ASEE Annual Conference & Exposition.Google ScholarCross Ref
- Huiyue Wu, Weizhou Luo, Neng Pan, Shenghuan Nan, Yanyi Deng, Shengqian Fu, and Liuqingqing Yang. 2019. Understanding freehand gestures: a study of freehand gestural interaction for immersive VR shopping applications. Human-centric Computing and Information Sciences 9, 1 (2019), 1--26.Google ScholarDigital Library
- Xiaokun Wu, Daniel Finnegan, Eamonn O'Neill, and Yong-Liang Yang. 2018. Handmap: Robust hand pose estimation via intermediate dense guidance map supervision. In Proceedings of the European Conference on Computer Vision (ECCV). 237--253.Google ScholarDigital Library
- Jie Yang, Lin Gao, Yu-Kun Lai, Paul L Rosin, and Shihong Xia. 2018. Biharmonic deformation transfer with automatic key point selection. Graphical Models 98 (2018), 1--13.Google ScholarCross Ref
- Yui-Pan Yau, Lik Hang Lee, Zheng Li, Tristan Braud, Yi-Hsuan Ho, and Pan Hui. 2020. How Subtle Can It Get? A Trimodal Study of Ring-Sized Interfaces for One-Handed Drone Control. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 2, Article 63 (June 2020), 29 pages. Google ScholarDigital Library
- Zongliang Zhanga, Jonathan Lia, Xin Lic, Yangbin Lina, Shanxin Zhanga, and Cheng Wanga. 2016. A fast method for measuring the similarity between 3d model and 3d point cloud. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 1 (2016).Google Scholar
Index Terms
- 3DeformR: freehand 3D model editing in virtual environments considering head movements on mobile headsets
Recommendations
The Gesture Authoring Space: Authoring Customised Hand Gestures for Grasping Virtual Objects in Immersive Virtual Environments
MuC '22: Proceedings of Mensch und Computer 2022Natural user interfaces are on the rise. Manufacturers for Augmented, Virtual, and Mixed Reality head mounted displays are increasingly integrating new sensors into their consumer grade products, allowing gesture recognition without additional hardware. ...
MagicalHands: Mid-Air Hand Gestures for Animating in VR
UIST '19: Proceedings of the 32nd Annual ACM Symposium on User Interface Software and TechnologyWe explore the use of hand gestures for authoring animations in virtual reality (VR). We first perform a gesture elicitation study to understand user preferences for a spatiotemporal, bare-handed interaction system in VR. Specifically, we focus on ...
VRGestures: Controller and Hand Gesture Datasets for Virtual Reality
Advances in Computer GraphicsAbstractGesture Recognition is attracting increasingly more attention over the years and has been adopted in main applications in the real world and the Virtual one. New generation Virtual Reality (VR) headsets like the Meta Quest 2 support hand tracking ...
Comments