skip to main content
10.1145/3411764.3445302acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

HandPainter - 3D Sketching in VR with Hand-based Physical Proxy

Published: 07 May 2021 Publication History

Abstract

3D sketching in virtual reality (VR) enables users to create 3D virtual objects intuitively and immersively. However, previous studies showed that mid-air drawing may lead to inaccurate sketches. To address this issue, we propose to use one hand as a canvas proxy and the index finger of the other hand as a 3D pen. To this end, we first perform a formative study to compare two-handed interaction with tablet-pen interaction for VR sketching. Based on the findings of this study, we design HandPainter, a VR sketching system which focuses on the direct use of two hands for 3D sketching without requesting any tablet, pen, or VR controller. Our implementation is based on a pair of VR gloves, which provide hand tracking and gesture capture. We devise a set of intuitive gestures to control various functionalities required during 3D sketching, such as canvas panning and drawing positioning. We show the effectiveness of HandPainter by presenting a number of sketching results and discussing the outcomes of a user study-based comparison with mid-air drawing and tablet-based sketching tools.

Supplementary Material

Supplementary Materials (3411764.3445302_supplementalmaterials.zip)
MP4 File (3411764.3445302_videofigure.mp4)
Supplemental video

References

[1]
Anand Agarawala and Ravin Balakrishnan. 2006. Keepin’it real: pushing the desktop metaphor with physics, piles and the pen. In Proceedings of the SIGCHI conference on Human Factors in computing systems. ACM, Montréal Québec, Canada, 1283–1292.
[2]
Rahul Arora, Rubaiat Habib Kazi, Tovi Grossman, George Fitzmaurice, and Karan Singh. 2018. Symbiosissketch: Combining 2d & 3d sketching for designing detailed 3d objects in situ. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montreal, 185.
[3]
Rahul Arora, Rubaiat Habib Kazi, Fraser Anderson, Tovi Grossman, Karan Singh, and George W Fitzmaurice. 2017. Experimental Evaluation of Sketching on Surfaces in VR. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Vol. 17. CHI, Denver Colorado, USA, 5643–5654.
[4]
Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies 4 (2009), 114–123.
[5]
Mayra Donaji Barrera Machuca, Wolfgang Stuerzlinger, and Paul Asente. 2019. The Effect of Spatial Ability on Immersive 3D Drawing. In Proceedings of the 2019 on Creativity and Cognition. ACM, San Diego CA USA, 173–186.
[6]
Wendy Bolier. 2017. Drawing in a virtual 3D space-Introducing VR drawing in elementary school art education. Master’s thesis. Utrecht University.
[7]
John Brooke 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4–7.
[8]
Jeff Butterworth, Andrew Davidson, Stephen Hench, and Marc T Olano. 1992. 3DM: A three dimensional modeler using a head-mounted display. In Proceedings of the 1992 symposium on Interactive 3D graphics. ACM, Cambridge Massachusetts, 135–138.
[9]
Bruno R De Araùjo, Géry Casiez, and Joaquim A Jorge. 2012. Mockup builder: direct 3D modeling on and above the surface in a continuous interaction space. In Proceedings of Graphics Interface 2012. ACM, Toronto Ontario, Canada, 173–180.
[10]
Michael F Deering. 1995. HoloSketch: a virtual reality sketching/animation tool. ACM Transactions on Computer-Human Interaction (TOCHI) 2, 3(1995), 220–238.
[11]
Ellen Yi-Luen Do. 2001. VR Sketchpad: Create Instant 3D Worlds by Sketching on a Transparent Window. In Computer Aided Architectural Design Futures 2001, Bauke de Vries, Jos van Leeuwen, and Henri Achten (Eds.). Springer Netherlands, Dordrecht, 161–172.
[12]
David H Douglas and Thomas K Peucker. 1973. Algorithms for the reduction of the number of points required to represent a digitized line or its caricature. Cartographica: the international journal for geographic information and geovisualization 10, 2 (1973), 112–122.
[13]
Tobias Drey, Jan Gugenheimer, Julian Karlbauer, Maximilian Milo, and Enrico Rukzio. 2020. VRSketchIn: Exploring the Design Space of Pen and Tablet Interaction for 3D Sketching in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu, 1–14.
[14]
John J Dudley, Hendrik Schuff, and Per Ola Kristensson. 2018. Bare-handed 3D drawing in augmented reality. In Proceedings of the 2018 Designing Interactive Systems Conference. ACM, Hong Kong China, 241–252.
[15]
Hesham Elsayed, Mayra Donaji Barrera Machuca, Christian Schaarschmidt, Karola Marky, Florian Müller, Jan Riemann, Andrii Matviienko, Martin Schmitz, Martin Weigel, and Max Mühlhäuser. 2020. VRSketchPen: Unconstrained Haptic Assistance for Sketching in Virtual 3D Environments. In 26th ACM Symposium on Virtual Reality Software and Technology. ACM, Virtual Event Canada, 1–11.
[16]
Julien Epps, Serge Lichman, and Mike Wu. 2006. A study of hand shape use in tabletop gesture interaction. In CHI’06 extended abstracts on human factors in computing systems. ACM, Montréal Québec, Canada, 748–753.
[17]
Mark Fuge, Mehmet Ersin Yumer, Gunay Orbay, and Levent Burak Kara. 2012. Conceptual design and modification of freeform surfaces using dual shape representations in augmented reality environments. Computer-Aided Design 44, 10 (2012), 1020–1032.
[18]
Liuhao Ge, Yujun Cai, Junwu Weng, and Junsong Yuan. 2018. Hand pointnet: 3d hand pose estimation using point sets. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Salt Lake City, UT, USA, 8417–8426.
[19]
Yves Guiard. 1987. Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model. Journal of motor behavior 19, 4 (1987), 486–517.
[20]
Sean Gustafson, Christian Holz, and Patrick Baudisch. 2011. Imaginary phone: learning imaginary interfaces by transferring spatial memory from a familiar device. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, Santa Barbara California USA, 283–292.
[21]
Kelly S. Hale and Kay M. Stanney. 2014. Handbook of Virtual Environments: Design, Implementation, and Applications (2nd ed.). CRC Press, Inc., Boca Raton, FL, USA.
[22]
Ken Hinckley, Michel Pahud, Hrvoje Benko, Pourang Irani, François Guimbretière, Marcel Gavriliu, Xiang’Anthony’ Chen, Fabrice Matulic, William Buxton, and Andrew Wilson. 2014. Sensing techniques for tablet+ stylus interaction. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, Honolulu Hawaii USA, 605–614.
[23]
Ken Hinckley, Koji Yatani, Michel Pahud, Nicole Coddington, Jenny Rodenhouse, Andy Wilson, Hrvoje Benko, and Bill Buxton. 2010. Pen+ touch= new tools. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, New York NY USA, 27–36.
[24]
Bret Jackson and Daniel F Keefe. 2016. Lift-off: Using reference imagery and freehand sketching to create 3d models in vr. IEEE transactions on visualization and computer graphics 22, 4(2016), 1442–1451.
[25]
Roy S Kalawsky. 1999. VRUSE—A computerised diagnostic tool: for usability evaluation of virtual/synthetic environment systems. Applied ergonomics 30, 1 (1999), 11–25.
[26]
Shaun K Kane, Jeffrey P Bigham, and Jacob O Wobbrock. 2008. Slide rule: making mobile touch screens accessible to blind people using multi-touch interaction techniques. In Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility. ACM, Halifax Nova Scotia Canada, 73–80.
[27]
Daniel Keefe, Robert Zeleznik, and David Laidlaw. 2007. Drawing on air: Input techniques for controlled 3D line illustration. IEEE Transactions on Visualization and Computer Graphics 13, 5(2007), 1067–1081.
[28]
Daniel F Keefe, Daniel Acevedo Feliz, Tomer Moscovich, David H Laidlaw, and Joseph J LaViola Jr. 2001. CavePainting: a fully immersive 3D artistic medium and interactive experience. In Proceedings of the 2001 symposium on Interactive 3D graphics. Citeseer, ACM, Chapel Hill, NC, USA, 85–93.
[29]
David Kim, Otmar Hilliges, Shahram Izadi, Alex D Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, Cambridge Massachusetts USA, 167–176.
[30]
Yongkwan Kim, Sang-Gyun An, Joon Hyub Lee, and Seok-Hyung Bae. 2018. Agile 3D sketching with air scaffolding. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montréal, Canada, 238.
[31]
Yongkwan Kim and Seok-Hyung Bae. 2016. SketchingWithHands: 3D sketching handheld products with first-person hand posture. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, Tokyo Japan, 797–808.
[32]
Kin Chung Kwan and Hongbo Fu. 2019. Mobi3DSketch: 3D Sketching in Mobile AR. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow Scotland UK, 176.
[33]
Mayra D Barrera Machuca, Wolfgang Stuerzlinger, and Paul Asente. 2019. Smart3DGuides: Making Unconstrained Immersive 3D Drawing More Accurate. In 25th ACM Symposium on Virtual Reality Software and Technology. ACM, Parramatta NSW Australia, 1–13.
[34]
Keigo Matsumoto, Yuki Ban, Takuji Narumi, Yohei Yanase, Tomohiro Tanikawa, and Michitaka Hirose. 2016. Unlimited corridor: redirected walking techniques using visuo haptic interaction. In ACM SIGGRAPH 2016 Emerging Technologies. ACM, Anaheim California USA, 1–2.
[35]
Franziska Mueller, Micah Davis, Florian Bernard, Oleksandr Sotnychenko, Mickeal Verschoor, Miguel A Otaduy, Dan Casas, and Christian Theobalt. 2019. Real-time pose and shape reconstruction of two interacting hands with a single depth camera. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1–13.
[36]
Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt. 2017. Real-time hand tracking under occlusion from an egocentric rgb-d sensor. In Proceedings of the IEEE International Conference on Computer Vision Workshops. IEEE, Venice, Italy, 1284–1293.
[37]
Andrew Nealen. 2007. Interfaces and Algorithms for the Creation, Modification, and Optimization of Surface Meshes. Doctoral Thesis. Technische Universität Berlin, Fakultät IV - Elektrotechnik und Informatik, Berlin. https://doi.org/10.14279/depositonce-1734
[38]
Huaishu Peng, Jimmy Briggs, Cheng-Yao Wang, Kevin Guo, Joseph Kider, Stefanie Mueller, Patrick Baudisch, and François Guimbretière. 2018. RoMA: Interactive fabrication with augmented reality and a robotic 3D printer. In Proceedings of the 2018 CHI conference on human factors in computing systems. ACM, Montreal QC Canada, 1–12.
[39]
Anna Pereira, Tevis Miller, Yi-Min Huang, Dan Odell, and David Rempel. 2013. Holding a tablet computer with one hand: effect of tablet design features on biomechanics and subjective usability among users with small hands. Ergonomics 56, 9 (2013), 1363–1375.
[40]
Ken Pfeuffer, Benedikt Mayer, Diako Mardanbegi, and Hans Gellersen. 2017. Gaze+ pinch interaction in virtual reality. In Proceedings of the 5th Symposium on Spatial User Interaction. ACM, Brighton United Kingdom, 99–108.
[41]
Dilip K Prasad, Maylor KH Leung, Chai Quek, and Siu-Yeung Cho. 2012. A novel framework for making dominant point detection methods non-parametric. Image and Vision Computing 30, 11 (2012), 843–859.
[42]
Chen Qian, Xiao Sun, Yichen Wei, Xiaoou Tang, and Jian Sun. 2014. Realtime and robust hand tracking from depth. In Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, Columbus, OH, USA, 1106–1113.
[43]
Karthik Ramani 2015. A gesture-free geometric approach for mid-air expression of design intent in 3D virtual pottery. Computer-Aided Design 69(2015), 11–24.
[44]
Steven Schkolne, Michael Pruett, and Peter Schröder. 2001. Surface drawing: creating organic 3D shapes with the hand and tangible tools. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, Seattle Washington USA, 261–268.
[45]
Jotaro Shigeyama, Takeru Hashimoto, Shigeo Yoshida, Takuji Narumi, Tomohiro Tanikawa, and Michitaka Hirose. 2019. Transcalibur: A weight shifting virtual reality controller for 2d shape rendering based on computational perception model. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow Scotland UK, 1–11.
[46]
Mike Sinclair, Eyal Ofek, Mar Gonzalez-Franco, and Christian Holz. 2019. Capstancrunch: A haptic vr controller with user-supplied force feedback. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. ACM, New Orleans, Louisiana, USA, 815–829.
[47]
Hemant Bhaskar Surale, Aakar Gupta, Mark Hancock, and Daniel Vogel. 2019. TabletinVR: Exploring the design space for using a multi-touch tablet in virtual reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow Scotland UK, 1–13.
[48]
Ying-Chao Tung, Chun-Yen Hsu, Han-Yu Wang, Silvia Chyou, Jhe-Wei Lin, Pei-Jung Wu, Andries Valstar, and Mike Y Chen. 2015. User-defined game input for smart glasses in public space. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, Seoul Republic of Korea, 3327–3336.
[49]
Gerold Wesche and Marc Droske. 2000. Conceptual free-form styling on the responsive workbench. In Proceedings of the ACM symposium on Virtual reality software and technology. ACM, Seoul Korea, 83–91.
[50]
Gerold Wesche and Hans-Peter Seidel. 2001. FreeDrawer: a free-form sketching system on the responsive workbench. In Proceedings of the ACM symposium on Virtual reality software and technology. ACM, Baniff Alberta Canada, 167–174.
[51]
Zheer Xu, Pui Chung Wong, Jun Gong, Te-Yen Wu, Aditya Shekhar Nittala, Xiaojun Bi, Jürgen Steimle, Hongbo Fu, Kening Zhu, and Xing-Dong Yang. 2019. TipText: Eyes-Free Text Entry on a Fingertip Keyboard. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. ACM, New Orleans LA USA, 883–899.
[52]
H. Ye, K. C. Kwan, and H. Fu. 2021. 3D Curve Creation on and around Physical Objects with Mobile AR. IEEE Transactions on Visualization and Computer Graphics 1 (2021), 1–1e. https://doi.org/10.1109/TVCG.2020.3049006
[53]
Ya-Ting Yue, Xiaolong Zhang, Yongliang Yang, Gang Ren, Yi-King Choi, and Wenping Wang. 2017. Wiredraw: 3d wire sculpturing guided with mixed reality. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, Denver Colorado USA, 3693–3704.
[54]
André Zenner and Antonio Krüger. 2019. Drag: on: A virtual reality controller providing haptic feedback based on drag and weight shift. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, Glasgow Scotland UK, 1–12.

Cited By

View all
  • (2024)Interactive Modelling in Augmented Reality with Subdivision Surfaces and Advanced User Gesture RecognitionApplied Sciences10.3390/app14241187314:24(11873)Online publication date: 19-Dec-2024
  • (2024)PhoneCanvas: 3D Sketching System Using a Depth Camera-Equipped Smartphone as a CanvasProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682078(1-12)Online publication date: 7-Oct-2024
  • (2024)Bimanual Interactions for Surfacing Curve Networks in VRExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650988(1-7)Online publication date: 11-May-2024
  • Show More Cited By

Index Terms

  1. HandPainter - 3D Sketching in VR with Hand-based Physical Proxy
          Index terms have been assigned to the content through auto-classification.

          Recommendations

          Comments

          Information & Contributors

          Information

          Published In

          cover image ACM Conferences
          CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
          May 2021
          10862 pages
          ISBN:9781450380966
          DOI:10.1145/3411764
          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

          Sponsors

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          Published: 07 May 2021

          Permissions

          Request permissions for this article.

          Check for updates

          Author Tags

          1. 3D sketching
          2. VR
          3. hand-based interaction

          Qualifiers

          • Research-article
          • Research
          • Refereed limited

          Funding Sources

          Conference

          CHI '21
          Sponsor:

          Acceptance Rates

          Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

          Upcoming Conference

          CHI 2025
          ACM CHI Conference on Human Factors in Computing Systems
          April 26 - May 1, 2025
          Yokohama , Japan

          Contributors

          Other Metrics

          Bibliometrics & Citations

          Bibliometrics

          Article Metrics

          • Downloads (Last 12 months)307
          • Downloads (Last 6 weeks)38
          Reflects downloads up to 27 Feb 2025

          Other Metrics

          Citations

          Cited By

          View all
          • (2024)Interactive Modelling in Augmented Reality with Subdivision Surfaces and Advanced User Gesture RecognitionApplied Sciences10.3390/app14241187314:24(11873)Online publication date: 19-Dec-2024
          • (2024)PhoneCanvas: 3D Sketching System Using a Depth Camera-Equipped Smartphone as a CanvasProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682078(1-12)Online publication date: 7-Oct-2024
          • (2024)Bimanual Interactions for Surfacing Curve Networks in VRExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650988(1-7)Online publication date: 11-May-2024
          • (2024)An Artists' Perspectives on Natural Interactions for Virtual Reality 3D SketchingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642758(1-20)Online publication date: 11-May-2024
          • (2024)DeepTreeSketch: Neural Graph Prediction for Faithful 3D Tree Modeling from SketchesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642125(1-19)Online publication date: 11-May-2024
          • (2024)EyeGuide & EyeConGuide: Gaze-based Visual Guides to Improve 3D Sketching SystemsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3641947(1-14)Online publication date: 11-May-2024
          • (2024)VRTree: Example‐Based 3D Interactive Tree Modeling in Virtual RealityComputer Graphics Forum10.1111/cgf.1525443:7Online publication date: 7-Nov-2024
          • (2024)Toward More Comprehensive Evaluations of 3D Immersive Sketching, Drawing, and PaintingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.327629130:8(4648-4664)Online publication date: Aug-2024
          • (2024)ARMedicalSketch: Exploring 3D Sketching for Medical Image Using True 2D-3D Interlinked Visualization and InteractionIEEE Transactions on Human-Machine Systems10.1109/THMS.2024.343273554:5(589-598)Online publication date: Oct-2024
          • (2024)Studying How Prompt-Generated 3D Models Affect the Creation Process of Mixed Reality Applications2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct64951.2024.00196(654-655)Online publication date: 21-Oct-2024
          • Show More Cited By

          View Options

          Login options

          View options

          PDF

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format.

          HTML Format

          Figures

          Tables

          Media

          Share

          Share

          Share this Publication link

          Share on social media