Fabrice Matulic
Daniel Vogel
ABSTRACT
World building or terrain modelling is an essential task when designing games, natural simulations or artistic creations involving virtual 3D landscapes. To support this task, we propose a virtual reality (VR) system based on a pen and touch tablet used in a sitting position (desktop VR) such that both hands are free to interact in an asymmetric way (pen hand + other bare hand). We present and compare several techniques to perform navigation, sculpting and menu operations using the two hands, which interact on and above the tablet surface, i.e. using the pen, touch and mid-air input spaces. A qualitative evaluation with 16 participants confirms the entertaining nature and practical benefits of our system. The study further underlines the complementarity of the different modalities and identifies the promising—and as of yet underexplored—combination of bimanual touch + pen + mid-air interaction in desktop VR.
Supplemental Material
- Bruno R De Araùjo, Géry Casiez, and Joaquim A Jorge. 2012. Mockup Builder: Direct 3D Modeling on and above the Surface in a Continuous Interaction Space. In Proceedings of Graphics Interface 2012 (GI ’12), 173–180.Google Scholar
- Rahul Arora, Rubaiat Habib Kazi, Tovi Grossman, George Fitzmaurice, and Karan Singh. 2018. SymbiosisSketch: Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–15. Retrieved from https://doi.org/10.1145/3173574.3173759Google ScholarDigital Library
- Peter Brandl, Clifton Forlines, Daniel Wigdor, Michael Haller, and Chia Shen. 2008. Combining and measuring the benefits of bimanual pen and direct-touch interaction on horizontal interfaces. Proceedings of the working conference on Advanced visual interfaces, 154–161. https://doi.org/10.1145/1385569.1385595Google ScholarDigital Library
- Theocharis Chatzis, Andreas Stergioulas, Dimitrios Konstantinidis, Kosmas Dimitropoulos, and Petros Daras. 2020. A Comprehensive Study on Deep Learning-Based 3D Hand Pose Estimation Methods. Applied Sciences 10, 19. https://doi.org/10.3390/app10196850Google ScholarCross Ref
- Noah Coomer, Sadler Bullard, William Clinton, and Betsy Williams-Sanders. 2018. Evaluating the Effects of Four VR Locomotion Methods: Joystick, Arm-Cycling, Point-Tugging, and Teleporting. In Proceedings of the 15th ACM Symposium on Applied Perception (SAP ’18). https://doi.org/10.1145/3225153.3225175Google ScholarDigital Library
- Tobias Drey, Jan Gugenheimer, Julian Karlbauer, Maximilian Milo, and Enrico Rukzio. 2020. VRSketchIn: Exploring the Design Space of Pen and Tablet Interaction for 3D Sketching in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376628Google ScholarDigital Library
- Sevinc Eroglu, Frederic Stefan, Alain Chevalier, Daniel Roettger, Daniel Zielasko, Torsten W Kuhlen, and Benjamin Weyers. 2021. Design and Evaluation of a Free-Hand VR-based Authoring Environment for Automated Vehicle Testing. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), 1–10. https://doi.org/10.1109/VR50410.2021.00020Google Scholar
- Thomas Fröhlich, Dmitry Alexandrovsky, Timo Stabbert, Tanja Döring, and Rainer Malaka. 2018. VRBox: A Virtual Reality Augmented Sandbox for Immersive Playfulness, Creativity and Exploration. In Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY ’18), 153–162. https://doi.org/10.1145/3242671.3242697Google ScholarDigital Library
- Travis Gesslein, Verena Biener, Philipp Gagel, Daniel Schneider, Per Ola Kristensson, Eyal Ofek, Michel Pahud, and Jens Grubert. 2020. Pen-based Interaction with Spreadsheets in Mobile Virtual Reality. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 361–373. https://doi.org/10.1109/ISMAR50242.2020.00063Google Scholar
- Yves Guiard. 1987. Asymmetric division of labor in human skilled bimanual action: the kinematic chain as a model. Journal of Motor Behavior 19, 4: 486–517.Google ScholarCross Ref
- François Guimbretière and Chau Nguyen. 2012. Bimanual Marking Menu for near Surface Interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 825–828. Retrieved from https://doi.org/10.1145/2207676.2208521Google ScholarDigital Library
- Ken Hinckley, Koji Yatani, Michel Pahud, Nicole Coddington, Jenny Rodenhouse, Andy Wilson, Hrvoje Benko, and Bill Buxton. 2010. Pen + touch = new tools. Proceedings of the 23nd annual ACM symposium on User interface software and technology, 27–36. https://doi.org/10.1145/1866029.1866036Google ScholarDigital Library
- Kenrick Kin, Björn Hartmann, and Maneesh Agrawala. 2011. Two-Handed Marking Menus for Multitouch Devices. ACM Trans. Comput.-Hum. Interact. 18, 3. https://doi.org/10.1145/1993060.1993066Google ScholarDigital Library
- Josen Daniel O De Leon, Romelio P Tavas, Rodolfo A Aranzanso, and Rowel O Atienza. 2016. Genesys: A Virtual Reality scene builder. In 2016 IEEE Region 10 Conference (TENCON), 3708–3711. https://doi.org/10.1109/TENCON.2016.7848751Google ScholarCross Ref
- Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang. 2021. Semi-supervised 3d hand-object poses estimation with interactions in time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14687–14697.Google ScholarCross Ref
- Fabrice Matulic and Moira Norrie. 2013. Pen and Touch Gestural Environment for Document Editing on Interactive Tabletops. Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces, 41–50.Google ScholarDigital Library
- Manuel Meier, Paul Streli, Andreas Fender, and Christian Holz. 2021. TaplD: Rapid Touch Interaction in Virtual Reality using Wearable Sensing. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), 519–528. https://doi.org/10.1109/VR50410.2021.00076Google Scholar
- Meta. 2021. Horizon Worlds. Retrieved from https://www.oculus.com/facebook-horizon/Google Scholar
- Daniel L Odell, Richard C Davis, Andrew Smith, and Paul K Wright. 2004. Toolglasses, Marking Menus, and Hotkeys: A Comparison of One and Two-Handed Command Selection Techniques. In Proceedings of Graphics Interface 2004 (GI ’04), 17–24.Google Scholar
- Duc-Minh Pham and Wolfgang Stuerzlinger. 2019. Is the Pen Mightier than the Controller? A Comparison of Input Devices for Selection in Virtual and Augmented Reality. In 25th ACM Symposium on Virtual Reality Software and Technology (VRST ’19). https://doi.org/10.1145/3359996.3364264Google ScholarDigital Library
- Hugo Romat, Andreas Fender, Manuel Meier, and Christian Holz. 2021. Flashpen: A High-Fidelity and High-Precision Multi-Surface Pen for Virtual Reality. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), 306–315. https://doi.org/10.1109/VR50410.2021.00053Google Scholar
- Hemant Bhaskar Surale, Aakar Gupta, Mark Hancock, and Daniel Vogel. 2019. TabletInVR: Exploring the Design Space for Using a Multi-Touch Tablet in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–13. Retrieved from https://doi.org/10.1145/3290605.3300243Google ScholarDigital Library
- Stephan P Swinnen. 2002. Intermanual coordination: From behavioural principles to neural-network interactions. Nature Reviews Neuroscience 3, 5: 348–359. https://doi.org/10.1038/nrn807Google ScholarCross Ref
- Daniel Vogel, Matthew Cudmore, Géry Casiez, Ravin Balakrishnan, and Liam Keliher. 2009. Hand Occlusion with Tablet-Sized Direct Pen Input. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 557–566. Retrieved from https://doi.org/10.1145/1518701.1518787Google ScholarDigital Library
- Jia Wang, Owen Leach, and Robert W Lindeman. 2013. DIY World Builder: An immersive level-editing system. In 2013 IEEE Symposium on 3D User Interfaces (3DUI), 195–196. https://doi.org/10.1109/3DUI.2013.6550245Google ScholarCross Ref
- Mike Wu, Shen Chia, Kathy Ryall, Clifton Forlines, and Ravin Balakrishnan. 2006. Gesture registration, relaxation, and reuse for multi-point direct-touch surfaces. First IEEE International Workshop on Horizontal Interactive Human-Computer Systems, 185–192. https://doi.org/10.1109/tabletop.2006.19Google ScholarDigital Library
- Robert Zeleznik, Andrew Bragdon, Ferdi Adeputra, and Hsu-Sheng Ko. 2010. Hands-on math: a page-based multi-touch and pen desktop for technical work and problem solving. Proceedings of the 23nd annual ACM symposium on User interface software and technology, 17–26. https://doi.org/10.1145/1866029.1866035Google ScholarDigital Library
- Unity. https://unity3d.comGoogle Scholar
- Unreal Engine VR Mode. https://docs.unrealengine.com/4.27/en-US/BuildingWorlds/VRModeGoogle Scholar
- Minecraft in VR. https://www.minecraft.net/en-us/vrGoogle Scholar
Recommendations
Pen + Mid-Air: An Exploration of Mid-Air Gestures to Complement Pen Input on Tablets
NordiCHI '16: Proceedings of the 9th Nordic Conference on Human-Computer InteractionIn this paper, we report on a series of studies, exploring the potential of pen and mid-air input on tablets. We describe a field study with an early prototype of a drawing application and follow-up inquiries, such as the development and comparison of ...
Eliciting Mid-Air Gestures for Wall-Display Interaction
NordiCHI '16: Proceedings of the 9th Nordic Conference on Human-Computer InteractionFreehand mid-air gestures are a promising input method for interacting with wall displays. However, work on mid-air gestures for wall-display interaction has mainly explored what is technically possible, which might not result in gestures that users ...
Towards document engineering on pen and touch-operated interactive tabletops
UIST Adjunct Proceedings '12: Adjunct proceedings of the 25th annual ACM symposium on User interface software and technologyTouch interfaces have now become mainstream thanks to modern smartphones and tablets. However, there are still very few "productivity" applications, i.e. tools that support mundane but essential work, especially for large interactive surfaces such as ...
Comments