abstract

Using Hand Tracking and Voice Commands to Physically Align Virtual Surfaces in AR for Handwriting and Sketching with HoloLens 2

Authors:

Florian Kern,

Thore Keser,

Florian Niebling,

Marc Erich LatoschikAuthors Info & Claims

VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

Article No.: 103, Pages 1 - 3

https://doi.org/10.1145/3489849.3489940

Published: 08 December 2021 Publication History

Get Access

Abstract

In this paper, we adapt an existing VR framework for handwriting and sketching on physically aligned virtual surfaces to AR environments using the Microsoft HoloLens 2. We demonstrate a multimodal input metaphor to control the framework’s calibration features using hand tracking and voice commands. Our technical evaluation of fingertip/surface accuracy and precision on physical tables and walls is in line with existing measurements on comparable hardware, albeit considerably lower compared to previous work using controller-based VR devices. We discuss design considerations and the benefits of our unified input metaphor suitable for controller tracking and hand tracking systems. We encourage extensions and replication by providing a publicly available reference implementation (https://go.uniwue.de/hci-otss-hololens).

References

[1]

Pietro Cipresso, Irene Alice Chicchi Giglioli, Mariano Alcañiz Raya, and Giuseppe Riva. 2018. The Past, Present, and Future of Virtual and Augmented Reality Research: A Network and Cluster Analysis of the Literature. Frontiers in Psychology 9 (Nov. 2018), 2086. https://doi.org/10.3389/fpsyg.2018.02086

Crossref

Google Scholar

[2]

Yann Glémarec, Jean-Luc Lugrin, Anne-Gwenn Bosser, Aryana Collins Jackson, Cédric Buche, and Marc Erich Latoschik. 2021. Indifferent or Enthusiastic? Virtual Audiences Animation and Perception in Virtual Reality. Frontiers in Virtual Reality 2 (June 2021), 666232. https://doi.org/10.3389/frvir.2021.666232

Crossref

Google Scholar

[3]

Tyler A. Jost, Bradley Nelson, and Jonathan Rylander. 2019. Quantitative analysis of the Oculus Rift S in controlled movement. Disability and Rehabilitation: Assistive Technology (Nov. 2019), 1–5. https://doi.org/10.1080/17483107.2019.1688398

Crossref

Google Scholar

[4]

Florian Kern. 2021. OTSS - Off-The-Shelf Stylus. https://go.uniwue.de/hci-otss

Google Scholar

[5]

Florian Kern, Peter Kullmann, Elisabeth Ganal, Kristof Korwisi, René Stingl, Florian Niebling, and Marc Erich Latoschik. 2021. Off-The-Shelf Stylus: Using XR Devices for Handwriting and Sketching on Physically Aligned Virtual Surfaces. Frontiers in Virtual Reality 2 (June 2021), 684498. https://doi.org/10.3389/frvir.2021.684498

Crossref

Google Scholar

[6]

Marc Erich Latoschik, Florian Kern, Jan-Philipp Stauffert, Andrea Bartl, Mario Botsch, and Jean-Luc Lugrin. 2019. Not Alone Here?! Scalability and User Experience of Embodied Ambient Crowds in Distributed Social Virtual Reality. IEEE Transactions on Visualization and Computer Graphics 25, 5 (May 2019), 2134–2144. https://doi.org/10.1109/TVCG.2019.2899250

Crossref

Google Scholar

[7]

Logitech. 2021. Logitech VR Ink Pilot Edition. https://www.logitech.com/en-roeu/promo/vr-ink.html

Google Scholar

[8]

Diederick C. Niehorster, Li Li, and Markus Lappe. 2017. The Accuracy and Precision of Position and Orientation Tracking in the HTC Vive Virtual Reality System for Scientific Research. i-Perception 8, 3 (June 2017), 204166951770820. https://doi.org/10.1177/2041669517708205

Crossref

Google Scholar

[9]

Leia C Shum, Bulmaro A Valdés, and HF Machiel Van der Loos. 2019. Determining the Accuracy of Oculus Touch Controllers for Motor Rehabilitation Applications Using Quantifiable Upper Limb Kinematics: Validation Study. JMIR Biomedical Engineering 4, 1 (June 2019), e12291. https://doi.org/10.2196/12291

Crossref

Google Scholar

[10]

Vuplex. 2021. Vuplex 3D WebView. https://developer.vuplex.com/

Google Scholar

[11]

Robert Xiao, Julia Schwarz, Nick Throm, Andrew D. Wilson, and Hrvoje Benko. 2018. MRTouch: Adding Touch Input to Head-Mounted Mixed Reality. IEEE Transactions on Visualization and Computer Graphics 24, 4 (April 2018), 1653–1660. https://doi.org/10.1109/TVCG.2018.2794222

Digital Library

Google Scholar

Cited By

View all

Kern FTschanter JLatoschik M(2024)Handwriting for Text Input and the Impact of XR Displays, Surface Alignments, and Sentence ComplexitiesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337212430:5(2357-2367)Online publication date: 5-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372124
Bondarenko VZhang JNguyen GFitzek F(2024)A Universal Method for Performance Assessment of Meta Quest XR Devices2024 IEEE Gaming, Entertainment, and Media Conference (GEM)10.1109/GEM61861.2024.10585428(1-6)Online publication date: 5-Jun-2024
https://doi.org/10.1109/GEM61861.2024.10585428
Kern FTschanter JLatoschik M(2023)Virtual-to-Physical Surface Alignment and Refinement Techniques for Handwriting, Sketching, and Selection in XR2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW58643.2023.00109(502-506)Online publication date: Mar-2023
https://doi.org/10.1109/VRW58643.2023.00109
Show More Cited By

Recommendations

3D Printing an Accessory Dock for XR Controllers and its Exemplary Use as XR Stylus
VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

This article introduces the accessory dock, a 3D printed multi-purpose extension for consumer-grade XR controllers that enables flexible mounting of self-made and commercial accessories. The uniform design of our concept opens new opportunities for XR ...
EyeSeeX: Visualization of Out-of-View Objects on Small Field-of-View Augmented and Virtual Reality Devices
PerDis '18: Proceedings of the 7th ACM International Symposium on Pervasive Displays

Recent advances in Virtual and Augmented Reality technology enable a variety of new applications (e.g., multi-player games in real environments). However, current devices suffer from having small fields of view, making the process of locating spatially ...
Exhibition approach using an AR and VR pillar
SA '17: SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications

This demonstration presents a development of an Augmented Reality (AR) and Virtual Reality (AR) pillar, a novel approach for showing AR and VR content in a public setting. A pillar in a public exhibition venue was converted to a four-sided AR and VR ...

Comments

Information & Contributors

Information

Published In

VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

December 2021

563 pages

ISBN:9781450390927

DOI:10.1145/3489849

Editors:
Yuichi Itoh
Aoyama Gakuin University, Japan
,
Kazuki Takashima
Tohoku University, Japan
,
Parinya Punpongsanon
Osaka University, Japan
,
Misha Sra
University of California, Santa Barbara, USA
,
Kazuyuki Fujita
Tohoku University, Japan
,
Shigeo Yoshida
The University of Tokyo, Japan
,
Tham Piumsomboon
University of Canterbury, New Zealand

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 December 2021

Check for updates

Badges

Best Poster

Author Tags

Qualifiers

Abstract
Research
Refereed limited

Funding Sources

Deutsche Forschungsgemeinschaft
German Federal Ministry of Education and Research

Conference

VRST '21

Sponsor:

VRST '21: 27th ACM Symposium on Virtual Reality Software and Technology

December 8 - 10, 2021

Osaka, Japan

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
423
Total Downloads

Downloads (Last 12 months)79
Downloads (Last 6 weeks)5

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Kern FTschanter JLatoschik M(2024)Handwriting for Text Input and the Impact of XR Displays, Surface Alignments, and Sentence ComplexitiesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337212430:5(2357-2367)Online publication date: 5-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372124
Bondarenko VZhang JNguyen GFitzek F(2024)A Universal Method for Performance Assessment of Meta Quest XR Devices2024 IEEE Gaming, Entertainment, and Media Conference (GEM)10.1109/GEM61861.2024.10585428(1-6)Online publication date: 5-Jun-2024
https://doi.org/10.1109/GEM61861.2024.10585428
Kern FTschanter JLatoschik M(2023)Virtual-to-Physical Surface Alignment and Refinement Techniques for Handwriting, Sketching, and Selection in XR2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW58643.2023.00109(502-506)Online publication date: Mar-2023
https://doi.org/10.1109/VRW58643.2023.00109
Kern FNiebling FLatoschik M(2023)Text Input for Non-Stationary XR Workspaces: Investigating Tap and Word-Gesture Keyboards in Virtual and Augmented RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324709829:5(2658-2669)Online publication date: 1-May-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3247098
Choi SMohaisen MNyang DMohaisen D(2023)Revisiting the Deep Learning-Based Eavesdropping Attacks via Facial Dynamics from VR Motion SensorsInformation and Communications Security10.1007/978-981-99-7356-9_24(399-417)Online publication date: 20-Oct-2023
https://doi.org/10.1007/978-981-99-7356-9_24

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Cited By

Recommendations

3D Printing an Accessory Dock for XR Controllers and its Exemplary Use as XR Stylus

EyeSeeX: Visualization of Out-of-View Objects on Small Field-of-View Augmented and Virtual Reality Devices

Exhibition approach using an AR and VR pillar

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Badges

Author Tags

Qualifiers

Funding Sources

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Share

Share this Publication link

Share on social media

Affiliations