research-article

Tactile hand motion and pose guidance for 3D interaction

Authors:

Alexander Marquardt,

Christina Trepkowski,

Andrea Schwandt,

André Hinkenjann,

Johannes Schöning,

Wolfgang StuerzlingerAuthors Info & Claims

VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology

Article No.: 3, Pages 1 - 10

https://doi.org/10.1145/3281505.3281526

Published: 28 November 2018 Publication History

Abstract

We present a novel forearm-and-glove tactile interface that can enhance 3D interaction by guiding hand motor planning and coordination. In particular, we aim to improve hand motion and pose actions related to selection and manipulation tasks. Through our user studies, we illustrate how tactile patterns can guide the user, by triggering hand pose and motion changes, for example to grasp (select) and manipulate (move) an object. We discuss the potential and limitations of the interface, and outline future work.

References

[1]

C. Afonso and S. Beckhaus. 2011. How to Not Hit a Virtual Wall: Aural Spatial Awareness for Collision Avoidance in Virtual Environments. In Proceedings of the 6th Audio Mostly Conference: A Conference on Interaction with Sound (AM '11). ACM, 101--108.

Digital Library

[2]

R. B. Ammons. 1956. Effects of Knowledge of Performance: A Survey and Tentative Theoretical Formulation. The Journal of General Psychology 54, 2 (1956), 279--299.

[3]

F. Argelaguet, A. Kulik, A. Kunert, C. Andujar, and B. Froehlich. 2011. See-through techniques for referential awareness in collaborative virtual reality. International Journal of Human-Computer Studies 69, 6 (2011), 387--400.

Digital Library

[4]

R. Bane and T. Hollerer. 2004. Interactive Tools for Virtual X-Ray Vision in Mobile Augmented Reality. In Proceedings of the 3rd IEEE/ACM International Symposium on Mixed and Augmented Reality (ISMAR '04). IEEE, 231--239.

Digital Library

[5]

K. Bark, P. Khanna, R. Irwin, P. Kapur, S. A. Jax, L. Buxbaum, and K. Kuchenbecker. 2011. Lessons in using vibrotactile feedback to guide fast arm motions. In World Haptics Conference (WHC), 2011 IEEE. IEEE, 355--360.

[6]

P. W. Battaglia, M. Di Luca, M. Ernst, P. R. Schrater, T. Machulla, and D. Kersten. 2010. Within- and Cross-Modal Distance Information Disambiguate Visual Size-Change Perception. PLOS Computational Biology 6, 3 (03 2010), 1--10.

[7]

S. Beckhaus, F. Ritter, and T. Strothotte. 2000. CubicalPath-dynamic potential fields for guided exploration in virtual environments. In Proceedings the Eighth Pacific Conference on Computer Graphics and Applications. 387--459.

Digital Library

[8]

H. Benko, C. Holz, M. Sinclair, and E. Ofek. 2016. NormalTouch and TextureTouch: High-fidelity 3D Haptic Shape Rendering on Handheld Virtual Reality Controllers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, 717--728.

Digital Library

[9]

J. Blake and H. B. Gurocak. 2009. Haptic Glove With MR Brakes for Virtual Reality. IEEE/ASME Transactions on Mechatronics 14, 5 (2009), 606--615.

[10]

A. Bloomfield and N. Badler. 2008. Virtual training via vibrotactile arrays. Presence: Teleoperators and Virtual Environments 17, 2 (2008), 103--120.

Digital Library

[11]

A. Bloomfield, Y. Deng, J. Wampler, P. Rondot, M. Harth, D. and McManus, and N. Badler. 2003. A taxonomy and comparison of haptic actions for disassembly tasks. In Virtual Reality, 2003. Proceedings. IEEE. IEEE, 225--231.

Digital Library

[12]

G. C. Burdea. 1996. Force and Touch Feedback for Virtual Reality. John Wiley & Sons, Inc.

Digital Library

[13]

C. Chen, Y. Chen, Y. Chung, and N. Yu. 2016. Motion Guidance Sleeve: Guiding the Forearm Rotation Through External Artificial Muscles. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, 3272--3276.

Digital Library

[14]

D. Drobny and J. O. Borchers. 2010. Learning basic dance choreographies with different augmented feedback modalities. In Proceedings of the 28th International Conference on Human Factors in Computing Systems, CHI 2010, Extended Abstracts Volume, 2010. 3793--3798.

Digital Library

[15]

M. Ernst and M. Banks. 2002. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 6870 (2002), 429.

[16]

P. Gallotti, A. Raposo, and L. Soares. 2011. v-Glove: A 3D Virtual Touch Interface. In 2011 XIII Symposium on Virtual Reality. 242--251.

Digital Library

[17]

U. Gollner, T. Bieling, and G. Joost. 2012. Mobile Lorm Glove: Introducing a Communication Device for Deaf-blind People. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction (TEI '12). ACM, 127--130.

Digital Library

[18]

P. Green and L. Wei-Haas. 1985. The rapid development of user interfaces: Experience with the Wizard of Oz method. In Proceedings of the Human Factors Society Annual Meeting, Vol. 29. SAGE Publications Sage CA: Los Angeles, CA, 470--474.

[19]

C. Hatzfeld and T.A. Kern. 2014. Engineering Haptic Devices: A Beginner's Guide. Springer London.

Digital Library

[20]

B. Holbert. 2007. Enhanced Targeting in a Haptic User Interface for the Physically Disabled Using a Force Feedback Mouse. Ph.D. Dissertation. Advisor(s) Huber, M. AAI3277666.

Digital Library

[21]

H. Jin, Q. Chen, Z. Chen, Y. Hu, and J. Zhang. 2016. Multi-LeapMotion sensor based demonstration for robotic refine tabletop object manipulation task. CAAI Transactions on Intelligence Technology 1, 1 (2016), 104 -- 113.

[22]

R. Johansson and R. Flanagan. 2009. Coding and use of tactile signals from the fingertips in object manipulation tasks. Nature reviews. Neuroscience 10, 5 (2009), 345.

[23]

R. S Johansson and Å B Vallbo. 1979. Tactile sensibility in the human hand: relative and absolute densities of four types of mechanoreceptive units in glabrous skin. The Journal of physiology 286, 1 (1979), 283--300.

[24]

K. Kaczmarek, J. Webster, P. Bach-y Rita, and W. Tompkins. 1991. Electrotactile and vibrotactile displays for sensory substitution systems. IEEE Transactions on Biomedical Engineering 38, 1 (1991), 1--16.

[25]

M. Klapdohr, B. Wöldecke, D. Marinos, J. Herder, C. Geiger, and W. Vonolfen. 2010. Vibrotactile Pitfalls: Arm Guidance for Moderators in Virtual TV Studios. In Proceedings of the 13th International Conference on Humans and Computers (HC '10). University of Aizu Press, 72--80.

Digital Library

[26]

E. Kruijff, A. Marquardt, C. Trepkowski, R. W. Lindeman, A. Hinkenjann, J. Maiero, and B. E. Riecke. 2016. On Your Feet!: Enhancing Vection in Leaning-Based Interfaces Through Multisensory Stimuli. In Proceedings of the 2016 Symposium on Spatial User Interaction (SUI '16). ACM, 149--158.

Digital Library

[27]

E. Kruijff, A. Marquardt, C. Trepkowski, J. Schild, and A. Hinkenjann. 2017. Designed Emotions: Challenges and Potential Methodologies for Improving Multisensory Cues to Enhance User Engagement in Immersive Systems. Vis. Comput. 33, 4 (April 2017), 471--488.

Digital Library

[28]

E. Kruijff, K. Wesche, G. and Riege, G. Goebbels, M. Kunstman, and D. Schmalstieg. 2006. Tactylus, a Pen-input Device Exploring Audiotactile Sensory Binding. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST '06). ACM, 312--315.

Digital Library

[29]

J.J. LaViola, E. Kruijff, R.P. McMahan, D. Bowman, and I.P. Poupyrev. 2017. 3D User Interfaces: Theory and Practice. Pearson Education.

[30]

D. Levac and H. Sveistrup. 2014. Motor Learning and Virtual Reality., 25--46 pages.

[31]

J. Lieberman and C. Breazeal. 2007. TIKL: Development of a Wearable Vibrotactile Feedback Suit for Improved Human Motor Learning. (2007).

[32]

P. Lopes, D. Yüksel, F. Guimbretière, and P. Baudisch. 2016. Muscle-plotter: An Interactive System Based on Electrical Muscle Stimulation That Produces Spatial Output. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, 207--217.

Digital Library

[33]

V. Maheshwari and R. Saraf. 2008. Tactile Devices To Sense Touch on a Par with a Human Finger. Angewandte Chemie International Edition 47, 41 (2008), 7808--7826.

[34]

A. Marquardt, E. Kruijff, C. Trepkowski, J. Maiero, A. Schwandt, A. Hinkenjann, W. Stuerzlinger, and J. Schoening. 2018. Audio-Tactile Feedback for Enhancing 3D Manipulation. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST '18). ACM.

Digital Library

[35]

J. Martinez, A. Garcia, M. Oliver, J. P. Molina, and P. Gonzalez. 2016. Identifying Virtual 3D Geometric Shapes with a Vibrotactile Glove. IEEE Computer Graphics and Applications 36, 1 (Jan 2016), 42--51.

[36]

T. H. Massie, K. Salisbury, et al. 1994. The phantom haptic interface: A device for probing virtual objects. In Proceedings of the ASME winter annual meeting, symposium on haptic interfaces for virtual environment and teleoperator systems, Vol. 55. 295--300.

[37]

T. McDaniel, D. Villanueva, S. Krishna, and S. Panchanathan. 2010. MOVeMENT: A framework for systematically mapping vibrotactile stimulations to fundamental body movements. In Haptic Audio-Visual Environments and Games (HAVE), 2010 IEEE International Symposium on. IEEE, 1--6.

[38]

R. P McMahan, D. A Bowman, D. J Zielinski, and R. B Brady. 2012. Evaluating display fidelity and interaction fidelity in a virtual reality game. IEEE transactions on visualization and computer graphics 18, 4 (2012), 626--633.

Digital Library

[39]

K. Nosaka, A. Aldayel, M. Jubeau, and T. C. Chen. 2011. Muscle damage induced by electrical stimulation. European Journal of Applied Physiology 111, 10 (03 Aug 2011), 2427.

[40]

D. Pai. 2005. Multisensory interaction: Real and virtual. In Robotics Research. The Eleventh International Symposium. Springer, 489--498.

[41]

E. Piateski and L. Jones. 2005. Vibrotactile pattern recognition on the arm and torso. In Eurohaptics Conference, 2005 and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2005. World Haptics 2005. First Joint. IEEE, 90--95.

Digital Library

[42]

H. Regenbrecht, J. Hauber, R. Schoenfelder, and A. Maegerlein. 2005. Virtual Reality Aided Assembly with Directional Vibro-tactile Feedback. In Proceedings of the 3rd International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia (GRAPHITE '05). ACM, 381--387.

Digital Library

[43]

E. Ruffaldi, A. Filippeschi, A. Frisoli, O. Sandoval, C. A. Avizzano, and M. Bergamasco. 2009. Vibrotactile perception assessment for a rowing training system. In World Haptics 2009 - Third Joint EuroHaptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. 350--355.

Digital Library

[44]

K. Sato, K. Minamizawa, N. Kawakami, and S. Tachi. 2007. Haptic Telexistence. In ACM SIGGRAPH 2007 Emerging Technologies (SIGGRAPH '07). ACM, Article 10.

Digital Library

[45]

R.A. Schmidt and C.A. Wrisberg. 2004. Motor Learning and Performance. Human Kinetics.

[46]

C. Schönauer, K. Fukushi, A. Olwal, H. Kaufmann, and R. Raskar. 2012. Multimodal Motion Guidance: Techniques for Adaptive and Dynamic Feedback. In Proceedings of the 14th ACM International Conference on Multimodal Interaction (ICMI '12). ACM, 133--140.

Digital Library

[47]

D. Spelmezan, M. Jacobs, A. Hilgers, and J. Borchers. 2009. Tactile Motion Instructions for Physical Activities. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, 2243--2252.

Digital Library

[48]

C. Spence and S. Squire. 2003. Multisensory integration: maintaining the perception of synchrony. Current Biology 13, 13 (2003), R519--R521.

[49]

A. A. Stanley and K. J. Kuchenbecker. 2012. Evaluation of Tactile Feedback Methods for Wrist Rotation Guidance. EEE Trans. Haptics 5, 3 (Jan. 2012), 240--251.

Digital Library

[50]

E. Tamaki, T. Miyaki, and J. Rekimoto. 2011. PossessedHand: Techniques for Controlling Human Hands Using Electrical Muscles Stimuli. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, 543--552.

Digital Library

[51]

H. Uematsu, D. Ogawa, R. Okazaki, T. Hachisu, and H. Kajimoto. 2016. HALUX: projection-based interactive skin for digital sports. In SIGGRAPH Emerging Technologies.

Digital Library

[52]

G. H. VanDoorn, V. Dubaj, D. B. Wuillemin, B. L. Richardson, and M. A. Symmons. 2012. Cognitive Load Can Explain Differences in Active and Passive Touch. In Haptics: Perception, Devices, Mobility, and Communication, P. Isokoski and J. Springare (Eds.). Springer Berlin Heidelberg, 91--102.

Digital Library

[53]

S. Vishniakou, B. W. Lewis, X. Niu, A. Kargar, K. Sun, M. Kalajian, N. Park, M. Yang, Y. Jing, P. Brochu, et al. 2013. Tactile Feedback Display with Spatial and Temporal Resolutions. Scientific reports 3 (2013), 2521.

[54]

H. S. Vitense, J. A. Jacko, and V. K. Emery. 2002. Multimodal Feedback: Establishing a Performance Baseline for Improved Access by Individuals with Visual Impairments. In Proceedings of the Fifth International ACM Conference on Assistive Technologies (Assets '02). ACM, 49--56.

Digital Library

[55]

J. Zheng, Y. and Morrell. 2010. A vibrotactile feedback approach to posture guidance. In Haptics Symposium, 2010 IEEE. IEEE, 351--358.

Digital Library

[56]

M. Zhou, D.B. Jones, S.D. Schwaitzberg, and C.G.L. Cao. 2007. Role of Haptic Feedback and Cognitive Load in Surgical Skill Acquisition. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 51, 11 (2007),631--635.

Cited By

Bashar MBatmaz A(2024)Virtual Task Environments Factors Explored in 3D Selection StudiesProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670983(1-16)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3670983
Elsayed HWeigel MSemsch JMühlhäuser MSchmitz MWard JMcGill MMarky K(2023)Tactile Vectors for Omnidirectional Arm GuidanceProceedings of the Augmented Humans International Conference 202310.1145/3582700.3582701(35-45)Online publication date: 12-Mar-2023
https://dl.acm.org/doi/10.1145/3582700.3582701
Kruijff ERiecke BTrepkowski CLindeman R(2022)First insights in perception of feet and lower-body stimuli for proximity and collision feedback in 3D user interfacesFrontiers in Virtual Reality10.3389/frvir.2022.9545873Online publication date: 18-Oct-2022
https://doi.org/10.3389/frvir.2022.954587
Show More Cited By

Index Terms

Tactile hand motion and pose guidance for 3D interaction
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Mobile gesture interaction using wearable tactile displays
CHI EA '09: CHI '09 Extended Abstracts on Human Factors in Computing Systems

We present an interaction method for mobile gesture interaction using wearable tactile displays. We are attempting to show that wrist-worn tactile displays provide adequate feedback to enable reversible and error-resistant gesture-based interaction. In ...
A robust and accurate 3d hand posture estimation method for interactive systems
TEI '10: Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction

In this paper, a new 3D hand posture estimation system using a single camera and 3 interactive systems are introduced. Existing hand gesture recognition systems estimate hand's 3D models based on image features such as contour or skin texture. However, ...
Tactile Feedback for Above-Device Gesture Interfaces: Adding Touch to Touchless Interactions
ICMI '14: Proceedings of the 16th International Conference on Multimodal Interaction

Above-device gesture interfaces let people interact in the space above mobile devices using hand and finger movements. For example, users could gesture over a mobile phone or wearable without having to use the touchscreen. We look at how above-device ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology

November 2018

570 pages

ISBN:9781450360869

DOI:10.1145/3281505

Editor:
Stephen N. Spencer
University of Washington
,
General Chair:
Shigeo Morishima
Waseda University
,
Program Chairs:
Yuichi Itoh
Osaka University
,
Takaaki Shiratori
Facebook Reality Labs
,
Yonghao Yue
The University of Tokyo
,
Rob Lindeman
University of Canterbury

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 November 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

VRST '18

Sponsor:

VRST '18: 24th ACM Symposium on Virtual Reality Software and Technology

November 28 - December 1, 2018

Tokyo, Japan

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
446
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)2

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bashar MBatmaz A(2024)Virtual Task Environments Factors Explored in 3D Selection StudiesProceedings of the 50th Graphics Interface Conference10.1145/3670947.3670983(1-16)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3670947.3670983
Elsayed HWeigel MSemsch JMühlhäuser MSchmitz MWard JMcGill MMarky K(2023)Tactile Vectors for Omnidirectional Arm GuidanceProceedings of the Augmented Humans International Conference 202310.1145/3582700.3582701(35-45)Online publication date: 12-Mar-2023
https://dl.acm.org/doi/10.1145/3582700.3582701
Kruijff ERiecke BTrepkowski CLindeman R(2022)First insights in perception of feet and lower-body stimuli for proximity and collision feedback in 3D user interfacesFrontiers in Virtual Reality10.3389/frvir.2022.9545873Online publication date: 18-Oct-2022
https://doi.org/10.3389/frvir.2022.954587
Tsai HChang YWei TTsao CKoo XWang HChen BKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)GuideBand: Intuitive 3D Multilevel Force Guidance on a Wristband in Virtual RealityProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445262(1-13)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445262
Loschner PHinkenjann AKruijff E(2021)IllumiWand: Improving 3D Interaction with Monoscopic Displays Through a Projected Physical 3D Pointer2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW52623.2021.00046(210-213)Online publication date: Mar-2021
https://doi.org/10.1109/VRW52623.2021.00046
Dandu BShao YVisell Y(2021)Rendering Spatiotemporal Haptic Effects Via the Physics of Waves in the SkinIEEE Transactions on Haptics10.1109/TOH.2020.302976814:2(347-358)Online publication date: 1-Apr-2021
https://doi.org/10.1109/TOH.2020.3029768
Li GWu ZLiu YZhang HNie YMao A(2021)3D hand reconstruction from a single image based on biomechanical constraintsThe Visual Computer10.1007/s00371-021-02250-yOnline publication date: 27-Jul-2021
https://doi.org/10.1007/s00371-021-02250-y
Marquardt ATrepkowski CEibich TMaiero JKruijff ESchoning J(2020)Comparing Non-Visual and Visual Guidance Methods for Narrow Field of View Augmented Reality DisplaysIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.302360526:12(3389-3401)Online publication date: Dec-2020
https://doi.org/10.1109/TVCG.2020.3023605
Kruijff EBiswas STrepkowski CMaiero JGhinea GStuerzlinger WBrewster SFitzpatrick GCox AKostakos V(2019)Multilayer Haptic Feedback for Pen-Based Tablet InteractionProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300373(1-14)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300373
Marquardt AKruijff ETrepkowski CMaiero JSchwandt AHinkenjann AStuerzlinger WSchöning JMorishima SItoh YShiratori TYue YLindeman R(2018)Audio-tactile proximity feedback for enhancing 3D manipulationProceedings of the 24th ACM Symposium on Virtual Reality Software and Technology10.1145/3281505.3281525(1-10)Online publication date: 28-Nov-2018
https://dl.acm.org/doi/10.1145/3281505.3281525

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten