research-article

Audio-tactile proximity feedback for enhancing 3D manipulation

Authors:

Alexander Marquardt,

Christina Trepkowski,

Andrea Schwandt,

André Hinkenjann,

Wolfgang Stuerzlinger,

Johannes SchöningAuthors Info & Claims

VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology

Article No.: 2, Pages 1 - 10

https://doi.org/10.1145/3281505.3281525

Published: 28 November 2018 Publication History

Abstract

In presence of conflicting or ambiguous visual cues in complex scenes, performing 3D selection and manipulation tasks can be challenging. To improve motor planning and coordination, we explore audio-tactile cues to inform the user about the presence of objects in hand proximity, e.g., to avoid unwanted object penetrations. We do so through a novel glove-based tactile interface, enhanced by audio cues. Through two user studies, we illustrate that proximity guidance cues improve spatial awareness, hand motions, and collision avoidance behaviors, and show how proximity cues in combination with collision and friction cues can significantly improve performance.

References

[1]

C. Afonso and S. Beckhaus. 2011. How to Not Hit a Virtual Wall: Aural Spatial Awareness for Collision Avoidance in Virtual Environments. In Proceedings of the 6th Audio Mostly Conference: A Conference on Interaction with Sound (AM '11). ACM, 101--108.

Digital Library

[2]

N. Ariza, P. Lubos, F. Steinicke, and G. Bruder. 2015. Ring-shaped Haptic Device with Vibrotactile Feedback Patterns to Support Natural Spatial Interaction. In ICAT - EGVE '15 Proceedings of the 25th International Conference on Artificial Reality and Telexistence and 20th Eurographics Symposium on Virtual Environments.

Digital Library

[3]

O. Ariza, G. Bruder, N. Katzakis, and F. Steinicke. 2018. Analysis of Proximity-Based Multimodal Feedback for 3D Selection in Immersive Virtual Environments. In Proceedings of IEEE Virtual Reality (VR).

[4]

S. Beckhaus, F. Ritter, and T. Strothotte. 2000. CubicalPath-dynamic potential fields for guided exploration in virtual environments. In Proceedings the Eighth Pacific Conference on Computer Graphics and Applications. 387--459.

Digital Library

[5]

H. Benko, C. Holz, M. Sinclair, and E. Ofek. 2016. NormalTouch and TextureTouch: High-fidelity 3D Haptic Shape Rendering on Handheld Virtual Reality Controllers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, 717--728.

Digital Library

[6]

A. Bloomfield, Y. Deng, J. Wampler, P. Rondot, D. Harth, M. McManus, and N. Badler. 2003. A taxonomy and comparison of haptic actions for disassembly tasks. In Virtual Reality, 2003. Proceedings. IEEE. IEEE, 225--231.

Digital Library

[7]

M. Bouzit, G. Burdea, G. Popescu, and R. Boian. 2002. The Rutgers Master II-new design force-feedback glove. IEEE/ASME Transactions on mechatronics 7, 2 (2002), 256--263.

[8]

G. Burdea. 1996. Force and Touch Feedback for Virtual Reality. John Wiley & Sons, Inc.

Digital Library

[9]

L. Chan, R. Liang, M. Tsai, C. Cheng, K. and Su, M. Chen, W. Cheng, and B. Chen. 2013. FingerPad: Private and Subtle Interaction Using Fingertips. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, 255--260.

Digital Library

[10]

E. Chancey, J. Brill, A. Sitz, U. Schmuntzsch, and J. Bliss. 2014. Vibrotactile Stimuli Parameters on Detection Reaction Times. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 58, 1 (2014), 1701--1705.

[11]

W. Chang, W. Hwang, and Y. Ji. 2011. Haptic seat interfaces for driver information and warning systems. International Journal of Human-Computer Interaction 27, 12 (2011), 1119--1132.

[12]

A. Cockburn and S. Brewster. 2005. Multimodal feedback for the acquisition of small targets. Ergonomics 48, 9 (2005), 1129--1150.

[13]

M. Ernst and M. Banks. 2002. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 6870 (2002), 429.

[14]

A. Kunert C. Andujar F. Argelaguet, A. Kulik and B. Froehlich. 2011. See-through techniques for referential awareness in collaborative virtual reality. International Journal of Human-Computer Studies 69, 6 (2011), 387--400.

Digital Library

[15]

P. Gallotti, A. Raposo, and L. Soares. 2011. v-Glove: A 3D Virtual Touch Interface. In 2011 XIII Symposium on Virtual Reality. 242--251.

Digital Library

[16]

U. Gollner, T. Bieling, and G. Joost. 2012. Mobile Lorm Glove: Introducing a Communication Device for Deaf-blind People. In Proceedings of the Sixth International Conference on Tangible, Embedded and Embodied Interaction (TEI '12). ACM, 127--130.

Digital Library

[17]

J. Hartcher-O'Brien, M. Auvray, and V. Hayward. 2015. Perception of distance-to-obstacle through time-delayed tactile feedback. In 2015 IEEE World Haptics Conference (WHC). 7--12.

[18]

C. Hatzfeld and T.A. Kern. 2014. Engineering Haptic Devices: A Beginner's Guide. Springer London.

Digital Library

[19]

B. Holbert. 2007. Enhanced Targeting in a Haptic User Interface for the Physically Disabled Using a Force Feedback Mouse. Ph.D. Dissertation. Advisor(s) Huber, Manfred. AAI3277666.

Digital Library

[20]

A. Israr and I. Poupyrev. 2011. Tactile Brush: Drawing on Skin with a Tactile Grid Display. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, 2019--2028.

Digital Library

[21]

H. Jin, Q. Chen, Z. Chen, Y. Hu, and J. Zhang. 2016. Multi-LeapMotion sensor based demonstration for robotic refine tabletop object manipulation task. CAAI Transactions on Intelligence Technology 1, 1 (2016), 104 -- 113.

[22]

R. Johansson and J. Flanagan. 2009. Coding and use of tactile signals from the fingertips in object manipulation tasks. Nature reviews. Neuroscience 10, 5 (2009), 345.

[23]

R. Johansson and A. Vallbo. 1979. Tactile sensibility in the human hand: relative and absolute densities of four types of mechanoreceptive units in glabrous skin. The Journal of physiology 286, 1 (1979), 283--300.

[24]

K. Johnson and S. Hsiao. 1992. Neural mechanisms of tactual form and texture perception. Annual review of neuroscience 15, 1 (1992), 227--250.

[25]

K. A Kaczmarek, J. Webster, P. Bach-y Rita, and W. Tompkins. 1991. Electrotactile and vibrotactile displays for sensory substitution systems. IEEE Transactions on Biomedical Engineering 38, 1 (1991), 1--16.

[26]

D. Karnopp. 1985. Computer simulation of stick-slip friction in mechanical dynamic systems. J. Dyn. Syst. Meas. Control. 107, 1 (1985), 100--103.

[27]

T. Kassuba, M. Menz, B. RÃőder, and H. Siebner. 2013. Multisensory Interactions between Auditory and Haptic Object Recognition. Cerebral Cortex 23, 5 (2013), 1097--1107.

[28]

K. Kozak, J. Pohl, W. Birk, J. Greenberg, B. Artz, M. Blommer, L. Cathey, and R. Curry. 2006. Evaluation of Lane Departure Warnings for Drowsy Drivers. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50, 22 (2006), 2400--2404.

[29]

E. Kruijff, A. Marquardt, C. Trepkowski, R. Lindeman, A. Hinkenjann, J. Maiero, and B. Riecke. 2016. On Your Feet!: Enhancing Vection in Leaning-Based Interfaces Through Multisensory Stimuli. In Proceedings of the 2016 Symposium on Spatial User Interaction (SUI '16). ACM, New York, NY, USA, 149--158.

Digital Library

[30]

E. Kruijff, A. Marquardt, C. Trepkowski, J. Schild, and A. Hinkenjann. 2017. Designed Emotions: Challenges and Potential Methodologies for Improving Multisensory Cues to Enhance User Engagement in Immersive Systems. Vis. Comput. 33, 4 (April 2017), 471--488.

Digital Library

[31]

E. Kruijff, G. Wesche, K. Riege, G. Goebbels, M. Kunstman, and D. Schmalstieg. 2006. Tactylus, a Pen-input Device Exploring Audiotactile Sensory Binding. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST '06). ACM, 312--315.

Digital Library

[32]

J.J. LaViola, E. Kruijff, R.P. McMahan, D. Bowman, and I.P. Poupyrev. 2017. 3D User Interfaces: Theory and Practice. Pearson Education.

[33]

R. Lindeman, R. Page, Y. Yanagida, and J. Sibert. 2004. Towards full-body haptic feedback: the design and deployment of a spatialized vibrotactile feedback system. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST 2004. 146--149.

Digital Library

[34]

L. Liu and R. van Liere. 2009. Designing 3D Selection Techniques Using Ballistic and Corrective Movements. In Proceedings of the 15th Joint Virtual Reality Eurographics Conference on Virtual Environments (JVRC'09). Eurographics Association, 1--8.

Digital Library

[35]

A. Marquardt, J. Maiero, E. Kruijff, C. Trepkowski, A. Schwandt, A. Hinkenjann, J. Schoening, and W. Stuerzlinger. 2018. Tactile Hand Motion and Pose Guidance for 3D Interaction. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST '18). ACM.

Digital Library

[36]

J. Martinez, A. Garcia, M. Oliver, J. P. Molina, and P. Gonzalez. 2016. Identifying Virtual 3D Geometric Shapes with a Vibrotactile Glove. IEEE Computer Graphics and Applications 36, 1 (2016), 42--51.

[37]

V. Mateevitsi, B. Haggadone, J. Leigh, B. Kunzer, and R. Kenyon. 2013. Sensing the environment through SpiderSense. In Proceedings of the 4th augmented human international conference. ACM, 51--57.

Digital Library

[38]

M. Mine, F. Brooks, Jr., and Carlo H. Sequin. 1997. Moving Objects in Space: Exploiting Proprioception in Virtual-environment Interaction. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '97). ACM Press/Addison-Wesley Publishing Co., 19--26.

Digital Library

[39]

V. Occelli, Charles Spence, and Massimiliano Zampini. 2011. Audiotactile interactions in temporal perception. Psychonomic Bulletin & Review 18, 3 (01 Jun 2011), 429--454.

[40]

T. Oron-Gilad, J. L. Downs, R. D. Gilson, and P. A. Hancock. 2007. Vibrotactile Guidance Cues for Target Acquisition. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37, 5 (2007), 993--1004.

Digital Library

[41]

D. Pai. 2005. Multisensory interaction: Real and virtual. In Robotics Research. The Eleventh International Symposium. Springer, 489--498.

[42]

R. Pastel. 2006. Measuring the Difficulty of Steering Through Corners. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '06). ACM, 1087--1096.

Digital Library

[43]

B. Petzold, M. Zaeh, B. Faerber, B. Deml, H. Egermeier, J. Schilp, and S. Clarke. 2004. A Study on Visual, Auditory, and Haptic Feedback for Assembly Tasks. Presence: Teleoper. Virtual Environ. 13, 1 (Feb. 2004), 16--21.

Digital Library

[44]

M. Prachyabrued and C. W. Borst. 2014. Visual feedback for virtual grasping. In 2014 IEEE Symposium on 3D User Interfaces (3DUI). 19--26.

[45]

H. Regenbrecht, J. Hauber, R. Schoenfelder, and A. Maegerlein. 2005. Virtual Reality Aided Assembly with Directional Vibro-tactile Feedback. In Proceedings of the 3rd International Conference on Computer Graphics and Interactive Techniques in Australasia and South East Asia (GRAPHITE '05). ACM, 381--387.

Digital Library

[46]

S. Roffler and R. Butler. 1968. Localization of tonal stimuli in the vertical plane. The Journal of the Acoustical Society of America 43, 6 (1968), 1260--1266.

[47]

R.A. Schmidt and C.A. Wrisberg. 2004. Motor Learning and Performance. Human Kinetics.

[48]

C. Seim, N. Doering, Y. Zhang, W. Stuerzlinger, and T. Starner. 2017. Passive Haptic Training to Improve Speed and Performance on a Keypad. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 100 (Sept. 2017), 13 pages.

Digital Library

[49]

S. Hazenberg and R. van Lier. 2016. Touching and Hearing Unseen Objects: Multisensory Effects on Scene Recognition. i-Perception 7, 4 (2016).

[50]

C. Spence and S. Squire. 2003. Multisensory integration: maintaining the perception of synchrony. Current Biology 13, 13 (2003), R519--R521.

[51]

J. Sreng, A. Lecuyer, C. Megard, and C. Andriot. 2006. Using Visual Cues of Contact to Improve Interactive Manipulation of Virtual Objects in Industrial Assembly/Maintenance Simulations. IEEE Transactions on Visualization and Computer Graphics 12, 5 (2006), 1013--1020.

Digital Library

[52]

H. Uchiyama, M. Covington, and W. Potter. 2008. Vibrotactile Glove guidance for semi-autonomous wheelchair operations. In Proceedings of the 46th Annual Southeast Regional Conference on XX. ACM, 336--339.

Digital Library

[53]

N. van Atteveldt, M. Murray, G. Thut, and C. Schroeder. 2014. Multisensory Integration: Flexible Use of General Operations. Neuron 81, 6 (2014), 1240--1253.

[54]

R. Van der Linde, P. Lammertse, E. Frederiksen, and B. Ruiter. 2002. The Haptic-Master, a new high-performance haptic interface. In Proc. Eurohaptics. 1--5.

[55]

S. Vishniakou, B. Lewis, X. Niu, A. Kargar, K. Sun, M. Kalajian, N. Park, M. Yang, Y. Jing, P. Brochu, et al. 2013. Tactile Feedback Display with Spatial and Temporal Resolutions. Scientific reports 3 (2013), 2521.

[56]

H. Vitense, J. Jacko, and V. Emery. 2002. Multimodal Feedback: Establishing a Performance Baseline for Improved Access by Individuals with Visual Impairments. In Proceedings of the Fifth International ACM Conference on Assistive Technologies (Assets '02). ACM, 49--56.

Digital Library

[57]

S. Yamanaka, W. Stuerzlinger, and H. Miyashita. 2018. Steering Through Successive Objects. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, Article 603, 13 pages.

Digital Library

Cited By

Yangfu RZhiqiang LSonghai Z(2024)Survey of visualization methods for multiscene visual cue information in immersive environmentsJournal of Image and Graphics10.11834/jig.22114729:1(1-21)Online publication date: 2024
https://doi.org/10.11834/jig.221147
Guarese RPretty ERenata APolson DZambetta F(2024)Exploring Audio Interfaces for Vertical Guidance in Augmented Reality via Hand-Based FeedbackIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337204030:5(2818-2828)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372040
Kruijff ERiecke BTrepkowski CLindeman R(2022)First insights in perception of feet and lower-body stimuli for proximity and collision feedback in 3D user interfacesFrontiers in Virtual Reality10.3389/frvir.2022.9545873Online publication date: 18-Oct-2022
https://doi.org/10.3389/frvir.2022.954587
Show More Cited By

Index Terms

Audio-tactile proximity feedback for enhancing 3D manipulation
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Haptic devices
    2. Interaction techniques
      1. Auditory feedback

Recommendations

Tactile hand motion and pose guidance for 3D interaction
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology

We present a novel forearm-and-glove tactile interface that can enhance 3D interaction by guiding hand motor planning and coordination. In particular, we aim to improve hand motion and pose actions related to selection and manipulation tasks. Through ...
Influence of tactile feedback and presence on egocentric distance perception in virtual environments
VR '10: Proceedings of the 2010 IEEE Virtual Reality Conference

A number of studies have reported that distance judgments are underestimated in virtual environments (VE) when compared to those made in the real world. Studies have also reported that providing users with visual feedback in the VE improves their ...
FocusFlow: 3D Gaze-Depth Interaction in Virtual Reality Leveraging Active Visual Depth Manipulation
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems

Gaze interaction presents a promising avenue in Virtual Reality (VR) due to its intuitive and efficient user experience. Yet, the depth control inherent in our visual system remains underutilized in current methods. In this study, we introduce FocusFlow,...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology

November 2018

570 pages

ISBN:9781450360869

DOI:10.1145/3281505

Editor:
Stephen N. Spencer
University of Washington
,
General Chair:
Shigeo Morishima
Waseda University
,
Program Chairs:
Yuichi Itoh
Osaka University
,
Takaaki Shiratori
Facebook Reality Labs
,
Yonghao Yue
The University of Tokyo
,
Rob Lindeman
University of Canterbury

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 November 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

VRST '18

Sponsor:

VRST '18: 24th ACM Symposium on Virtual Reality Software and Technology

November 28 - December 1, 2018

Tokyo, Japan

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
434
Total Downloads

Downloads (Last 12 months)55
Downloads (Last 6 weeks)6

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yangfu RZhiqiang LSonghai Z(2024)Survey of visualization methods for multiscene visual cue information in immersive environmentsJournal of Image and Graphics10.11834/jig.22114729:1(1-21)Online publication date: 2024
https://doi.org/10.11834/jig.221147
Guarese RPretty ERenata APolson DZambetta F(2024)Exploring Audio Interfaces for Vertical Guidance in Augmented Reality via Hand-Based FeedbackIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337204030:5(2818-2828)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372040
Kruijff ERiecke BTrepkowski CLindeman R(2022)First insights in perception of feet and lower-body stimuli for proximity and collision feedback in 3D user interfacesFrontiers in Virtual Reality10.3389/frvir.2022.9545873Online publication date: 18-Oct-2022
https://doi.org/10.3389/frvir.2022.954587
Batmaz AYu KLiang HStuerzlinger W(2022)Improving Effective Throughput Performance using Auditory Feedback in Virtual RealityProceedings of the 2022 ACM Symposium on Spatial User Interaction10.1145/3565970.3567702(1-11)Online publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1145/3565970.3567702
Woźniak MDominiak JGrudzień KWróbel-Lachowska MNiess JWoźniak PRomanowski A(2022)Gapeau: Enhancing the Sense of Distance to Others with a Head-Mounted SensorProceedings of the Sixteenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3490149.3501323(1-19)Online publication date: 13-Feb-2022
https://dl.acm.org/doi/10.1145/3490149.3501323
Trepkowski CMarquardt AEibich TShikanai YMaiero JKiyokawa KKruijff ESchoning JKonig P(2022)Multisensory Proximity and Transition Cues for Improving Target Awareness in Narrow Field of View Augmented Reality DisplaysIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.311667328:2(1342-1362)Online publication date: 1-Feb-2022
https://doi.org/10.1109/TVCG.2021.3116673
Navarro SMuhlbacher-Karrer SAlagi HZangl HKoyama KHein BDuriez CSmith J(2022)Proximity Perception in Human-Centered Robotics: A Survey on Sensing Systems and ApplicationsIEEE Transactions on Robotics10.1109/TRO.2021.311178638:3(1599-1620)Online publication date: Jun-2022
https://doi.org/10.1109/TRO.2021.3111786
Kaneko SYokosaka TKajimoto HKawabe T(2022)A Pseudo-Haptic Method Using Auditory Feedback: The Role of Delay, Frequency, and Loudness of Auditory Feedback in Response to a User’s Button Click in Causing a Sensation of HeavinessIEEE Access10.1109/ACCESS.2022.317232410(50008-50022)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3172324
Lu YGao BTu HLuo WKim H(2021)Effects of Different Proximity-Based Feedback on Virtual Hand Pointing in Virtual RealityAdvances in Computer Graphics10.1007/978-3-030-89029-2_45(608-620)Online publication date: 11-Oct-2021
https://doi.org/10.1007/978-3-030-89029-2_45
Jones BMaiero JMogharrab AAguliar IAdhikari ARiecke BKruijff ENeustaedter CLindeman RTruong KHeylen DCzerwinski MBerthouze NChetouani MNakano M(2020)FeetBack: Augmenting Robotic Telepresence with Haptic Feedback on the FeetProceedings of the 2020 International Conference on Multimodal Interaction10.1145/3382507.3418820(194-203)Online publication date: 21-Oct-2020
https://dl.acm.org/doi/10.1145/3382507.3418820
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten