skip to main content
research-article

Evaluating Multimodal Feedback for Assembly Tasks in a Virtual Environment

Published: 13 June 2019 Publication History

Abstract

Operating power tools over extended periods of time can pose significant risks to humans, due to the strong forces and vibrations they impart to the limbs. Telemanipulation systems can be employed to minimize these risks, but may impede effective task performance due to the reduced sensory cues they typically convey. To address this shortcoming, we explore the benefits of augmenting these cues with the addition of audition, vibration, and force feedback, and evaluate them on users' performance in a VR mechanical assembly task employing a simulated impact wrench. Our research focuses on the utility of vibrotactile feedback, rendered as a simplified and attenuated version of the vibrations experienced while operating an actual impact wrench. We investigate whether such feedback can serve to enhance the operator's awareness of the state of the tool, as well as a proxy for the forces experienced during collisions and coupling, while operating the tool an actual impact wrench. Results from our user study comparing feedback modalities confirm that the introduction of vibrotactile, in addition to auditory feedback can significantly improve user performance as assessed by completion time. However, the addition of force feedback to these two modalities did not further improve performance.

References

[1]
Abdulaziz Alshaer, Holger Regenbrecht, and David O'Hare. 2015. Investigating visual dominance with a virtual driving task. In 2015 IEEE Virtual Reality (VR). IEEE, 145--146.
[2]
Amrish O. Chourasia, Mary E. Sesto, Walter F. Block, and Robert G. Radwin. 2009. Prolonged mechanical and physiological changes in the upper extremity following short-term simulated power hand tool use. Ergonomics, Vol. 52, 1 (2009), 15--24.
[3]
J.V. Draper, W.E. Moore, J.N. Herndon, B.S., and Weil. 1997. Effects of Force Reflection on Servomanipulator Task Performance. In Proc. IEEE International Topical Meeting Remote Systems and Robots in Hostile Environments .
[4]
B. Hannaford, L. Wood, D.A. McAffee, and H. Zak. 1991. Performance Evaluation of a Six-Axis Generalized Force-Reflecting Teleoperator., Vol. 21 (May/June 1991), 620--633.
[5]
Sandra G. Hart and Lowell E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research. Advances in Psychology, Vol. 52 (1988), 139 -- 183.
[6]
Jun Okamoto Jr and Anderson Nishihara. 2016. Assembly Assisted by Augmented Reality (A3R). In Intelligent Systems and Applications. Springer-Verlag, 281--300.
[7]
Roberta L. Klatzky, Susan Lederman, and Catherine Reed. 1987. There's More to Touch than Meets the Eye: The Salience of Object Attributes for Haptics with and without Vision. Journal of Experimental Psychology: General, Vol. 116 (1987), 356--369.
[8]
Raymond M Klein. 1977. Attention and visual dominance: a chronometric analysis. Journal of Experimental Psychology: Human Perception and Performance, Vol. 3, 3 (1977), 365.
[9]
Jia-Hua Lin, Raymond W. McGorry, and Jacob J. Banks. 2010. Exposures and physiological responses in power tool operations: fastening vs. unfastening threaded hardware. Journal of occupational and environmental hygiene, Vol. 7, 5 (2010), 290--297.
[10]
Robert W Lindeman, James N Templeman, John L Sibert, and Justin R Cutler. 2002. Handling of virtual contact in immersive virtual environments: beyond visuals. Virtual Reality, Vol. 6, 3 (2002), 130--139.
[11]
Sotiris Makris, Panagiotis Karagiannis, Spyridon Koukas, and Aleksandros-Stereos Matthaiakis. 2016. Augmented reality system for operator support in human-robot collaborative assembly. CIRP Annals - Manufacturing Technology, Vol. 65, 1 (2016), 61 -- 64.
[12]
M. Massimo and T. Sheridan. 1989. Variable Force and Visual Feedback Effects on Teleoperator Man-Machine Performance. In Proceedings of the NASA Conference on Space Telerobotics, Vol. 1. 89--98.
[13]
Gilberto Osorio-Gómez, Roberto Viganò, and Juan Carlos Arbeláez. 2016. An augmented reality tool to validate the assembly sequence of a discrete product. International Journal of Computer Aided Engineering and Technology, Vol. 8, 1--2 (2016), 164--178.
[14]
Claudio Pacchierotti, Domenico Prattichizzo, and Katherine J Kuchenbecker. 2016. Cutaneous feedback of fingertip deformation and vibration for palpation in robotic surgery. IEEE Transactions on Biomedical Engineering, Vol. 63, 2 (2016), 278--287.
[15]
Bernd Petzold, Michael F Zaeh, Berthold Faerber, Barbara Deml, Hans Egermeier, Johannes Schilp, and Stella Clarke. 2004. A study on visual, auditory, and haptic feedback for assembly tasks. Presence: teleoperators and virtual environments, Vol. 13, 1 (2004), 16--21.
[16]
M. D. Pottenger and P. Benhaim. 2004. Vibration absorbing brace for study of work-related upper extremity musculoskeletal disorders. In Engineering in Medicine and Biology Society, 2004. IEMBS'04. 26th Annual International Conference of the IEEE, Vol. 1. IEEE, 2466--2469.
[17]
M Poyade, M Kargas, and V Portela. 2014. Haptic plug-in for unity. Digital Design Studio (DDS), Glasgow School of Art, Glasgow, United Kingdom (2014).
[18]
John Rosecrance, Dan Anton, Thomas Cook, and Linda Merlino. 2005. Effect of pneumatic power tool use on nerve conduction velocity across the wrist. Human Factors and Ergonomics in Manufacturing & Service Industries, Vol. 15, 4 (2005), 339--352.
[19]
C. H. Roux, F. Guillemin, S. Boini, F. Longuetaud, N. Arnault, S. Hercberg, and S. Briancc on. 2005. Impact of musculoskeletal disorders on quality of life: An inception cohort study. Annals of the Rheumatic Diseases, Vol. 64, 4 (2005), 606--611.
[20]
Mikel Sagardia and Thomas Hulin. 2017. Multimodal evaluation of the differences between real and virtual assemblies. IEEE transactions on haptics (2017).
[21]
ZB Wang, LX Ng, SK Ong, and AYC Nee. 2013. Assembly planning and evaluation in an augmented reality environment. International Journal of Production Research, Vol. 51, 23--24 (2013), 7388--7404.
[22]
Sabine Webel, Uli Bockholt, Timo Engelke, Nirit Gavish, Manuel Olbrich, and Carsten Preusche. 2013. An augmented reality training platform for assembly and maintenance skills. Robotics and Autonomous Systems, Vol. 61, 4 (2013), 398--403.
[23]
Bernhard Weber, Mikel Sagardia, Thomas Hulin, and Carsten Preusche. 2013. Visual, vibrotactile, and force feedback of collisions in virtual environments: effects on performance, mental workload and spatial orientation. In International Conference on Virtual, Augmented and Mixed Reality. Springer, 241--250.
[24]
Jeroen G.W. Wildenbeest, David A. Abbink, Cock J.M. Heemskerk, Frans C.T. van der Helm, and Henri Boessenkool. 2013. The Impact of Haptic Feedback Quality on the Performance of Teleoperated Assembly Tasks., Vol. 6 (April/June 2013), 242--252.
[25]
M.L. Yuan, S.K. Ong, and A.Y.C. Nee. 2008. Augmented reality for assembly guidance using a virtual interactive tool. International Journal of Production Research, Vol. 46 (2008), 1745--1767.

Cited By

View all
  • (2024)Haptic Feedback, Performance and Arousal: A Comparison Study in an Immersive VR Motor Skill Training TaskIEEE Transactions on Haptics10.1109/TOH.2023.331903417:2(249-262)Online publication date: Apr-2024
  • (2023)Providing Dual Awareness using Multimodal Cues for Collaborative Manipulation in Virtual EnvironmentsInternational Journal of Human–Computer Interaction10.1080/10447318.2023.218853340:12(3298-3312)Online publication date: 3-Apr-2023
  • (2023)The effect of audio on the experience in virtual reality: a scoping reviewBehaviour & Information Technology10.1080/0144929X.2022.215837143:1(165-199)Online publication date: 2-Jan-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction
Proceedings of the ACM on Human-Computer Interaction  Volume 3, Issue EICS
June 2019
553 pages
EISSN:2573-0142
DOI:10.1145/3340630
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2019
Published in PACMHCI Volume 3, Issue EICS

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. multimodal feedback
  2. power tools
  3. teleoperation
  4. virtual reality

Qualifiers

  • Research-article

Funding Sources

  • Fonds de Recherche du Québec - Nature et Technologies

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)62
  • Downloads (Last 6 weeks)4
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Haptic Feedback, Performance and Arousal: A Comparison Study in an Immersive VR Motor Skill Training TaskIEEE Transactions on Haptics10.1109/TOH.2023.331903417:2(249-262)Online publication date: Apr-2024
  • (2023)Providing Dual Awareness using Multimodal Cues for Collaborative Manipulation in Virtual EnvironmentsInternational Journal of Human–Computer Interaction10.1080/10447318.2023.218853340:12(3298-3312)Online publication date: 3-Apr-2023
  • (2023)The effect of audio on the experience in virtual reality: a scoping reviewBehaviour & Information Technology10.1080/0144929X.2022.215837143:1(165-199)Online publication date: 2-Jan-2023
  • (2022)VIREO: Web-based Graphical Authoring of Vibrotactile Feedback for Interactions with Mobile and Wearable DevicesInternational Journal of Human–Computer Interaction10.1080/10447318.2022.210958439:20(4162-4180)Online publication date: 18-Aug-2022
  • (2021)HearIt: Auditory-Cue-Based Audio Playback Control to Facilitate Information Browsing in Lecture AudioApplied Sciences10.3390/app1109380311:9(3803)Online publication date: 22-Apr-2021
  • (2021)Evaluating the Efficiency of Six-DoF Haptic Rendering-Based Virtual Assembly TrainingIEEE Transactions on Haptics10.1109/TOH.2020.300894114:1(212-224)Online publication date: 1-Jan-2021
  • (2021)Multi-view data capture for dynamic object reconstruction using handheld augmented reality mobilesJournal of Real-Time Image Processing10.1007/s11554-021-01095-x18:2(345-355)Online publication date: 1-Apr-2021
  • (2020)Effectiveness of Virtual Versus Physical Training: The Case of Assembly Tasks, Trainer's Verbal Assistance, and Task ComplexityIEEE Computer Graphics and Applications10.1109/MCG.2020.300633040:5(41-56)Online publication date: 24-Aug-2020

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media