research-article

An Augmented Reality Online Assistance Platform for Repair Tasks

Authors:

Hussein Al Osman,

Jochen LangAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 17, Issue 2

Article No.: 50, Pages 1 - 23

https://doi.org/10.1145/3429285

Published: 11 May 2021 Publication History

Abstract

Our augmented reality online assistance platform enables an expert to specify 6DoF movements of a component and apply the geometrical and physical constraints in real-time. We track the real components on the expert’s side to monitor the operations of an expert. We leverage a remote rendering technique that we proposed previously to relieve the rendering burden of the augmented reality end devices. By conducting a user study, we show that the proposed method outperforms conventional instructional videos and sketches. The answers to the questionnaires show that the proposed method receives higher recommendation than sketching, and, compared to conventional instructional videos, is outstanding in terms of instruction clarity, preference, recommendation, and confidence of task completion. Moreover, as to the overall user experience, the proposed method has an advantage over the video method.

References

[1]

Matt Adcock, Dulitha Ranatunga, Ross Smith, and Bruce H. Thomas. 2014. Object-based touch manipulation for remote guidance of physical tasks. In Proceedings of the 2nd ACM Symposium on Spatial User Interaction. 113--122.

[2]

Paul J. Besl and Neil D. McKay. 1992. A method for registration of 3-D shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 2 (1992), 239--256.

Digital Library

[3]

Bhaskar Bhattacharya and Eliot Winer. 2015. A method for real-time generation of augmented reality work instructions via expert movements. In Proceedings of the Engineering Reality of Virtual Reality, Vol. 9392. 109--121.

[4]

Ócar Blanco-Novoa, Tiago M. FernáNdez-Caramés, Paula Fraga-Lamas, and Miguel A. Vilar-Montesinos. 2018. A practical evaluation of commercial industrial augmented reality systems in an industry 4.0 shipyard. IEEE Access 6 (2018), 8201--8218.

[5]

Silvia Blanco-Pons, Berta Carrión-Ruiz, and José Lerma. 2018. Augmented reality application assessment for disseminating rock art. Multimedia Tools and Applications 78 (2018), 10265--10286.

Digital Library

[6]

Pierre Boulanger. 2004. Application of augmented reality to industrial tele-training. In Proceedings of the 1st Canadian Conference on Computer and Robot Vision. 320--328.

[7]

Thomas P. Caudell and David W. Mizell. 1992. Augmented reality: An application of heads-up display technology to manual manufacturing processes. In Proceedings of the 25th Hawaii International Conference on System Sciences, Vol. II. 659--669.

[8]

Francesca De Crescenzio, Massimiliano Fantini, Franco Persiani, Luigi Di Stefano, Pietro Azzari, and Samuele Salti. 2011. Augmented reality for aircraft maintenance training and operations support. IEEE Computer Graphics and Applications 31, 1 (2011), 96--101.

Digital Library

[9]

Michele Gattullo, Giulia Wally Scurati, Michele Fiorentino, Antonio Emmanuele Uva, Francesco Ferrise, and Monica Bordegoni. 2019. Towards augmented reality manuals for industry 4.0: A methodology. Robotics and Computer-Integrated Manufacturing 56 (2019), 276--286.

[10]

Nirit Gavish, Teresa Gutiérrez, Sabine Webel, Jorge Rodríguez, Matteo Peveri, Uli Bockholt, and Franco Tecchia. 2015. Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interactive Learning Environments 23, 6 (2015), 778--798.

[11]

Google LLC. 2020. Build New Augmented Reality Experiences that Seamlessly Blend the Digital and Physical Worlds. Retrieved October 4, 2020 from https://developers.google.com/ar.

[12]

Google LLC. 2020. Download Android Studio and SDK tools | Android Developers. Retrieved July 14, 2020 from https://developer.android.com/studio.

[13]

Pavel Gurevich, Joel Lanir, Benjamin Cohen, and Ran Stone. 2012. TeleAdvisor: A versatile augmented reality tool for remote assistance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 619--622.

Digital Library

[14]

Matthias Haringer and Holger Regenbrecht. 2002. A pragmatic approach to augmented reality authoring. In Proceedings of the International Symposium on Mixed and Augmented Reality. 237--245.

Digital Library

[15]

Roger Harrabin. 2019. EU brings in ‘right to repair’ rules for appliances—BBC News. Retrieved April 30, 2020 from https://www.bbc.com/news/business-49884827.

[16]

Steven J. Henderson and Steven K. Feiner. 2009. Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret. In Proceedings of the 8th IEEE International Symposium on Mixed and Augmented Reality. 135--144.

[17]

Steven J. Henderson and Steven K. Feiner. 2011. Augmented reality in the psychomotor phase of a procedural task. In Proceedings of the 10th IEEE International Symposium on Mixed and Augmented Reality. 191--200.

[18]

Xueshi Hou, Yao Lu, and Sujit Dey. 2017. Wireless VR/AR with edge/cloud computing. In Proceedings of the 26th International Conference on Computer Communication and Networks. 1--8.

[19]

jMonkeyEngine. 2020. jMonkeyEngine. Retrieved August 16, 2020 from https://jmonkeyengine.org.

[20]

Juhwan Lee, Sangwon Hwang, Jisun Lee, and Seungwoo Kang. 2020. Comparative performance characterization of mobile AR frameworks in the context of AR-based grocery shopping applications. Applied Sciences 10, 4 (2020), 1547--1561.

[21]

MAXST Co., Ltd. 2019. MAXST | Technology company specialized in AR to make a better augmented reality world.Retrieved October 4, 2020 from http://maxst.com.

[22]

Antonija Mitrovic, Brent Martin, Pramuditha Suraweera, Konstantin Zakharov, Nancy Milik, Holl, Jay, and Nicholas Mcguigan. 2009. ASPIRE: An authoring system and deployment environment for constraint-based tutors. International Journal of Artificial Intelligence in Education 19, 2 (2009), 155--188.

Digital Library

[23]

Ohan Oda, Carmine Elvezio, Mengu Sukan, Steven Feiner, and Barbara Tversky. 2015. Virtual replicas for remote assistance in virtual and augmented reality. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. 405--415.

Digital Library

[24]

Riccardo Palmarini, John Ahmet Erkoyuncu, Rajkumar Roy, and Hosein Torabmostaedi. 2018. A systematic review of augmented reality applications in maintenance. Robotics and Computer-Integrated Manufacturing 49 (2018), 215--228.

[25]

Philip Lamb. 2020. ARToolKit Home Page. Retrieved August 16, 2020 from http://www.hitl.washington.edu/artoolkit/.

[26]

PTC. 2020. Vuforia—engine. Retrieved April 30, 2020 from https://engine.vuforia.com/engine.

[27]

Dulitha Ranatunga, Matt Adcock, David Feng, and Bruce Thomas. 2013. Towards object based manipulation in remote guidance. In IEEE International Symposium on Mixed and Augmented Reality. 1--6.

[28]

RE’FLEKT GmbH. 2019. Augmented reality platform for maintenance operations and training. Retrieved April 30, 2020 from https://www.re-flekt.com/reflekt-one.

[29]

Maximilian Speicher, Kristina Tenhaft, Simon Heinen, and Harry Handorf. 2015. Enabling industry 4.0 with holobuilder. In INFORMATIK. 1561--1575.

[30]

Hongling Sun, Yue Liu, Zhenliang Zhang, Xiaoxu Liu, and Yongtian Wang. 2018. Employing different viewpoints for remote guidance in a collaborative augmented environment. In Proceedings of the 6th International Symposium of Chinese CHI. 64--70.

Digital Library

[31]

Lu Sun, Hussein Al Osman, and Jochen Lang. 2019. A hybrid remote rendering method for mobile applications. Multimedia Tools and Applications 79, 5 (2019), 1--26.

[32]

Ivan E. Sutherland. 1968. A head-mounted three dimensional display. In Proceedings of the Fall Joint Computer Conference, Part I. 757--764.

Digital Library

[33]

Anna Syberfeldt, Oscar Danielsson, and Patrik Gustavsson. 2017. Augmented reality smart glasses in the smart factory: Product evaluation guidelines and review of available products. IEEE Access 5 (2017), 9118--9130.

[34]

John W. Tukey. 1949. Comparing individual means in the analysis of variance. Biometrics 5, 2 (1949), 99--114.

[35]

Ming-Jen Wang, Chien-Hao Tseng, and Cherng-Yeu Shen. 2010. An easy to use augmented reality authoring tool for use in examination purpose. In Human-Computer Interaction. 285--288.

[36]

Xiangyu Wang, Sohkhim K. Ong, and Andrew Y. C. Nee. 2016. A comprehensive survey of augmented reality assembly research. Advances in Manufacturing 4 (2016), 1--22.

[37]

Sabine Webel, Uli Bockholt, Timo Engelke, Nirit Gavish, Manuel Olbrich, and Carsten Preusche. 2013. An augmented reality training platform for assembly and maintenance skills. Robotics and Autonomous Systems 61, 4 (2013), 398--403.

Digital Library

Cited By

Guo KTan XZhu XGuo SXi Z(2024)ATMNet: Adaptive Texture Migration Network for Guided Depth Super-ResolutionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/370264221:1(1-21)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1145/3702642
Richard RMcKay MMolyneaux H(2024)Legal, Technical and User Perspectives on the Right to ModifyHCI International 2024 – Late Breaking Papers10.1007/978-3-031-76821-7_23(327-341)Online publication date: 17-Dec-2024
https://doi.org/10.1007/978-3-031-76821-7_23
Fidalgo CYan YCho HSousa MLindlbauer DJorge J(2023)A Survey on Remote Assistance and Training in Mixed Reality EnvironmentsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324708129:5(2291-2303)Online publication date: 1-May-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3247081
Show More Cited By

Index Terms

An Augmented Reality Online Assistance Platform for Repair Tasks
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User studies
    2. Interaction paradigms
      1. Mixed / augmented reality
  2. Interaction design
    1. Interaction design process and methods
      1. Activity centered design
      2. Scenario-based design

Recommendations

Haptics in Augmented Reality
ICMCS '99: Proceedings of the IEEE International Conference on Multimedia Computing and Systems - Volume 2

An augmented reality system merges synthetic sensory information into a user's perception of a three-dimensional environment. An important performance goal for an augmented reality system is that the user perceives a single seamless environment. In most ...
Augmented reality as perceptual reality
VSMM'06: Proceedings of the 12th international conference on Interactive Technologies and Sociotechnical Systems

As shown in Paul Milgram et al’s Reality-Virtuality Continuum (1994), Augmented Reality occupies a very unique status in the spectrum of Mixed Reality. Unlike Virtual Reality, which is completely made up of the virtual and has been the most important ...
Multimodal augmented reality: the norm rather than the exception
MVAR '16: Proceedings of the 2016 workshop on Multimodal Virtual and Augmented Reality

Augmented reality (AR) is commonly seen as a technology that overlays virtual imagery onto a participant's view of the world. In line with this, most AR research is focused on what we see. In this paper, we challenge this focus on vision and make a case ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 17, Issue 2

May 2021

410 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3461621

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 May 2021

Accepted: 01 October 2020

Revised: 01 October 2020

Received: 01 May 2020

Published in TOMM Volume 17, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
330
Total Downloads

Downloads (Last 12 months)63
Downloads (Last 6 weeks)8

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Guo KTan XZhu XGuo SXi Z(2024)ATMNet: Adaptive Texture Migration Network for Guided Depth Super-ResolutionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/370264221:1(1-21)Online publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1145/3702642
Richard RMcKay MMolyneaux H(2024)Legal, Technical and User Perspectives on the Right to ModifyHCI International 2024 – Late Breaking Papers10.1007/978-3-031-76821-7_23(327-341)Online publication date: 17-Dec-2024
https://doi.org/10.1007/978-3-031-76821-7_23
Fidalgo CYan YCho HSousa MLindlbauer DJorge J(2023)A Survey on Remote Assistance and Training in Mixed Reality EnvironmentsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324708129:5(2291-2303)Online publication date: 1-May-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3247081
Djenouri YBelhadi ADjenouri DSrivastava GLin J(2023)A Secure Intelligent System for Internet of Vehicles: Case Study on Traffic ForecastingIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.324354224:11(13218-13227)Online publication date: 27-Feb-2023
https://dl.acm.org/doi/10.1109/TITS.2023.3243542
Eswaran MGulivindala AInkulu ARaju Bahubalendruni M(2023)Augmented reality-based guidance in product assembly and maintenance/repair perspectiveExpert Systems with Applications: An International Journal10.1016/j.eswa.2022.118983213:PAOnline publication date: 1-Mar-2023
https://dl.acm.org/doi/10.1016/j.eswa.2022.118983
Mezair TDjenouri YBelhadi ASrivastava GLin J(2022)Towards an Advanced Deep Learning for the Internet of Behaviors: Application to Connected VehiclesACM Transactions on Sensor Networks10.1145/352619219:2(1-18)Online publication date: 20-Dec-2022
https://dl.acm.org/doi/10.1145/3526192
Liu JWang PTversky BFeiner S(2022)Adaptive Visual Cues for Guiding a Bimanual Unordered Task in Virtual Reality2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR55827.2022.00059(431-440)Online publication date: Oct-2022
https://doi.org/10.1109/ISMAR55827.2022.00059
Adriana Cárdenas-Robledo LHernández-Uribe ÓReta CAntonio Cantoral-Ceballos J(2022)Extended reality applications in industry 4.0. – A systematic literature reviewTelematics and Informatics10.1016/j.tele.2022.10186373:COnline publication date: 1-Sep-2022
https://dl.acm.org/doi/10.1016/j.tele.2022.101863

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Issue’s Table of Contents