skip to main content
10.1145/3313831.3376688acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

An Exploratory Study of Augmented Reality Presence for Tutoring Machine Tasks

Published: 23 April 2020 Publication History

Abstract

Machine tasks in workshops or factories are often a compound sequence of local, spatial, and body-coordinated human-machine interactions. Prior works have shown the merits of video-based and augmented reality (AR) tutoring systems for local tasks. However, due to the lack of a bodily representation of the tutor, they are not as effective for spatial and body-coordinated interactions. We propose avatars as an additional tutor representation to the existing AR instructions. In order to understand the design space of tutoring presence for machine tasks, we conduct a comparative study with 32 users. We aim to explore the strengths/limitations of the following four tutor options: video, non-avatar-AR, half-body+AR, and full-body+AR. The results show that users prefer the half-body+AR overall, especially for the spatial interactions. They have a preference for the full-body+AR for the body-coordinated interactions and the non-avatar-AR for the local interactions. We further discuss and summarize design recommendations and insights for future machine task tutoring systems.

Supplemental Material

MP4 File
Preview video
MP4 File
Supplemental video

References

[1]
2019. Oculus. (2019). https://www.oculus.com/.
[2]
Heni Ben Amor, Gerhard Neumann, Sanket Kamthe, Oliver Kroemer, and Jan Peters. 2014. Interaction primitives for human-robot cooperation tasks. In 2014 IEEE international conference on robotics and automation (ICRA). IEEE, 2831--2837.
[3]
Fraser Anderson, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2013. YouMove: enhancing movement training with an augmented reality mirror. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 311--320.
[4]
armedia. 2019. I-Mechanic, the AR App that turns yourself into a Mechanic. (2019). Retrieved September 1, 2019 from http://www.armedia.it/i-mechanic.
[5]
Aaron Bangor, Philip T Kortum, and James T Miller. 2008. An empirical evaluation of the system usability scale. Intl. Journal of Human--Computer Interaction 24, (2008), 574--594.
[6]
Connelly Barnes, David E Jacobs, Jason Sanders, Dan B Goldman, Szymon Rusinkiewicz, Adam Finkelstein, and Maneesh Agrawala. 2008. Video puppetry: a performative interface for cutout animation. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 124.
[7]
Frank Biocca, C. Harms, and Jennifer Gregg. 2001. The Networked Minds Measure of Social Presence: Pilot Test of the Factor Structure and Concurrent Validity. 4th annual International Workshop on Presence, Philadelphia (01 2001).
[8]
Jean-Rémy Chardonnet, Guillaume Fromentin, and José Outeiro. 2017. Augmented reality as an aid for the use of machine tools. Res. & Sci. Today 13 (2017), 25.
[9]
Pei-Yu Chi, Sally Ahn, Amanda Ren, Mira Dontcheva, Wilmot Li, and Björn Hartmann. 2012. MixT: automatic generation of step-by-step mixed media tutorials. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 93--102.
[10]
Pei-Yu Chi, Joyce Liu, Jason Linder, Mira Dontcheva, Wilmot Li, and Bjoern Hartmann. 2013. Democut: generating concise instructional videos for physical demonstrations. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 141--150.
[11]
Pei-Yu Peggy Chi, Daniel Vogel, Mira Dontcheva, Wilmot Li, and Björn Hartmann. 2016. Authoring illustrations of human movements by iterative physical demonstration. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 809--820.
[12]
Philo Tan Chua, Rebecca Crivella, Bo Daly, Ning Hu, Russ Schaaf, David Ventura, Todd Camill, Jessica Hodgins, and Randy Pausch. 2003. Training for physical tasks in virtual environments: Tai Chi. In IEEE Virtual Reality, 2003. Proceedings. IEEE, 87--94.
[13]
Dima Damen, Teesid Leelasawassuk, and Walterio Mayol-Cuevas. 2016. You-Do, I-Learn: Egocentric unsupervised discovery of objects and their modes of interaction towards video-based guidance. Computer Vision and Image Understanding 149 (2016), 98--112.
[14]
Francesca De Crescenzio, Massimiliano Fantini, Franco Persiani, Luigi Di Stefano, Pietro Azzari, and Samuele Salti. 2010. Augmented reality for aircraft maintenance training and operations support. IEEE Computer Graphics and Applications 31, 1 (2010), 96--101.
[15]
Gino Dini and Michela Dalle Mura. 2015. Application of augmented reality techniques in through-life engineering services. Procedia Cirp 38 (2015), 14--23.
[16]
M-P Dubuisson and Anil K Jain. 1994. A modified Hausdorff distance for object matching. In Proceedings of 12th international conference on pattern recognition, Vol. 1. IEEE, 566--568.
[17]
Daniel Eckhoff, Christian Sandor, Christian Lins, Ulrich Eck, Denis Kalkofen, and Andreas Hein. 2018. TutAR: augmented reality tutorials for hands-only procedures. In Proceedings of the 16th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry. ACM, 8.
[18]
Allen J Fairchild, Simon P Campion, Arturo S García, Robin Wolff, Terrence Fernando, and David J Roberts. 2016. A mixed reality telepresence system for collaborative space operation. IEEE Transactions on Circuits and Systems for Video Technology 27, 4 (2016), 814--827.
[19]
FinalIK. 2019. FinalIK. (2019). Retrieved September 1, 2019 from https://assetstore.unity.com/packages/ tools/animation/final-ik-14290.
[20]
Markus Funk. 2016. Augmented reality at the workplace: a context-aware assistive system using in-situ projection. (2016).
[21]
Dominic Gorecky, Mohamed Khamis, and Katharina Mura. 2017. Introduction and establishment of virtual training in the factory of the future. International Journal of Computer Integrated Manufacturing 30, 1 (2017), 182--190.
[22]
Dominic Gorecky, Mathias Schmitt, Matthias Loskyll, and Detlef Zühlke. 2014. Human-machine-interaction in the industry 4.0 era. In 2014 12th IEEE international conference on industrial informatics (INDIN). Ieee, 289--294.
[23]
Michihiko Goto, Yuko Uematsu, Hideo Saito, Shuji Senda, and Akihiko Iketani. 2010. Task support system by displaying instructional video onto AR workspace. In 2010 IEEE International Symposium on Mixed and Augmented Reality. IEEE, 83--90.
[24]
Ankit Gupta, Maneesh Agrawala, Brian Curless, and Michael Cohen. 2014. Motionmontage: A system to annotate and combine motion takes for 3d animations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2017--2026.
[25]
Ankit Gupta, Dieter Fox, Brian Curless, and Michael Cohen. 2012. DuploTrack: a real-time system for authoring and guiding duplo block assembly. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 389--402.
[26]
Ping-Hsuan Han, Kuan-Wen Chen, Chen-Hsin Hsieh, Yu-Jie Huang, and Yi-Ping Hung. 2016. Ar-arm: Augmented visualization for guiding arm movement in the first-person perspective. In Proceedings of the 7th Augmented Human International Conference 2016. ACM, 31.
[27]
Ping-Hsuan Han, Yang-Sheng Chen, Yilun Zhong, Han-Lei Wang, and Yi-Ping Hung. 2017. My Tai-Chi coaches: an augmented-learning tool for practicing Tai-Chi Chuan. In Proceedings of the 8th Augmented Human International Conference. ACM, 25.
[28]
Ping-Hsuan Han, Jia-Wei Lin, Chen-Hsin Hsieh, Jhih-Hong Hsu, and Yi-Ping Hung. 2018. tARget: limbs movement guidance for learning physical activities with a video see-through head-mounted display. In ACM SIGGRAPH 2018 Posters. ACM, 26.
[29]
Robert Held, Ankit Gupta, Brian Curless, and Maneesh Agrawala. 2012. 3D puppetry: a kinect-based interface for 3D animation. In UIST. Citeseer, 423--434.
[30]
Steven J Henderson and Steven K Feiner. 2011. Augmented reality in the psychomotor phase of a procedural task. In 2011 10th IEEE International Symposium on Mixed and Augmented Reality. IEEE, 191--200.
[31]
Thuong N Hoang, Martin Reinoso, Frank Vetere, and Egemen Tanin. 2016. Onebody: remote posture guidance system using first person view in virtual environment. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction. ACM, 25.
[32]
Jean-Michel Hoc. 2001. Towards a cognitive approach to human--machine cooperation in dynamic situations. International journal of human-computer studies 54, 4 (2001), 509--540.
[33]
Geun-Sik Jo, Kyeong-Jin Oh, Inay Ha, Kee-Sung Lee, Myung-Duk Hong, Ulrich Neumann, and Suya You. 2014. A unified framework for augmented reality and knowledge-based systems in maintaining aircraft. In Twenty-Sixth IAAI Conference.
[34]
Juho Kim, Phu Tran Nguyen, Sarah Weir, Philip J Guo, Robert C Miller, and Krzysztof Z Gajos. 2014. Crowdsourcing step-by-step information extraction to enhance existing how-to videos. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 4017--4026.
[35]
Kangsoo Kim, Gerd Bruder, and Greg Welch. 2017. Exploring the effects of observed physicality conflicts on real-virtual human interaction in augmented reality. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology. ACM, 31.
[36]
Seungwon Kim, Gun Lee, Weidong Huang, Hayun Kim, Woontack Woo, and Mark Billinghurst. 2019. Evaluating the Combination of Visual Communication Cues for HMD-based Mixed Reality Remote Collaboration. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 173.
[37]
Yongkwan Kim and Seok-Hyung Bae. 2016. SketchingWithHands: 3D sketching handheld products with first-person hand posture. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 797--808.
[38]
Bokyung Lee, Minjoo Cho, Joonhee Min, and Daniel Saakes. 2016. Posing and acting as input for personalizing furniture. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction. ACM, 44.
[39]
David Lindlbauer, Anna Maria Feit, and Otmar Hilliges. 2019. Context-Aware Online Adaptation of Mixed Reality Interfaces. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. ACM, 147--160.
[40]
Matthias Loskyll, Ines Heck, Jochen Schlick, and Michael Schwarz. 2012. Context-based orchestration for control of resource-efficient manufacturing processes. Future Internet 4, 3 (2012), 737--761.
[41]
Alejandro Monroy Reyes, Osslan Osiris Vergara Villegas, Erasmo Miranda Bojórquez, Vianey Guadalupe Cruz Sánchez, and Manuel Nandayapa. 2016. A mobile augmented reality system to support machinery operations in scholar environments. Computer Applications in Engineering Education 24, 6 (2016), 967--981.
[42]
Christian Murlowski, Florian Daiber, Felix Kosmalla, and Antonio Krüger. 2019. Slackliner 2.0: Real-time Training Assistance through Life-size Feedback. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, INT012.
[43]
Noitom. 2019. Perception Neuron by Noitom. (2019). Retrieved September 1, 2019 from https://neuronmocap.com.
[44]
SK Ong and ZB Wang. 2011. Augmented assembly technologies based on 3D bare-hand interaction. CIRP annals 60, 1 (2011), 1--4.
[45]
Masaki Oshita, Takumi Inao, Tomohiko Mukai, and Shigeru Kuriyama. 2018. Self-training system for tennis shots with motion feature assessment and visualization. In 2018 International Conference on Cyberworlds (CW). IEEE, 82--89.
[46]
Tomislav Pejsa, Julian Kantor, Hrvoje Benko, Eyal Ofek, and Andrew Wilson. 2016. Room2room: Enabling life-size telepresence in a projected augmented reality environment. In Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing. ACM, 1716--1725.
[47]
Amaury Peniche, Christian Diaz, Helmuth Trefftz, and Gabriel Paramo. 2012. Combining virtual and augmented reality to improve the mechanical assembly training process in manufacturing. In American Conference on applied mathematics. 292--297.
[48]
Thammathip Piumsomboon, Gun A Lee, Jonathon D Hart, Barrett Ens, Robert W Lindeman, Bruce H Thomas, and Mark Billinghurst. 2018. Mini-me: an adaptive avatar for mixed reality remote collaboration. In Proceedings of the 2018 CHI conference on human factors in computing systems. ACM, 46.
[49]
Thammathip Piumsomboon, Gun A Lee, Andrew Irlitti, Barrett Ens, Bruce H Thomas, and Mark Billinghurst. 2019. On the Shoulder of the Giant: A Multi-Scale Mixed Reality Collaboration with 360 Video Sharing and Tangible Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 228.
[50]
David Porfirio, Evan Fisher, Allison Sauppé, Aws Albarghouthi, and Bilge Mutlu. 2019. Bodystorming Human-Robot Interactions. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology.
[51]
D Preuveneers. 2015. The GhostHands UX: telementoring with hands-on augmented reality instruction. In Workshop Proceedings of the 11th International Conference on Intelligent Environments, Vol. 19. IOS Press, 236.
[52]
Nazmus Saquib, Rubaiat Habib Kazi, Li-Yi Wei, and Wilmot Li. 2019. Interactive Body-Driven Graphics for Augmented Video Performance. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 622.
[53]
Jeff Sauro. 2012. 10 THINGS TO KNOW ABOUT THE SINGLE EASE QUESTION (SEQ). (2012). Retrieved September 1, 2019 from https://measuringu.com/seq10/.
[54]
Jeff Sauro and Joseph S Dumas. 2009. Comparison of three one-question, post-task usability questionnaires. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 1599--1608.
[55]
Eldon Schoop, Michelle Nguyen, Daniel Lim, Valkyrie Savage, Sean Follmer, and Björn Hartmann. 2016. Drill Sergeant: Supporting physical construction projects through an ecosystem of augmented tools. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 1607--1614.
[56]
Rajinder Sodhi, Hrvoje Benko, and Andrew Wilson. 2012. LightGuide: projected visualizations for hand movement guidance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 179--188.
[57]
Stereolabs. 2019. ZED Mini Stereo Camera - Stereolabs. (2019). Retrieved September 1, 2019 from https://www.stereolabs.com/zed-mini/.
[58]
Lucy Suchman. 2007. Human-machine reconfigurations: Plans and situated actions. Cambridge University Press.
[59]
Guoyu Sun, Paisarn Muneesawang, M Kyan, Haiyan Li, Ling Zhong, Nan Dong, Bruce Elder, and Ling Guan. 2014. An advanced computational intelligence system for training of ballet dance in a cave virtual reality environment. In 2014 IEEE International Symposium on Multimedia. IEEE, 159--166.
[60]
Balasaravanan Thoravi Kumaravel, Fraser Anderson, George Fitzmaurice, Bjoern Hartmann, and Tovi Grossman. 2019a. Loki: Facilitating Remote Instruction of Physical Tasks Using Bi-Directional Mixed-Reality Telepresence. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. ACM, 161--174.
[61]
Balasaravanan Thoravi Kumaravel, Cuong Nguyen, Stephen DiVerdi, and Björn Hartmann. 2019b. TutoriVR: A Video-Based Tutorial System for Design Applications in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, 284.
[62]
Unity. 2019. Unity Real-Time Development Platform. (2019). Retrieved September 1, 2019 from https://unity.com/.
[63]
David Vogt, Simon Stepputtis, Steve Grehl, Bernhard Jung, and Heni Ben Amor. 2017. A system for learning continuous human-robot interactions from human-human demonstrations. In 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2882--2889.
[64]
VRChat. 2019. Create And Play in Virtual Worlds. (2019). Retrieved September 1, 2019 from https://www.vrchat.net/.
[65]
Sabine Webel, Uli Bockholt, Timo Engelke, Nirit Gavish, Manuel Olbrich, and Carsten Preusche. 2013. An augmented reality training platform for assembly and maintenance skills. Robotics and Autonomous Systems 61, 4 (2013), 398--403.
[66]
Giles Westerfield. 2012. Intelligent augmented reality training for assembly and maintenance. (2012).
[67]
Tomasz Wójcicki. 2014. Supporting the diagnostics and the maintenance of technical devices with augmented reality. Diagnostyka 15, 1 (2014), 43--47.
[68]
Shuo Yan, Gangyi Ding, Zheng Guan, Ningxiao Sun, Hongsong Li, and Longfei Zhang. 2015. OutsideMe: Augmenting Dancer's External Self-Image by Using A Mixed Reality System. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 965--970.
[69]
Yupeng Zhang, Teng Han, Zhimin Ren, Nobuyuki Umetani, Xin Tong, Yang Liu, Takaaki Shiratori, and Xiang Cao. 2013. BodyAvatar: creating freeform 3D avatars using first-person body gestures. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 387--396.
[70]
J Zhu, Soh-Khim Ong, and Andrew YC Nee. 2015. A context-aware augmented reality assisted maintenance system. International Journal of Computer Integrated Manufacturing 28, 2 (2015), 213--225.
[71]
Zhiwei Zhu, Vlad Branzoi, Michael Wolverton, Glen Murray, Nicholas Vitovitch, Louise Yarnall, Girish Acharya, Supun Samarasekera, and Rakesh Kumar. 2014. AR-mentor: Augmented reality based mentoring system. In 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 17--22.

Cited By

View all
  • (2024)Development and Evaluation of Training Scenarios for the Use of Immersive Assistance SystemsApplied System Innovation10.3390/asi70500737:5(73)Online publication date: 26-Aug-2024
  • (2024)RubiXR: Demonstration of dynamic task augmentation through co-design of interactive 3D content and 3D user interfacesProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3688875(1-3)Online publication date: 7-Oct-2024
  • (2024)TAGGAR: General-Purpose Task Guidance from Natural Language in Augmented Reality using Vision-Language ModelsProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682095(1-12)Online publication date: 7-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
April 2020
10688 pages
ISBN:9781450367080
DOI:10.1145/3313831
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 April 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. augmented reality
  2. avatar tutor
  3. exploratory study
  4. machine task
  5. tutoring system design

Qualifiers

  • Research-article

Funding Sources

Conference

CHI '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)262
  • Downloads (Last 6 weeks)6
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Development and Evaluation of Training Scenarios for the Use of Immersive Assistance SystemsApplied System Innovation10.3390/asi70500737:5(73)Online publication date: 26-Aug-2024
  • (2024)RubiXR: Demonstration of dynamic task augmentation through co-design of interactive 3D content and 3D user interfacesProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3688875(1-3)Online publication date: 7-Oct-2024
  • (2024)TAGGAR: General-Purpose Task Guidance from Natural Language in Augmented Reality using Vision-Language ModelsProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682095(1-12)Online publication date: 7-Oct-2024
  • (2024)AudioMove: Applying the Spatial Audio to Multi-Directional Limb Exercise GuidanceProceedings of the ACM on Human-Computer Interaction10.1145/36764898:MHCI(1-26)Online publication date: 24-Sep-2024
  • (2024)Investigation into Recording, Replay and Simulation of Interactions in Virtual RealityProceedings of the 2024 8th International Conference on Virtual and Augmented Reality Simulations10.1145/3657547.3657557(13-21)Online publication date: 14-Mar-2024
  • (2024)Virtual Triplets: A Mixed Modal Synchronous and Asynchronous Collaboration with Human-Agent Interaction in Virtual RealityExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650766(1-8)Online publication date: 11-May-2024
  • (2024)Fast-Forward Reality: Authoring Error-Free Context-Aware Policies with Real-Time Unit Tests in Extended RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642158(1-17)Online publication date: 11-May-2024
  • (2024)Design Space of Visual Feedforward And Corrective Feedback in XR-Based Motion Guidance SystemsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642143(1-15)Online publication date: 11-May-2024
  • (2024)AMMA: Adaptive Multimodal Assistants Through Automated State Tracking and User Model-Directed Guidance Planning2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00108(892-902)Online publication date: 16-Mar-2024
  • (2024)Visual Cue Based Corrective Feedback for Motor Skill Training in Mixed Reality: A SurveyIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.322799930:7(3121-3134)Online publication date: 1-Jul-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media