Skip to main content
SpringerLink
Log in

An Industry-Adapted AR Training Method for Manual Assembly Operations

  • Conference paper
  • First Online:
HCI International 2021 - Late Breaking Papers: Multimodality, eXtended Reality, and Artificial Intelligence (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 13095))

Included in the following conference series:

Abstract

The adoption of Augmented Reality (AR) in the industry is in early stages, mainly due to technological and organizational limitations. This research work, carried out in a manufacturing factory, aims at providing an effective AR training method for manual assembly, adapted for industrial context. We define the 2W1H (What, Where, How) principle to formalize the description of any manual assembly operation in AR, independently on its type or complexity. Further, we propose a head-mounted display (HMD)-based method for conveying the manual assembly information, which relies on low-cost visual assets - i.e. text, image, video and predefined auxiliary content. We evaluate the effectiveness and usability of our proposal by conducting a field experiment with 30 participants. Additionally, we comparatively evaluate two sets of AR instructions, low-cost vs. CAD-based, to identify benefits of conveying assembly information by using CAD models. Our objective evaluation indicates that (i) manual assembly expertise can be effectively delivered by using spatially registered low-cost visual assets and that (ii) CAD-based instructions lead to faster assembly times, but persuade lower user attentiveness, eventually leading to higher error rates. Finally, by considering the diminishing utility of the AR instructions over three assembly cycles, we question the worthiness of authoring CAD-based AR instructions for similar industrial scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Dey, A., Billinghurst, M., Lindeman, R.W., Swan, J.E.: A systematic review of 10 Years of augmented reality usability studies: 2005 to 2014, Front. Robot. AI, 5 (2018). https://doi.org/10.3389/frobt.2018.00037

  2. de Souza Cardoso, L.F., Mariano, F.C.M.Q., Zorzal, E.R.: A survey of industrial augmented reality, Comput. Ind. Eng. 139, 106159 (2020). https://doi.org/10.1016/j.cie.2019.106159

  3. Röltgen, D., Dumitrescu, P.R., Stief, P., Dantan, J., Etienne, A., Siadat, A.: Sciencedirect functional and physical architecture of existing products for an assembly product family identification classification of industrial augmented reality use cases classification of industrial augmented reality use cases. Proc. CIRP 91, 93–100 (2020). https://doi.org/10.1016/j.procir.2020.01.137

    Article  Google Scholar 

  4. Bellalouna, P.F.: Industrial use cases for augmented reality application, pp. 10–18 (2020). https://doi.org/10.1109/CogInfoCom50765.2020.9237882

  5. Bosch, T., Könemann, R., de Cock, H., van Rhijn, G.: The effects of projected versus display instructions on productivity, quality and workload in a simulated assembly task. In: Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA ’17). Association for Computing Machinery, pp. 412–415. New York, NY, USA (2017). https://doi.org/10.1145/3056540.3076189

  6. Sanna, A., Manuri, F., Lamberti, F., Paravati, G., Pezzolla, P.: Using handheld devices to support augmented reality-based maintenance and assembly tasks. In: 2015 IEEE International Conference on Consumer Electronics (ICCE), pp. 178–179. Las Vegas, NV (2015). https://doi.org/10.1109/ICCE.2015.7066370

  7. Hahn, J., Ludwig, B., Wolff, C.: Augmented reality-based training of the PCB assembly process. In: Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia (MUM ’15). Association for Computing Machinery, pp. 395–399. New York, NY, USA (2015). https://doi.org/10.1145/2836041.2841215

  8. Funk, M., Kosch, T., Schmidt, A.: Interactive worker assistance: comparing the effects of in-situ projection, head-mounted displays, tablet, and paper instructions. In: UbiComp 2016 – Proceedings 2016 ACM International Jt. Conference Pervasive Ubiquitous Computing, pp. 934–939 (2016) https://doi.org/10.1145/2971648.2971706

  9. Masood, T., Egger, J.: Augmented reality in support of industry 4.0—implementation challenges and success factors, Robot. Comput. Integr. Manuf. 58, 181–195 (2019). https://doi.org/10.1016/j.rcim.2019.02.003

  10. Martinetti, A., Marques, H., Singh, S., Dongen, L.: Reflections on the limited pervasiveness of augmented reality in industrial sectors. Appl. Sci. 9, 3382 (2019). https://doi.org/10.3390/app9163382

    Article  Google Scholar 

  11. Merino, L., Schwarzl, M., Kraus, M., Sedlmair, M., Schmalstieg, D., Weiskopf, D.: Evaluating mixed and augmented reality: a systematic literature review (2009–2019) (2020) https://doi.org/10.1109/ISMAR50242.2020.00069

  12. Egger, J., Masood, T.: Augmented reality in support of intelligent manufacturing – a systematic literature review, Comput. Ind. Eng. 140, 106195 (2020). https://doi.org/10.1016/j.cie.2019.106195

  13. Kim, K., Billinghurst, M., Bruder, G., Duh, H.B.L., Welch, G.F.: Revisiting trends in augmented reality research: a review of the 2nd Decade of ISMAR (2008–2017). IEEE Trans. Vis. Comput. Graph. 24(11), 2947–2962 (2018). https://doi.org/10.1109/TVCG.2018.2868591

    Article  Google Scholar 

  14. Wang, X., Ong, S.K., Nee, A.Y.C.: A comprehensive survey of augmented reality assembly research. Adv. Manuf. 4(1), 1–22 (2016). https://doi.org/10.1007/s40436-015-0131-4

    Article  Google Scholar 

  15. Gattullo, M., Evangelista, A., Uva, A.E., Fiorentino, M., Gabbard, J.: What, How, and Why are visual assets used in industrial augmented reality? a systematic review and classification in maintenance, assembly, and training (from 1997 to 2019), IEEE Trans. Vis. Comput. Graph. 2626, 1 (2020). https://doi.org/10.1109/tvcg.2020.3014614

  16. Caudell, T.P., Mizell, D.W.: Augmented reality: an application of heads-up display technology to manual manufacturing processes, 2, 659–669 (2003). https://doi.org/10.1109/hicss.1992.183317

  17. Tang, A., Owen, C., Biocca, F.: Comparative effectiveness of augmented reality in object assembly (2003) https://doi.org/10.1145/642611.642626

  18. Loch, F., Quint, F., Brishtel, I.: Comparing video and augmented reality assistance in manual assembly. In: 2016 12th International Conference on Intelligent Environments (IE), pp. 147–150 (2016). https://doi.org/10.1109/IE.2016.31

  19. Henderson, S., Feiner, S.: Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Trans. Vis. Comput. Graph. 17(10), 1355–1368 (2011). https://doi.org/10.1109/TVCG.2010.245

    Article  Google Scholar 

  20. Ceruti, A., Marzocca, P., Liverani, A., Bil, C.: Maintenance in aeronautics in an industry 4.0 context: the role of augmented reality and additive manufacturing, J. Comput. Des. Eng. 6(4), 516–526 (2019) https://doi.org/10.1016/j.jcde.2019.02.001

  21. Lai, Z.H., Tao, W., Leu, M.C., Yin, Z.: Smart augmented reality instructional system for mechanical assembly towards worker-centered intelligent manufacturing, J. Manuf. Syst. 55, 69–81, (2020) https://doi.org/10.1016/j.jmsy.2020.02.010

  22. Werrlich, S., Daniel, A., Ginger, A., Nguyen, P.A., Notni, G.: Comparing HMD-based and paper-based training. In: Proceedings 2018 IEEE Internatioanl Symposium Mixed Augmental Reality, ISMAR 2018, pp. 134–142 (2019) https://doi.org/10.1109/ISMAR.2018.00046

  23. Gavish, N., et al.: Evaluating virtual reality and augmented reality training for industrial maintenance and assembly tasks. Interact. Learn. Environ. 23(6), 778–798 (2015). https://doi.org/10.1080/10494820.2013.815221

    Article  Google Scholar 

  24. Blattgerste, J., Renner, P., Strenge, B., Pfeiffer, T.: In-Situ instructions exceed side-by-side instructions in augmented reality assisted assembly. In: Proceedings of the 11th PErvasive Technologies Related to Assistive Environments Conference, pp. 133–140 (2018) https://doi.org/10.1145/3197768.3197778

  25. Funk, M., Mayer, S., Schmidt, A.: Using in-situ projection to support cognitively impaired workers at the workplace. In: ASSETS 2015 - Proceedings 17th Interanational ACM SIGACCESS Conference Computing Access, pp. 185–192 (2015). https://doi.org/10.1145/2700648.2809853.Des

  26. Mengoni, M., Ceccacci, S., Generosi, A., Leopardi, A.: Spatial augmented reality: an application for human work in smart manufacturing environment. Procedia Manuf. 17, 476–483 (2018). https://doi.org/10.1016/j.promfg.2018.10.072

    Article  Google Scholar 

  27. Uva, A.E., Gattullo, M., Manghisi, V.M., Spagnulo, D., Cascella, G.L., Fiorentino, M.: Evaluating the effectiveness of spatial augmented reality in smart manufacturing: a solution for manual working stations. Int. J. Adv. Manuf. Technol. 94(1–4), 509–521 (2017). https://doi.org/10.1007/s00170-017-0846-4

    Article  Google Scholar 

  28. Fiorentino, M., Uva, A.E., Gattullo, M., Debernardis, S., Monno, G.: Augmented reality on large screen for interactive maintenance instructions. Comput. Ind. 65(2), 270–278 (2014). https://doi.org/10.1016/j.compind.2013.11.004

    Article  Google Scholar 

  29. Lee, G.A., Hoff, W.: Enhancing first-person view task instruction videos with augmented reality cues, pp. 666–676, (2020). https://doi.org/10.1109/ISMAR50242.2020.00078

  30. Palmarini, R., Erkoyuncu, J.A., Roy, R., Torabmostaedi, H.: A systematic review of augmented reality applications in maintenance. In: Robotics and Computer-Integrated Manufacturing, vol. 49, pp. 215–228, ISSN 0736–5845 (2018). https://doi.org/10.1016/j.rcim.2017.06.002

  31. Quandt, M., Knoke, B., Gorldt, C., Freitag, M., Thoben, K.D.: General requirements for industrial augmented reality applications. Proc. CIRP 72, 1130–1135 (2018). https://doi.org/10.1016/j.procir.2018.03.061

    Article  Google Scholar 

  32. Nicolai, T., Sindt, T., Kenn, H., Witt, H.: Case study of wearable computing for aircraft maintenance. In: 3rd International Forum Applicable Wearable Computing, pp. 1–12 (2006)

    Google Scholar 

  33. Masood, T., Egger, J.: Adopting augmented reality in the age of industrial digitalisation, Comput. Ind., 115, 103112 (2020) https://doi.org/10.1016/j.compind.2019.07.002

  34. van Lopik, K., Sinclair, M., Sharpe, R., Conway, P., West, A.: Developing augmented reality capabilities for industry 4.0 small enterprises: lessons learnt from a content authoring case study, Comput. Ind., 117, 103208 (2020). https://doi.org/10.1016/j.compind.2020.103208

  35. HoloLens 2. https://www.microsoft.com/en-us/hololens/hardware. Accessed 6 Jul 2021

  36. Li, W., Wang, J., Jiao, S., Wang, M., Li, S.: Research on the visual elements of augmented reality assembly processes. Virtual Real. Intell. Hardw. 1(6), 622–634 (2019). https://doi.org/10.1016/j.vrih.2019.09.006

    Article  Google Scholar 

  37. Tainaka, K.: Guideline and tool for designing an assembly task support system using augmented reality, pp. 654–665 (2020) https://doi.org/10.1109/ISMAR50242.2020.00077

  38. PTC. Vuforia Expert Capture (2021). https://www.ptc.com/en/products/augmented-reality/vuforia-expert-capture. Accessed 27 April 2021

  39. Microsoft. Mixed Reality Dynamics 365 Guides (2021). https://dynamics.microsoft.com/en-us/mixed-reality/guides/. Accessed 27 Apr 2021

  40. Lu, F., Davari, S., Lisle, L., Li, Y., Bowman, D.A.: Glanceable AR: evaluating information access methods for head-worn augmented reality. In: 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 930–939 (2020) https://doi.org/10.1109/VR46266.2020.00113

  41. Muhammad Nizam, S., Zainal Abidin, R., Che Hashim, N., Lam, M.C., Arshad, H., Abd Majid, N.A.: A review of multimodal interaction technique in augmented reality environment. Int. J. Adv. Sci. Eng. Inf. Technol., 8(4–2), 1460 (2018) https://doi.org/10.18517/ijaseit.8.4-2.6824

  42. Irawati, S., Green, S., Billinghurst, M., Duenser, A., Ko, H.: An Evaluation of an Augmented Reality Multimodal Interface Using Speech and Paddle Gestures. In: Pan, Z., Cheok, A., Haller, M., Lau, R.W.H., Saito, H., Liang, R. (eds.) ICAT 2006. LNCS, vol. 4282, pp. 272–283. Springer, Heidelberg (2006). https://doi.org/10.1007/11941354_28

    Chapter  Google Scholar 

  43. Marquardt, A., Trepkowski, C., Eibich, T.D., Maiero, J., Kruijff, E., Schoning, J.: Comparing Non-Visual and Visual Guidance Methods for Narrow Field of View Augmented Reality Displays, IEEE Trans. Vis. Comput. Graph., 1 (2020). https://doi.org/10.1109/tvcg.2020.3023605

  44. Hanson, R., Falkenström, W., Miettinen, M.: Augmented reality as a means of conveying picking information in kit preparation for mixed-model assembly. Comput. Ind. Eng. 113, 570–575 (2017). https://doi.org/10.1016/j.cie.2017.09.048

    Article  Google Scholar 

  45. Introducing instinctual interactions. https://docs.microsoft.com/en-us/windows/mixed-reality/design/interaction-fundamentals. Accessed 06 Jul 2021

  46. Gabbard, J.L., Fitch, G.M., Kim, H.: Behind the glass: driver challenges and opportunities for AR automotive applications. Proc. IEEE 102(2), 124–136 (2014). https://doi.org/10.1109/JPROC.2013.2294642

    Article  Google Scholar 

  47. Unity 3D, v2019.4. https://unity3d.com/get-unity/download/archive

  48. Microsoft Mixed Reality Toolkit v2.4.0. https://github.com/microsoft/MixedRealityToolkit-Unity/releases/tag/v2.4.0. Accessed 06 Jul 2021

  49. https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html

  50. Hart, S.G.: Nasa-Task Load Index (NASA-TLX); 20 Years Later. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, pp. 904–908. https://doi.org/10.1177/154193120605000909

Download references

Author information

Authors and Affiliations

  1. IP Paris, Telecom SudParis, Evry, France

    Traian Lavric, Marius Preda & Titus Zaharia

  2. Elm Leblanc SAS, Drancy, France

    Traian Lavric & Emmanuel Bricard

Authors
  1. Traian Lavric
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Emmanuel Bricard
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marius Preda
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Titus Zaharia
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Traian Lavric .

Editor information

Editors and Affiliations

  1. University of Creteand Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece

    Constantine Stephanidis

  2. The Open University of Japan, Chiba, Japan

    Masaaki Kurosu

  3. U.S. Army Research Laboratory, Aberdeen Proving Ground, MD, USA

    Jessie Y. C. Chen

  4. U.S. Army Combat Capabilities Development Command Soldier Center, Orlando, FL, USA

    Gino Fragomeni

  5. Smart Future Initiative, Frankfurt am Main, Hessen, Germany

    Norbert Streitz

  6. Faculty of Arts and Science, Kyushu University, Fukuoka, Japan

    Shin'ichi Konomi

  7. Siemens (United States), Princeton, NJ, USA

    Helmut Degen

  8. Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece

    Stavroula Ntoa

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lavric, T., Bricard, E., Preda, M., Zaharia, T. (2021). An Industry-Adapted AR Training Method for Manual Assembly Operations. In: Stephanidis, C., et al. HCI International 2021 - Late Breaking Papers: Multimodality, eXtended Reality, and Artificial Intelligence. HCII 2021. Lecture Notes in Computer Science(), vol 13095. Springer, Cham. https://doi.org/10.1007/978-3-030-90963-5_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-90963-5_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-90962-8

  • Online ISBN: 978-3-030-90963-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation