Skip to main content
SpringerLink
Log in

Skillab - A Multimodal Augmented Reality Environment for Learning Manual Tasks

  • Conference paper
  • First Online:
Human-Computer Interaction – INTERACT 2023 (INTERACT 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14144))

Included in the following conference series:

  • 714 Accesses

Abstract

The paper investigates the usage of AR-based systems in teaching manual skills. We propose Skillab, a novel AR-based scaffolding system. It aids in the learning of manual work and functions as a multimodal immersive tool for feedback, including muscle actuation. As our initial investigation, we made a floor lamination tutorial. We evaluated our system’s performance and user experience in comparison to traditional paper instructions. With 20 participants, we conducted a between-group user study and obtained both qualitative and quantitative data. In terms of task performance, learnability, and overall user experience, Skillab significantly outperformed conventional paper instructions. In contrast to learning from paper instructions, Skillab training demonstrated a significant improvement in the systematic rating on the quality of the performed task. We believe that by demonstrating the potential of immersive multi-modal feedback technology for skill-building, researchers would be motivated to explore this area further.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://docs.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk2/features/ux-building-blocks/interactable?view=mrtkunity-2022-05, accessed on 14.07.2022.

  2. 2.

    https://docs.microsoft.com/en-us/windows/mixed-reality/mrtk-unity/mrtk2/features/ux-building-blocks/manipulation-handler?view=mrtkunity-2022-05, accessed on 14.07.2022.

References

  1. Anderson, F., Grossman, T., Matejka, J., Fitzmaurice, G.: YouMove: enhancing movement training with an augmented reality mirror. In: Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, pp. 311–320 (2013)

    Google Scholar 

  2. Azuma, R.T.: A Survey of Augmented Reality. Presence: Teleoperators Virtual Environ. 6(4), 355–385 (1997). https://doi.org/10.1162/pres.1997.6.4.355

  3. Barfield, W.: Fundamentals of Wearable Computers and Augmented Reality. CRC Press (2015)

    Google Scholar 

  4. Chang, G., Morreale, P., Medicherla, P.: Applications of augmented reality systems in education. In: Society for Information Technology & Teacher Education International Conference, pp. 1380–1385. Association for the Advancement of Computing in Education (AACE) (2010)

    Google Scholar 

  5. Colley, A., Leinonen, A., Forsman, M.T., Häkkilä, J.: Ems painter: co-creating visual art using electrical muscle stimulation. In: Proceedings of the Twelfth International Conference on Tangible, Embedded, and Embodied Interaction, pp. 266–270 (2018)

    Google Scholar 

  6. Collins, A., Brown, J.S., Newman, S.E.: Cognitive apprenticeship: teaching the crafts of reading, writing, and mathematics. In: Knowing, Learning, and Instruction, pp. 453–494. Routledge (2018)

    Google Scholar 

  7. Ebisu, A., Hashizume, S., Ochiai, Y.: Building a feedback loop between electrical stimulation and percussion learning. In: ACM SIGGRAPH 2018 Studio, pp. 1–2 (2018)

    Google Scholar 

  8. Faltaous, S., Koelle, M., Schneegass, S.: From perception to action: a review and taxonomy on electrical muscle stimulation in HCI. In: Proceedings of the 21st International Conference on Mobile and Ubiquitous Multimedia, pp. 159–171 (2022)

    Google Scholar 

  9. Gamble, J.: Modelling the invisible: the pedagogy of craft apprenticeship. Stud. Contin. Educ. 23(2), 185–200 (2001)

    Article  Google Scholar 

  10. Goel, A., et al.: Learning functional and causal abstractions of classroom aquaria. In: Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 32 (2010)

    Google Scholar 

  11. Hinchliffe, G.: Situating skills. J. Philos. Educ. 36(2), 187–205 (2002)

    Article  Google Scholar 

  12. Hiyama, A., Onimaru, H., Miyashita, M., Ebuchi, E., Seki, M., Hirose, M.: Augmented reality system for measuring and learning tacit artisan skills. In: Yamamoto, S. (ed.) HIMI 2013. LNCS, vol. 8017, pp. 85–91. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39215-3_10

    Chapter  Google Scholar 

  13. Hmelo-Silver, C.E., Marathe, S., Liu, L.: Fish swim, rocks sit, and lungs breathe: expert-novice understanding of complex systems. J. Learn. Sci. 16(3), 307–331 (2007)

    Article  Google Scholar 

  14. Iyobe, M., Ishida, T., Miyakawa, A., Uchida, N., Sugita, K., Shibata, Y.: Development of a mobile virtual traditional crafting presentation system using augmented reality technology. Int. J. Space-Based and Situated Comput. 6(4), 239–251 (2016)

    Article  Google Scholar 

  15. Jain, S., Sharma, S., Babbar, D.: Star-force: a playful implementation of the jedi-force. In: Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction, pp. 761–766 (2017)

    Google Scholar 

  16. Kang, S., Norooz, L., Bonsignore, E., Byrne, V., Clegg, T., Froehlich, J.E.: PrototypAR: prototyping and simulating complex systems with paper craft and augmented reality. In: Proceedings of the 18th ACM International Conference on Interaction Design and Children, pp. 253–266 (2019)

    Google Scholar 

  17. Khan, W.A., Raouf, A., Cheng, K.: Augmented reality for manufacturing. In: Virtual Manufacturing. Springer Series in Advanced Manufacturing. Springer, London (2011). https://doi.org/10.1007/978-0-85729-186-8_1

  18. Knibbe, J., Alsmith, A., Hornbæk, K.: Experiencing electrical muscle stimulation. Proc. ACM Interact. Mob. Wear. Ubiquitous Technol. 2(3), 1–14 (2018)

    Google Scholar 

  19. Kono, M., Ishiguro, Y., Miyaki, T., Rekimoto, J.: Design and study of a multi-channel electrical muscle stimulation toolkit for human augmentation. In: Proceedings of the 9th Augmented Human International Conference, pp. 1–8 (2018)

    Google Scholar 

  20. Lee, K.: Augmented reality in education and training. TechTrends 56(2), 13–21 (2012)

    Article  Google Scholar 

  21. Loch, F., Quint, F., Brishtel, I.: Comparing video and augmented reality assistance in manual assembly. In: 2016 12th International Conference on Intelligent Environments (IE), pp. 147–150 (2016). https://doi.org/10.1109/IE.2016.31

  22. Lopes, P., Ion, A., Baudisch, P.: Impacto: simulating physical impact by combining tactile stimulation with electrical muscle stimulation. In: Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology, pp. 11–19 (2015)

    Google Scholar 

  23. Lopes, P., You, S., Ion, A., Baudisch, P.: Adding force feedback to mixed reality experiences and games using electrical muscle stimulation. In: Proceedings of the 2018 chi conference on human factors in computing systems. pp. 1–13 (2018)

    Google Scholar 

  24. Lu, K., Brombacher, A.: Haptic feedback in running: is it possible for information transfer through electrical muscle signalling? In: Proceedings of the Fourteenth International Conference on Tangible, Embedded, and Embodied Interaction, pp. 479–485 (2020)

    Google Scholar 

  25. Nawahdah, M., Inoue, T.: Motion adaptive orientation adjustment of a virtual teacher to support physical task learning. Inf. Media Technol. 7(1), 506–515 (2012)

    Google Scholar 

  26. Osman, S., Zin, N.A.H.M.: Proposed model for courseware development of virtual teaching and learning traditional craft. In: 2010 International Symposium on Information Technology, vol. 1, pp. 1–6. IEEE (2010)

    Google Scholar 

  27. Pan, Z., Cheok, A.D., Yang, H., Zhu, J., Shi, J.: Virtual reality and mixed reality for virtual learning environments. Comput. graph. 30(1), 20–28 (2006)

    Article  Google Scholar 

  28. Papert, S., Harel, I.: Situating constructionism. Constructionism 36(2), 1–11 (1991)

    Google Scholar 

  29. Pfeiffer, M., Duente, T., Rohs, M.: Let your body move: a prototyping toolkit for wearable force feedback with electrical muscle stimulation. In: Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services, pp. 418–427 (2016)

    Google Scholar 

  30. Quintana, C., Reiser, B.J., Davis, E.A., Krajcik, J., Fretz, E., Duncan, R.G., Kyza, E., Edelson, D., Soloway, E.: A scaffolding design framework for software to support science inquiry. In: Scaffolding, pp. 337–386. Psychology Press (2018)

    Google Scholar 

  31. Repenning, A., Ioannidou, A., Phillips, J.: Collaborative use & design of interactive simulations (1999)

    Google Scholar 

  32. Shahu, A., Wintersberger, P., Michahelles, F.: Scenario-based investigation of acceptance of electric muscle stimulation. In: Proceedings of the Augmented Humans International Conference 2022, pp. 184–194 (2022)

    Google Scholar 

  33. Shahu, A., Wintersberger, P., Michahelles, F.: Would users accept electric muscle stimulation controlling their body? insights from a scenario-based investigation. In: CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1–7 (2022)

    Google Scholar 

  34. Stager, G.: Papertian constructionism and the design of productive contexts for learning. In: Proceedings of the EuroLogo, pp. 43–53. Citeseer (2005)

    Google Scholar 

  35. Steer, C., Robinson, S., Pearson, J., Sahoo, D., Mabbett, I., Jones, M.: A liquid tangible display for mobile colour mixing. In: Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services, pp. 1–7 (2018)

    Google Scholar 

  36. Syberfeldt, A., Danielsson, O., Gustavsson, P.: Augmented reality smart glasses in the smart factory: product evaluation guidelines and review of available products. IEEE Access 5, 9118–9130 (2017)

    Article  Google Scholar 

  37. Tajima, D., Nishida, J., Lopes, P., Kasahara, S.: Whose touch is this?: understanding the agency trade-off between user-driven touch vs. computer-driven touch. ACM Trans. Comput.-Hum. Interact. 29(3), 1–27 (2022). https://doi.org/10.1145/3489608

  38. Tang, Y., Au, K., Leung, Y.: Comprehending products with mixed reality: geometric relationships and creativity. Int. J. Eng. Bus. Manage. 10, 1847979018809599 (2018)

    Article  Google Scholar 

  39. Thomas, P., David, W.: Augmented reality: an application of heads-up display technology to manual manufacturing processes. In: Hawaii International Conference on System Sciences, vol. 2. ACM SIGCHI Bulletin (1992)

    Google Scholar 

  40. Wang, M., Callaghan, V., Bernhardt, J., White, K., Peña-Rios, A.: Augmented reality in education and training: pedagogical approaches and illustrative case studies. J. Ambient. Intell. Humaniz. Comput. 9(5), 1391–1402 (2018)

    Article  Google Scholar 

  41. Zacharia, Z., Anderson, O.R.: The effects of an interactive computer-based simulation prior to performing a laboratory inquiry-based experiment on students’ conceptual understanding of physics. Am. J. Phys. 71(6), 618–629 (2003)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. TU Wien, Vienna, Austria

    Ambika Shahu, Sonja Dorfbauer & Florian Michahelles

  2. FH Hagenberg, Hagenberg, Austria

    Philipp Wintersberger

Authors
  1. Ambika Shahu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sonja Dorfbauer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Philipp Wintersberger
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Florian Michahelles
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ambika Shahu .

Editor information

Editors and Affiliations

  1. University of West London, London, UK

    José Abdelnour Nocera

  2. Reykjavik University, Reykjavik, Iceland

    Marta Kristín Lárusdóttir

  3. University of York, York, UK

    Helen Petrie

  4. University of Bari Aldo Moro, Bari, Italy

    Antonio Piccinno

  5. Université Côte d’Azur, Sophia Antipolis Cedex, France

    Marco Winckler

1 Electronic supplementary material

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shahu, A., Dorfbauer, S., Wintersberger, P., Michahelles, F. (2023). Skillab - A Multimodal Augmented Reality Environment for Learning Manual Tasks. In: Abdelnour Nocera, J., Kristín Lárusdóttir, M., Petrie, H., Piccinno, A., Winckler, M. (eds) Human-Computer Interaction – INTERACT 2023. INTERACT 2023. Lecture Notes in Computer Science, vol 14144. Springer, Cham. https://doi.org/10.1007/978-3-031-42286-7_33

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-42286-7_33

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-42285-0

  • Online ISBN: 978-3-031-42286-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation