Skip to main content
SpringerLink
Log in

Teaching by Demonstrating – How Smart Assistive Systems Can Learn from Users

  • Conference paper
  • First Online:
Distributed, Ambient and Pervasive Interactions (HCII 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12203))

Included in the following conference series:

Abstract

Projection-based assitive systems that guide users through assembly work are on their way to industrial application. Previous research work investigated how people can be supported with such systems. However, there has been little work on the question on how to generate and author sequential instructions for assitive systems. In this paper, we present a new concept and a prototypical implementation of an assitive system that can be taught by demonstrating an assembly process. By using a combination of RGB and depth cameras, we can generate an assembly instruction of Lego Duplo bricks based on the demonstration of a user. This generated manual can later on be used for assisting other users in the assembly process. By our prototype system, we show the technological feasibility of assistive systems that can learn from users.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Agrawala, M., et al.: Designing effective step-by-step assembly instructions. ACM Trans. Graph. 22(3), 828–837 (2003). https://doi.org/10.1145/882262.882352

    Article  Google Scholar 

  2. Blattgerste, J., Renner, P., Pfeiffer, T.: Authorable augmented reality instructions for assistance and training in work environments. In: Proceedings of the 18th International Conference on Mobile and Ubiquitous Multimedia, MUM 2019, pp. 34:1–34:11. ACM, New York (2019). https://doi.org/10.1145/3365610.3365646

  3. Büttner, S., Besginow, A., Prilla, M., Röcker, C.: Mobile projection-based augmented reality in work environments-an exploratory approach. In: Mensch und Computer 2018-Workshopband (2018)

    Google Scholar 

  4. Büttner, S., Funk, M., Sand, O., Röcker, C.: Using head-mounted displays and in-situ projection for assistive systems: A comparison. In: Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2016, pp. 44:1–44:8. ACM, New York (2016). https://doi.org/10.1145/2910674.2910679

  5. Büttner, S., et al.: The design space of augmented and virtual reality applications for assistive environments in manufacturing: a visual approach. In: Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments, PETRA 2017, pp. 433–440. ACM, New York (2017). https://doi.org/10.1145/3056540.3076193

  6. Büttner, S., Mucha, H., Robert, S., Hellweg, F., Röcker, C.: HCI in der SmartFactoryOWL - Angewandte Forschung & Entwicklung. In: 4. Workshop zu Smart Factories: Mitarbeiter-zentrierte Informationssysteme für die Zusammenarbeit der Zukunft, Mensch und Computer 2017, Regensburg (2017). [German]

    Google Scholar 

  7. Büttner, S., Prilla, M., Röcker, C.: Augmented reality training for industrial assembly work - are projection-based ar assistive systems an appropriate tool for assembly training? In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (forthcoming), CHI 2020. ACM, New York (2020)

    Google Scholar 

  8. Büttner, S., Sand, O., Röcker, C.: Extending the design space in industrial manufacturing through mobile projection. In: Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, pp. 1130–1133. ACM (2015)

    Google Scholar 

  9. Büttner, S., Sand, O., Röcker, C.: Exploring design opportunities for intelligent worker assistance: a new approach using projetion-based AR and a novel hand-tracking algorithm. In: Braun, A., Wichert, R., Mana, A. (eds.) Ambient Intelligence, Am I 2017. Lecture Notes in Computer Science, vol. 10217, pp. 33–45. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-56997-0_3

    Chapter  Google Scholar 

  10. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 6, 679–698 (1986)

    Article  Google Scholar 

  11. Chi, P.Y.P., Vogel, D., Dontcheva, M., Li, W., Hartmann, B.: Authoring illustrations of human movements by iterative physical demonstration. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, UIST 2016, pp. 809–820. ACM, New York (2016). https://doi.org/10.1145/2984511.2984559

  12. Fellmann, M., Robert, S., Büttner, S., Mucha, H., Rocker, C.: Towards a framework for assistance systems to support work processes in smart factories. In: Holzinger, A., Kieseberg, P., Tjoa, A., Weippl, E. (eds.) Machine Learning and Knowledge Extraction, CD-MAKE 2017. Lecture Notes in Computer Science, vol. 10410, pp. 59–68. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66808-6_5

    Chapter  Google Scholar 

  13. Fisher, R., Perkins, S., Walker, A., Wolfart, E.: Hough Transform (2003). http://homepages.inf.ed.ac.uk/rbf/HIPR2/hough.htm. Accessed 14 Jan 2020

  14. Funk, M., Mayer, S., Schmidt, A.: Using in-situ projection to support cognitively impaired workers at the workplace. In: Proceedings of the 17th International ACM SIGACCESS Conference on Computers, ASSETS 2015, pp. 185–192. ACM, New York (2015). https://doi.org/10.1145/2700648.2809853

  15. Hakkarainen, M., Woodward, C., Billinghurst, M.: Augmented assembly using a mobile phone. In: Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 167–168. IEEE Computer Society (2008)

    Google Scholar 

  16. Hinrichsen, S., Jasperneite, J., Schrader, F., Lücke, B., Fraunhofer, I.I.: Versatile assembly systems-requirements, design principles and examples. Proc. PEM 14, 37–46 (2014)

    Google Scholar 

  17. Korn, O., Schmidt, A., Hörz, T.: The potentials of in-situ-projection for augmented workplaces in production: A study with impaired persons. In: CHI 2013 Extended Abstracts on Human Factors in Computing Systems, CHI EA 2013, pp. 979–984. ACM, New York (2013). https://doi.org/10.1145/2468356.2468531

  18. Mohr, P., Kerbl, B., Donoser, M., Schmalstieg, D., Kalkofen, D.: Retargeting technical documentation to augmented reality. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI 2015, pp. 3337–3346. ACM, New York (2015). https://doi.org/10.1145/2702123.2702490

  19. Mohr, P., Mandl, D., Tatzgern, M., Veas, E., Schmalstieg, D., Kalkofen, D.: Retargeting video tutorials showing tools with surface contact to augmented reality. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI 2017, pp. 6547–6558. ACM, New York (2017)

    Google Scholar 

  20. OpenCV: https://opencv.org/ (2020), [Online; accessed 14-January-2020]

  21. Paelke, V., Röcker, C., Koch, N., Flatt, H., Büttner, S.: User interfaces for cyber-physical systems. at-Automatisierungstechnik 63(10), 833–843 (2015)

    Google Scholar 

  22. pyrealsense (2020). https://pypi.org/project/pyrealsense/, Accessed 14 Jan 2020

  23. rexroth - a Bosch Company: ActiveAssist assistance system - modular, connected, interactive (2019). https://www.boschrexroth.com/en/xc/products/product-groups/assembly-technology/news/activeassist-assistance-system/index, Accessed 11 Dec 2019

  24. Röcker, C., Robert, S.: Projektionsbasierte Montageunterstützung mit visueller Fortschrittserkennung [German]. visIT Industrie 4 (2016)

    Google Scholar 

  25. Sand, O., Büttner, S., Paelke, V., Röcker, C.: smARt.Assembly - projection-based augmented reality for supporting assembly workers. In: Lackey, S., Shumaker, R. (eds.) Virtual, Augmented and Mixed Reality, VAMR 2016. Lecture Notes in Computer Science, vol. 9740, pp. 643–652. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39907-2_61

    Chapter  Google Scholar 

  26. Streitz, N.: Beyond ‘smart-only’ cities: redefining the ‘smart-everything’ paradigm. J. Ambi. Intell. Hum. Comput. 10(2), 791–812 (2019). https://doi.org/10.1007/s12652-018-0824-1

    Article  Google Scholar 

  27. Suzuki, S., et al.: Topological structural analysis of digitized binary images by border following. Comput. Vis. Graph. Image Process. 30(1), 32–46 (1985)

    Article  Google Scholar 

  28. Tang, A., Owen, C., Biocca, F., Mou, W.: Comparative effectiveness of augmented reality in object assembly. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 73–80. ACM (2003)

    Google Scholar 

  29. Wang, B., et al.: Active assembly guidance with online video parsing. In: 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 459–466. IEEE (2018)

    Google Scholar 

Download references

Acknowledgments

The authors acknowledge the financial support by the Federal Ministry of Education and Research of Germany for the project “Augmented-Reality-Assistenzsysteme für mobile Anwendungsszenarien in der Industrie (FKZ: 03FH005IX6)”.

Author information

Authors and Affiliations

  1. Ostwestfalen-Lippe University of Applied Sciences and Arts, Lemgo, Germany

    Sebastian Büttner, Andreas Peda, Mario Heinz & Carsten Röcker

  2. Human-Centered Information Systems, Clausthal University of Technology, Clausthal-Zellerfeld, Germany

    Sebastian Büttner

  3. Fraunhofer IOSB-INA, Lemgo, Germany

    Carsten Röcker

Authors
  1. Sebastian Büttner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andreas Peda
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mario Heinz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Carsten Röcker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sebastian Büttner .

Editor information

Editors and Affiliations

  1. Smart Future Initiative, Frankfurt/Main, Germany

    Norbert Streitz

  2. Kyushu University, Fukuoka, Japan

    Shin'ichi Konomi

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Büttner, S., Peda, A., Heinz, M., Röcker, C. (2020). Teaching by Demonstrating – How Smart Assistive Systems Can Learn from Users. In: Streitz, N., Konomi, S. (eds) Distributed, Ambient and Pervasive Interactions. HCII 2020. Lecture Notes in Computer Science(), vol 12203. Springer, Cham. https://doi.org/10.1007/978-3-030-50344-4_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-50344-4_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-50343-7

  • Online ISBN: 978-3-030-50344-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation