skip to main content
10.1145/3316782.3321526acmotherconferencesArticle/Chapter ViewAbstractPublication PagespetraConference Proceedingsconference-collections
research-article

Low-cost tracking of assembly tasks in industrial environments

Published: 05 June 2019 Publication History

Abstract

The fourth industrial revolution brings a lot of new challenges to the production process in a so called smart factory. The flexible production process and many different product variants call for assisting systems to support workers during their assembly tasks.
We conducted a Contextual Inquiry in a real-life production environment to find typical problems during assembling products. In our work we use the General Assembly Task Model (GATM) proposed by Funk et al. [13] to identify and assess potential assistance systems in how they can supports each phase of an assembly step. Our analysis revealed that tracking of assembly tasks is very helpful to automatically forward work instructions and to check if intended parts where taken from the Kanban bin and were used in the proposed order.
We built in a further step two vision-based low-cost systems, one with Halcon machine vision and one with TensorFlow deep leaning, and one low-cost system based on ultrasound (i.e. Marvelmind) to track assembly tasks. This paper compares the three approaches with the aid of three prototypes, one for visual recognition of assembly parts, one for visual recognition of assembly parts and tools, and one for ultrasound-based tracking of picking assembly parts from a bin. Finally, we discuss selected findings which are relevant for an industrial application setting.

References

[1]
Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16). USENIX Association, Berkeley, CA, 265--283.
[2]
Mario Aehnelt and Sebastian Bader. 2016. Providing and Adapting Information Assistance for Smart Assembly Stations. In Proceedings of SAI Intelligent Systems Conference (IntelliSys) 2016. Springer International Publishing, Cham, 540--562.
[3]
J.K. Aggarwal and M.S. Ryoo. 2011. Human Activity Analysis: A Review. ACM Comput. Surv. 43, 3, Article 16 (April 2011), 43 pages.
[4]
Stavros Antifakos, Florian Michahelles, and Bernt Schiele. 2002. Proactive Instructions for Furniture Assembly. In UbiComp 2002: Ubiquitous Computing, Gaetano Borriello and Lars Erik Holmquist (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 351--360.
[5]
Mirjam Augstein, Thomas Neumayr, Sebastian Pimminger, Christine Ebner, Josef Altmann, and Werner Kurschl. 2018. Contextual Design in Industrial Settings: Experiences and Recommendations. In Proceedings of the 20th International Conference on Enterprise Information Systems - Volume 2: ICEIS,. INSTICC, SciTePress, Portugal, 429--440.
[6]
Sebastian Bader and Mario Aehnelt. 2014. Tracking Assembly Processes and Providing Assistance in Smart Factories. In Proceedings of the 6th International Conference on Agents and Artificial Intelligence - Volume 1 (ICAART 2014). SCITEPRESS - Science and Technology Publications, Lda, Portugal, 161--168.
[7]
Thomas Bauernhansl, Michael Ten Hompel, and Birgit Vogel-Heuser. 2014. Industrie 4.0 in Produktion, Automatisierung und Logistik: Anwendung, Technologien und Migration. Springer Vieweg, Wiesbaden.
[8]
Patrick Bertram, Max Birtel, Fabian Quint, and Martin Ruskowski. 2018. Intelligent Manual Working Station through Assistive Systems. IFAC-PapersOnLine 51, 11 (2018), 170--175.
[9]
Hugh Beyer and Karen Holtzblatt. 1997. Contextual Design: Defining Customer-Centered Systems (1 ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
[10]
Jonas Blattgerste, Benjamin Strenge, Patrick Renner, Thies Pfeiffer, and Kai Essig. 2017. Comparing Conventional and Augmented Reality Instructions for Manual Assembly Tasks. In Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA '17). ACM, New York, NY, USA, 75--82.
[11]
Rainer Bokranz and Kurt Landau. 2012. Handbuch Industrial Engineering: Produktivitätsmanagement mit MTM. Schäffer-Poeschel, Stuttgart.
[12]
Joep W. Frens. 2016. Cardboard Modeling: Exploring, Experiencing and Communicating. Springer International Publishing, Cham, 149--177.
[13]
Markus Funk, Thomas Kosch, Scott W. Greenwald, and Albrecht Schmidt. 2015. A Benchmark for Interactive Augmented Reality Instructions for Assembly Tasks. In Proceedings of the 14th International Conference on Mobile and Ubiquitous Multimedia (MUM '15). ACM, New York, NY, USA, 253--257.
[14]
Markus Funk, Thomas Kosch, Romina Kettner, Oliver Korn, and Albrecht Schmidt. 2016. motionEAP: An Overview of 4 Years of Combining Industrial Assembly with Augmented Reality for Industry 4.0. In Proceedings of the 16th International Conference on Knowledge Technologies and Data-driven Business (i-KNOW '16). ACM, New York, NY, USA, 4.
[15]
Pawel Gorecki and Peter Pautsch. 2012. Cardboard-Engineering: Kostensenkungspotenziale in der industriellen Automation. Productivity Management 17, 2 (2012), 29--31.
[16]
Dominic Gorecky, Matthias Schmitt, and Matthias Loskyll. 2014. Mensch-Maschine Interaktion im Industrie 4.0-Zeitalter. Springer Fachmedien Wiesbaden, Wiesbaden, 525--542.
[17]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, Washington, DC, USA, 770--778.
[18]
Karen Holtzblatt and Hugh Beyer. 2017. Contextual Design. Design for Life (2 ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
[19]
Karen Holtzblatt, Jessamyn Burns Wendell, and Shelley Wood. 2004. Rapid Contextual Design. A How-To Guide to Key Techniques for User-Centered Design. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
[20]
Qiannan Jiang, Mingzhou Liu, Xiaoqiao Wang, Maogen Ge, and Ling Lin. 2016. Human motion segmentation and recognition using machine vision for mechanical assembly operation. SpringerPlus 5, 1 (2016), 1629.
[21]
Yoram Koren. 2010. The Global Manufacturing Revolution: Product-Process-Business Integration and Reconfigurable Systems. John Wiley and Sons, New York, NY, USA.
[22]
Oliver Korn, Markus Funk, and Albrecht Schmidt. 2015. Towards a Gamification of Industrial Production: A Comparative Study in Sheltered Work Environments. In Proceedings of the 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '15). ACM, New York, NY, USA, 84--93.
[23]
Oliver Korn, Albrecht Schmidt, and Thomas Hörz. 2012. Assistive Systems in Production Environments: Exploring Motion Recognition and Gamification. In Proceedings of the 5th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA '12). ACM, New York, NY, USA, Article 9, 5 pages.
[24]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1 (NIPS'12). Curran Associates Inc., USA, 1097--1105.
[25]
Arndt Lueder. 2014. Integration des Menschen in Szenarien der Industrie 4.0. Springer Fachmedien Wiesbaden, Wiesbaden, 493--508.
[26]
Takuya Maekawa, Daisuke Nakai, Kazuya Ohara, and Yasuo Namioka. 2016. Toward Practical Factory Activity Recognition: Unsupervised Understanding of Repetitive Assembly Work in a Factory. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). ACM, New York, NY, USA, 1088--1099.
[27]
Felix Mannhardt, Riccardo Bovo, Manuel Fradinho Oliveira, and Simon Julier. 2018. A Taxonomy for Combining Activity Recognition and Process Discovery in Industrial Environments. In Intelligent Data Engineering and Automated Learning - IDEAL 2018, Hujun Yin, David Camacho, Paulo Novais, and Antonio J. Tallón-Ballesteros (Eds.). Springer International Publishing, Cham, 84--93.
[28]
Bastian C. Müller, The Duy Nguyen, Quang-Vinh Dang, Bui Minh Duc, Günther Seliger, Jörg Krüger, and Holger Kohl. 2016. Motion Tracking Applied in Assembly for Worker Training in different Locations. Procedia CIRP 48 (2016), 460 -- 465. The 23rd CIRP Conference on Life Cycle Engineering.
[29]
Martin Neumann and Thomas Dietz. 2014. Mensch-Maschine-Interaktion. Springer Fachmedien Wiesbaden, Wiesbaden, 509--523.
[30]
Matthias Peissner and Cornelia Hipp. 2013. Potenziale der Mensch-Technik Interaktion für die effiziente und vernetzte Produktion von morgen. Fraunhofer Verlag, Stuttgart.
[31]
Sabine Pfeiffer. 2016. Robots, Industry 4.0 and Humans, or Why Assembly Work Is More than Routine Work. Societies 6, 2 (2016), 16.
[32]
Fabian Quint, Frieder Loch, Marius Orfgen, and Detlef Zuehlke. 2016. A System Architecture for Assistance in Manual Tasks. In Intelligent Environments (Workshops) (Ambient Intelligence and Smart Environments), Vol. 21. IOS Press, Amsterdam, NL, 43--52.
[33]
Rafael Radkowski. 2016. Object tracking with a range camera for augmented reality assembly assistance. Journal of Computing and Information Science in Engineering 16, 1 (2016), 011004.
[34]
Alina Roitberg, Nikhil Somani, Alexander Perzylo, Markus Rickert, and Alois Knoll. 2015. Multimodal Human Activity Recognition for Industrial Manufacturing Processes in Robotic Workcells. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (ICMI '15). ACM, New York, NY, USA, 259--266.
[35]
Günther Schuh, Achim Kampker, Bastian Franzkoch, Cathrin Wesch-Potente, and Mateusz Swist. 2010. Praxisnahe Montagegestaltung mit Cardboard-Engineering. wt Werkstattstechnik online 100, 9 (2010), 659--664.
[36]
J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake. 2011. Real-time Human Pose Recognition in Parts from Single Depth Images. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR '11). IEEE Computer Society, Washington, DC, USA, 1297--1304.
[37]
Thomas Stiefmeier, Clemens Lombriser, Daniel Roggen, Holger Junker, Georg Ogris, and Gerhard Tröster. 2006. Event-Based Activity Tracking in Work Environments. In 3rd International Forum on Applied Wearable Computing 2006. VDE Verlag, Berlin, Germany, 1--10.
[38]
Alvin R. Tilley. 2002. The Measure of Man and Woman: Human Factors in Design. Wiley, New York, NY, USA.
[39]
J. A. Ward, P. Lukowicz, G. Troster, and T. E. Starner. 2006. Activity Recognition of Assembly Tasks Using Body-Worn Microphones and Accelerometers. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 10 (2006), 1553--1567.
[40]
Xiaozhou Yang and Daniela Alina Plewe. 2016. Assistance Systems in Manufacturing: A Systematic Review. In Advances in Ergonomics of Manufacturing: Managing the Enterprise of the Future. Springer International Publishing, Cham, 279--289.

Cited By

View all
  • (2025)Towards cognition-augmented human-centric assemblyRobotics and Computer-Integrated Manufacturing10.1016/j.rcim.2024.10285291:COnline publication date: 1-Feb-2025
  • (2023)A literature review on the prediction and monitoring of assembly and disassembly processes in discrete make-to-order production in SMEs with machine vision technologiesProceedings of the 2023 10th International Conference on Industrial Engineering and Applications10.1145/3587889.3588217(318-327)Online publication date: 9-Jan-2023
  • (2022)Learn from the Best: Harnessing Expert Skill and Knowledge to Teach Unskilled WorkersProceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3529190.3529203(93-102)Online publication date: 29-Jun-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
PETRA '19: Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments
June 2019
655 pages
ISBN:9781450362320
DOI:10.1145/3316782
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 June 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. human-centered design
  2. intelligent assistive systems
  3. manual assembly
  4. smart production
  5. tracking

Qualifiers

  • Research-article

Funding Sources

  • Österreichische Forschungsförderungsgesellschaft

Conference

PETRA '19

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)22
  • Downloads (Last 6 weeks)2
Reflects downloads up to 25 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Towards cognition-augmented human-centric assemblyRobotics and Computer-Integrated Manufacturing10.1016/j.rcim.2024.10285291:COnline publication date: 1-Feb-2025
  • (2023)A literature review on the prediction and monitoring of assembly and disassembly processes in discrete make-to-order production in SMEs with machine vision technologiesProceedings of the 2023 10th International Conference on Industrial Engineering and Applications10.1145/3587889.3588217(318-327)Online publication date: 9-Jan-2023
  • (2022)Learn from the Best: Harnessing Expert Skill and Knowledge to Teach Unskilled WorkersProceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3529190.3529203(93-102)Online publication date: 29-Jun-2022
  • (2022)Determining Best Hardware, Software and Data Structures for Worker Guidance during a Complex Assembly TaskProceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments10.1145/3529190.3529200(63-72)Online publication date: 29-Jun-2022
  • (2022)Etude exploratoire d’un assistant digital d’aide au montage basé sur la projection en réalité augmentéeProceedings of the 33rd Conference on l'Interaction Humain-Machine10.1145/3500866.3516375(1-12)Online publication date: 5-Apr-2022
  • (2021)A Faster R-CNN Implementation of Presence Inspection for Parts on Industrial Produce2021 Emerging Trends in Industry 4.0 (ETI 4.0)10.1109/ETI4.051663.2021.9619228(1-4)Online publication date: 19-May-2021
  • (2021)A Human-Centered Assembly Workplace For Industry: Challenges and Lessons LearnedProcedia Computer Science10.1016/j.procs.2021.01.166180(290-300)Online publication date: 2021

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media