Abstract
Manual activities are often hidden deep down in discrete manufacturing processes. To analyze and optimize such processes, process discovery techniques allow the mining of the actual process behavior. Those techniques require the availability of complete event logs representing the execution of manual activities. Related works about collecting such information from sensor data unobtrusively for the worker are rare. Papers either address the sensor-based recognition of activities or focus on the process discovery part using process mining-compatible data sets. This paper builds on previous works to provide a solution on how execution-level information can be extracted from videos in manual assembly. The test bed consists of an assembly workstation equipped with a single RGB camera. A neural network-based real-time object detector delivers the input for an algorithm, which generates trajectories reflecting the movement paths of the worker’s hands. Those trajectories are automatically assigned to work steps using hierarchical clustering of similar behavior with dynamic time warping. The system has been evaluated in a task-based study with ten participants in a laboratory under realistic conditions. The generated logs have been loaded into the process mining toolkit ProM to discover the underlying process model and to measure the system’s performance using conformance checking.
S. Knoch and S. Ponpathirkoottam—These authors contributed equally.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Adriansyah, A., van Dongen, B.F., van der Aalst, W.M.P.: Conformance checking using cost-based fitness analysis. In: 2011 IEEE 15th International Enterprise Distributed Object Computing Conference, pp. 55–64, August 2011
van der Aalst, W., et al.: Process mining manifesto. In: Daniel, F., Barkaoui, K., Dustdar, S. (eds.) BPM 2011. LNBIP, vol. 99, pp. 169–194. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-28108-2_19
Bambach, S., Lee, S., Crandall, D.J., Yu, C.: Lending a hand: detecting hands and recognizing activities in complex egocentric interactions. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1949–1957 (2015)
Bauters, K., Cottyn, J., Claeys, D., Slembrouck, M., Veelaert, P., Van Landeghem, H.: Automated work cycle classification and performance measurement for manual work stations. Robot. Comput.-Integr. Manufact. 51, 139–157 (2018)
Berndt, D.J., Clifford, J.: Using dynamic time warping to find patterns in time series. In: KDD Workshop, Seattle, WA, vol. 10, pp. 359–370 (1994)
Carmona, J., van Dongen, B., Solti, A., Weidlich, M.: Conformance Checking: Relating Processes and Models. Springer, Heidelberg (2018). https://doi.org/10.1007/978-3-319-99414-7
Diba, K., Batoulis, K., Weidlich, M., Weske, M.: Extraction, correlation, and abstraction of event data for process mining. WIREs Data Min. Knowl. Discov. 10(3), e1346 (2020)
Eindhoven Technical University: Prom6.8 - revision 38904 (2010). http://www.promtools.org/doku.php?id=prom68
Kaczmarek, S., Hogreve, S., Tracht, K.: Progress monitoring and gesture control in manual assembly systems using 3D-image sensors. Procedia CIRP 37, 1–6 (2015)
Knoch, S., Ponpathirkoottam, S., Schwartz, T.: Video-to-model data set, March 2020. https://doi.org/10.6084/m9.figshare.12026850.v1. https://figshare.com/articles/Video-to-Model_Data_Set/12026850
Leemans, S.J.J., Fahland, D., van der Aalst, W.M.P.: Discovering block-structured process models from event logs - a constructive approach. In: Colom, J.-M., Desel, J. (eds.) PETRI NETS 2013. LNCS, vol. 7927, pp. 311–329. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-38697-8_17
Lerato, L., Niesler, T.: Investigating parameters for unsupervised clustering of speech segments using TIMIT. In: Twenty-Third Annual Symposium of the Pattern Recognition Association of South Africa, p. 83 (2012)
Mannhardt, F., Bovo, R., Oliveira, M.F., Julier, S.: A taxonomy for combining activity recognition and process discovery in industrial environments. In: Yin, H., Camacho, D., Novais, P., Tallón-Ballesteros, A.J. (eds.) IDEAL 2018. LNCS, vol. 11315, pp. 84–93. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03496-2_10
Müller, B.C., et al.: Motion tracking applied in assembly for worker training in different locations. Procedia CIRP 48, 460–465 (2016)
Nater, F., Grabner, H., Van Gool, L.: Unsupervised workflow discovery in industrial environments. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1912–1919. IEEE (2011)
Redmon, J., Farhadi, A.: YOLOv3: an incremental improvement (2018). https://arxiv.org/abs/1804.02767
Roitberg, A., Somani, N., Perzylo, A., Rickert, M., Knoll, A.: Multimodal human activity recognition for industrial manufacturing processes in robotic workcells. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 259–266. ACM (2015)
Shilkrot, R., Narasimhaswamy, S., Vazir, S., Hoai, M.: WorkingHands: a hand-tool assembly dataset for image segmentation and activity mining (2019)
Victor, D.: Real-time hand tracking using SSD on tensorflow (2017). https://github.com/victordibia/handtracking
Wahlster, W.: SmartF-IT - cyber-physical IT systems to master complexness of a new generation of multi-adaptive factories (2016). http://www.smartf-it-projekt.de
Acknowledgments
We thank Peter Fettke and Jana Koehler for discussions related to this work. This research was funded in part by the German Federal Ministry of Education and Research under grant number 01IS19022E (project BaSys4.2). The responsibility for this publication lies with the authors.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Knoch, S., Ponpathirkoottam, S., Schwartz, T. (2020). Video-to-Model: Unsupervised Trace Extraction from Videos for Process Discovery and Conformance Checking in Manual Assembly. In: Fahland, D., Ghidini, C., Becker, J., Dumas, M. (eds) Business Process Management. BPM 2020. Lecture Notes in Computer Science(), vol 12168. Springer, Cham. https://doi.org/10.1007/978-3-030-58666-9_17
Download citation
DOI: https://doi.org/10.1007/978-3-030-58666-9_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58665-2
Online ISBN: 978-3-030-58666-9
eBook Packages: Computer ScienceComputer Science (R0)