Skip to main content
SpringerLink
Log in

Analytics Pipeline for Process Mining on Video Data

  • Conference paper
  • First Online:
Business Process Management Forum (BPM 2023)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 490))

Included in the following conference series:

Abstract

Process mining has shown that it provides valuable insights in terms of uncovering bottlenecks and inefficiencies in processes or identifying tasks for automation. However, process mining techniques expect structured input data that is at a high (business) level of abstraction. Recently, the benefits of process mining for unstructured data which is at a much lower level of abstraction have been demonstrated, e.g., for IoT data or time series data. It can be expected that the demand for methods efficiently processing these kinds of data for process mining will continuously increase. Hence, in this paper, we present an approach that allows the translation of video data into higher-level, discrete event data, thus enabling existing process mining techniques to work on data tracked in videos. Particularly, we used a combination of object tracking, spatio-temporal action detection, and techniques for raising the abstraction level of events. The evaluation results show that meaningful event logs can be extracted from an unlabeled video dataset, validating both the implementation and the feasibility of our approach.

We thank Jan Bosselmann for his support with the implementation, and the Institute of Agricultural Engineering at Kiel University for providing a dataset and use case. This project has received funding from the State of Schleswig-Holstein under the Datencampus project grant no. 220 21 016.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 74.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/arvidle/video-process-mining-public.

  2. 2.

    https://ffmpeg.org/.

  3. 3.

    https://fluxicon.com/disco/.

References

  1. van der Aalst, W., et al.: Process mining manifesto. In: Daniel, F., Barkaoui, K., Dustdar, S. (eds.) BPM 2011. LNBIP, vol. 99, pp. 169–194. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-28108-2_19

    Chapter  Google Scholar 

  2. Bergamini, L., et al.: Extracting accurate long-term behavior changes from a large pig dataset. In: VISAPP 2021, vol. 4, pp. 524–533. SCITEPRESS, Online Streaming (2021). https://doi.org/10.5220/0010288405240533

  3. Berti, A., van Zelst, S.J., van der Aalst, W.: Process Mining for Python (PM4Py): bridging the gap between process- and data science (2019). https://doi.org/10.48550/arXiv.1905.06169

  4. Cao, J., Weng, X., Khirodkar, R., Pang, J., Kitani, K.: Observation-centric sort: rethinking sort for robust multi-object tracking. arXiv preprint (2022). https://doi.org/10.48550/arXiv.2203.14360

  5. Chen, C., Zhu, W., Norton, T.: Behaviour recognition of pigs and cattle: journey from computer vision to deep learning. Comput. Electron. Agric. 187, 106255 (2021). https://doi.org/10.1016/j.compag.2021.106255

    Article  Google Scholar 

  6. Diamantini, C., Genga, L., Potena, D.: Behavioral process mining for unstructured processes. J. Intell. Inf. Syst. 47(1), 5–32 (2016). https://doi.org/10.1007/s10844-016-0394-7

    Article  Google Scholar 

  7. Diba, K., Batoulis, K., Weidlich, M., Weske, M.: Extraction, correlation, and abstraction of event data for process mining. WIREs Data Min. Knowl. Discov. 10(3), e1346 (2020). https://doi.org/10.1002/widm.1346

    Article  Google Scholar 

  8. Feichtenhofer, C., Fan, H., Malik, J., He, K.: SlowFast networks for video recognition. In: ICCV 2019, pp. 6201–6210. Seoul, Korea (South) (2019). https://doi.org/10.1109/ICCV.2019.00630

  9. Gu, C., et al.: AVA: a video dataset of spatio-temporally localized atomic visual actions. In: CVPR 2018, pp. 6047–6056 (2018). https://doi.org/10.1109/CVPR.2018.00633

  10. Janssen, D., Mannhardt, F., Koschmider, A., van Zelst, S.J.: Process model discovery from sensor event data. In: Leemans, S., Leopold, H. (eds.) ICPM 2020. LNBIP, vol. 406, pp. 69–81. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72693-5_6

    Chapter  Google Scholar 

  11. Knoch, S., Ponpathirkoottam, S., Schwartz, T.: Video-to-Model: unsupervised trace extraction from videos for process discovery and conformance checking in manual assembly. In: Fahland, D., Ghidini, C., Becker, J., Dumas, M. (eds.) BPM 2020. LNCS, vol. 12168, pp. 291–308. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58666-9_17

    Chapter  Google Scholar 

  12. Koschmider, A., Mannhardt, F., Heuser, T.: On the contextualization of event-activity mappings. In: Daniel, F., Sheng, Q.Z., Motahari, H. (eds.) BPM 2018. LNBIP, vol. 342, pp. 445–457. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11641-5_35

    Chapter  Google Scholar 

  13. Koschmider, A., Oppelt, N., Hundsdörfer, M.: Confidence-driven communication of process mining on time series. Informatik Spektrum 45(4), 223–228 (2022). https://doi.org/10.1007/s00287-022-01470-3

    Article  Google Scholar 

  14. Kratsch, W., König, F., Röglinger, M.: Shedding light on blind spots - developing a reference architecture to leverage video data for process mining. Decis. Support Syst. 158, 113794 (2022). https://doi.org/10.1016/j.dss.2022.113794

    Article  Google Scholar 

  15. Leemans, S.J.J., Fahland, D., van der Aalst, W.M.P.: Discovering block-structured process models from event logs containing infrequent behaviour. In: Lohmann, N., Song, M., Wohed, P. (eds.) BPM 2013. LNBIP, vol. 171, pp. 66–78. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-06257-0_6

    Chapter  Google Scholar 

  16. Lepsien, A.: Quantifying uncertainty for explainable process mining. In: Proceedings of the 13th International Workshop on Enterprise Modeling and Information Systems Architectures (EMISA 2023). CEUR Workshop Proceedings, vol. 3397. CEUR-WS.org, Stockholm, Sweden (May 2023). https://ceur-ws.org/Vol-3397/

  17. Lepsien, A., Bosselmann, J., Melfsen, A., Koschmider, A.: Process mining on video data. In: Manner, J., Lübke, D., Haarmann, S., Kolb, S., Herzberg, N., Kopp, O. (eds.) ZEUS 2022. CEUR Workshop Proceedings, vol. 3113, pp. 56–62. CEUR-WS.org, Bamberg, Germany (2022). https://ceur-ws.org/Vol-3113/paper9.pdf

  18. Lepsien, A., Koschmider, A., Kratsch, W.: Video process mining evaluation data (2023). https://doi.org/10.5281/zenodo.7763839

  19. Luo, W., Xing, J., Milan, A., Zhang, X., Liu, W., Kim, T.K.: Multiple object tracking: a literature review. Artif. Intell. 293, 103448 (2021). https://doi.org/10.1016/j.artint.2020.103448

    Article  MathSciNet  MATH  Google Scholar 

  20. Mannhardt, F., Koschmider, A., Baracaldo, N., Weidlich, M., Michael, J.: Privacy-preserving process mining. Bus. Inf. Syst. Eng. 61(5), 595–614 (2019). https://doi.org/10.1007/s12599-019-00613-3

    Article  Google Scholar 

  21. MMAction2 Contributors: openmmlab’s next generation video understanding toolbox oand benchmark. https://github.com/open-mmlab/mmaction2 (2020)

  22. Nannoni, E., Aarnink, A.J.A., Vermeer, H.M., Reimert, I., Fels, M., Bracke, M.B.M.: Soiling of Pig Pens: a review of eliminative behaviour. Animals 10(11), 2025 (2020). https://doi.org/10.3390/ani10112025

    Article  Google Scholar 

  23. Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

  24. Pegoraro, M., Uysal, M.S., van der Aalst, W.M.P.: Efficient construction of behavior graphs for uncertain event data. In: Abramowicz, W., Klein, G. (eds.) BIS 2020. LNBIP, vol. 389, pp. 76–88. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-53337-3_6

    Chapter  Google Scholar 

  25. Rebmann, A., Emrich, A., Fettke, P.: Enabling the discovery of manual processes using a multi-modal activity recognition approach. In: Di Francescomarino, C., Dijkman, R., Zdun, U. (eds.) BPM 2019. LNBIP, vol. 362, pp. 130–141. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-37453-2_12

    Chapter  Google Scholar 

  26. Shermin, T., Teng, S.W., Murshed, M., Lu, G., Sohel, F., Paul, M.: Enhanced transfer learning with imagenet trained classification layer. In: Lee, C., Su, Z., Sugimoto, A. (eds.) PSIVT 2019. LNCS, vol. 11854, pp. 142–155. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-34879-3_12

    Chapter  Google Scholar 

  27. Wada, K.: LabelMe: image polygonal annotation with Python (2023). https://doi.org/10.5281/zenodo.5711226, https://github.com/wkentaro/labelme

  28. Wang, C.Y., Bochkovskiy, A., Liao, H.Y.M.: YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors (2022). https://doi.org/10.48550/arXiv.2207.02696

  29. Weijters, A., van Der Aalst, W.M., De Medeiros, A.A.: Process mining with the heuristics miner-algorithm. TU Eindhoven, Tech. Rep. WP 166(July 2017), 1–34 (2006)

    Google Scholar 

  30. Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2. https://github.com/facebookresearch/detectron2 (2019)

  31. Zandkarimi, F., Rehse, J.R., Soudmand, P., Hoehle, H.: A generic framework for trace clustering in process mining. In: ICPM 2020, pp. 177–184. IEEE, Padua, Italy (2020). https://doi.org/10.1109/ICPM49681.2020.00034

  32. van Zelst, S.J., Mannhardt, F., de Leoni, M., Koschmider, A.: Event abstraction in process mining: literature review and taxonomy. Granular Comput. 6(3), 719–736 (2020). https://doi.org/10.1007/s41066-020-00226-2

    Article  Google Scholar 

  33. Zhang, Y., et al.: ByteTrack: multi-object tracking by associating every detection box. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022, pp. 1–21. LNCS, Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20047-2_1

  34. Zisgen, Y., Janssen, D., Koschmider, A.: Generating synthetic sensor event logs for process mining. In: De Weerdt, J., Polyvyanyy, A. (eds.) CAiSE Forum 2022. LNBIP, vol. 452, pp. 130–137. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-07481-3_15

  35. Zoric, M., Johansson, S.E., Wallgren, P.: Behaviour of fattening pigs fed with liquid feed and dry feed. Porc. Health Manag. 1(1), 14 (2015). https://doi.org/10.1186/s40813-015-0009-7

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Kiel University, Kiel, Germany

    Arvid Lepsien

  2. Chair of Business Informatics and Process Analytics, University of Bayreuth, Bayreuth, Fraunhofer FIT, Bayreuth, Germany

    Agnes Koschmider

  3. Technical University of Applied Sciences Augsburg, Fraunhofer FIT, Augsburg, Germany

    Wolfgang Kratsch

Authors
  1. Arvid Lepsien
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Agnes Koschmider
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wolfgang Kratsch
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arvid Lepsien .

Editor information

Editors and Affiliations

  1. University of Trento, Trento, Italy

    Chiara Di Francescomarino

  2. Technical University of Denmark, Kgs. Lyngby, Denmark

    Andrea Burattin

  3. TU Dortmund University, Dortmund, Germany

    Christian Janiesch

  4. The University of Queensland, Brisbane, QLD, Australia

    Shazia Sadiq

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lepsien, A., Koschmider, A., Kratsch, W. (2023). Analytics Pipeline for Process Mining on Video Data. In: Di Francescomarino, C., Burattin, A., Janiesch, C., Sadiq, S. (eds) Business Process Management Forum. BPM 2023. Lecture Notes in Business Information Processing, vol 490. Springer, Cham. https://doi.org/10.1007/978-3-031-41623-1_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-41623-1_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-41622-4

  • Online ISBN: 978-3-031-41623-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation