Abstract
Predicting future activity occurrences for a process instance is a key challenge in predictive process monitoring. Sequential deep learning models have been improving the prediction accuracy for this suffix prediction task. Training such models with many parameters on large event logs requires expensive hardware and is often time consuming. Transfer learning addresses this issue by starting from a pre-trained model to be used as starting point for the training on other data sets thereby reducing training time or improving accuracy in a given time budget. Transfer learning has shown to be very effective for natural language processing and image classification. However, research on transfer learning for predictive process monitoring is scarce and missing for suffix prediction. This paper contributes an experimental study on the effectiveness of transfer learning for suffix prediction using two sequential deep learning architectures (GPT and LSTM). Base models are trained on two public event logs and used as starting point for transfer learning on eight event logs from different domains. The experiments show that even with half of the available training budget and without using very large event logs for the base model, the results obtained in the transfer learning setting are often better and in some cases competitive to when trained using random initialization. A notable exception is an event log with a very large vocabulary of activity labels. This seems to indicate dependence of transfer learning on specific data properties such as vocabulary size and warranting further research.
This work is part of the Smart Journey Mining project, funded by the Research Council of Norway (grant no. 312198).
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Francescomarino, C.D., Ghidini, C.: Predictive process monitoring. In: van der Aalst, W.M.P., Carmona, J. (eds.) Process Mining Handbook. LNBIP, vol. 448, pp. 320–346. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-08848-3_10
Teinemaa, I., Dumas, M., Rosa, M.L., Maggi, F.M.: Outcome-oriented predictive process monitoring: review and benchmark. ACM Trans. Knowl. Discov. Data 13(2), 17:1–17:57 (2019)
Pasquadibisceglie, V., Appice, A., Castellano, G., Malerba, D.: A multi-view deep learning approach for predictive business process monitoring. IEEE Trans. Serv. Comput. 15(4), 2382–2395 (2022)
Ketykó, I., Mannhardt, F., Hassani, M., van Dongen, B.F.: What averages do not tell: predicting real life processes with sequential deep learning. In: SAC, pp. 1128–1131. ACM (2022)
Neu, D.A., Lahann, J., Fettke, P.: A systematic literature review on state-of-the-art deep learning methods for process prediction. Artif. Intell. Rev. 55(2), 801–827 (2022)
Rama-Maneiro, E., Vidal, J., Lama, M.: Deep learning for predictive business process monitoring: review and benchmark. IEEE Trans. Serv. Comput. (2021)
Weiss, K., Khoshgoftaar, T.M., Wang, D.: A survey of transfer learning. J. Big data 3(1), 1–40 (2016)
Zhuang, F., et al.: A comprehensive survey on transfer learning. Proc. IEEE 109(1), 43–76 (2020)
Duan, L., Xu, D., Tsang, I.: Learning with augmented features for heterogeneous domain adaptation. arXiv preprint arXiv:1206.4660 (2012)
Kulis, B., Saenko, K., Darrell, T.: What you saw is not what you get: domain adaptation using asymmetric kernel transforms. In: CVPR 2011, pp. 1785–1792. IEEE (2011)
Wang, C., Mahadevan, S.: Heterogeneous domain adaptation using manifold alignment. In: Twenty-Second International Joint Conference on Artificial Intelligence (2011)
Zhou, J., Pan, S., Tsang, I., Yan, Y.: Hybrid heterogeneous transfer learning through deep learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 28 (2014)
Tong, L., Weijian, N., Yujian, S., Qingtian, Z.: Predicting remaining business time with deep transfer learning. Data Anal. Knowl. Discov. 4(2/3), 134 (2020)
Chen, H., Fang, X., Fang, H.: Multi-task prediction method of business process based on BERT and transfer learning. Knowl.-Based Syst. 254, 109603 (2022)
Ni, W., Yan, M., Liu, T., Zeng, Q.: Predicting remaining execution time of business process instances via auto-encoded transition system. Intell. Data Anal. 26(2), 543–562 (2022)
Peeperkorn, J., vanden Broucke, S., De Weerdt, J.: Can deep neural networks learn process model structure? An assessment framework and analysis. In: Munoz-Gama, J., Lu, X. (eds.) ICPM 2021. LNBIP, vol. 433, pp. 127–139. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-98581-3_10
Augusto, A., Mendling, J., Vidgof, M., Wurm, B.: The connection between process complexity of event sequences and models discovered by process mining. Inf. Sci. 598, 196–215 (2022)
Evermann, J., Rehse, J., Fettke, P.: Predicting process behaviour using deep learning. Decis. Support Syst. 100, 129–140 (2017)
Tax, N., Verenich, I., La Rosa, M., Dumas, M.: Predictive business process monitoring with LSTM neural networks. In: Dubois, E., Pohl, K. (eds.) CAiSE 2017. LNCS, vol. 10253, pp. 477–492. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59536-8_30
Camargo, M., Dumas, M., González-Rojas, O.: Learning accurate LSTM models of business processes. In: Hildebrandt, T., van Dongen, B.F., Röglinger, M., Mendling, J. (eds.) BPM 2019. LNCS, vol. 11675, pp. 286–302. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-26619-6_19
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training. Technical report, OpenAI (2018)
Moon, J., Park, G., Jeong, J.: Pop-on: prediction of process using one-way language model based on NLP approach. Appl. Sci. 11(2), 864 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
van Luijken, M., Ketykó, I., Mannhardt, F. (2024). An Experiment on Transfer Learning for Suffix Prediction on Event Logs. In: De Weerdt, J., Pufahl, L. (eds) Business Process Management Workshops. BPM 2023. Lecture Notes in Business Information Processing, vol 492. Springer, Cham. https://doi.org/10.1007/978-3-031-50974-2_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-50974-2_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-50973-5
Online ISBN: 978-3-031-50974-2
eBook Packages: Computer ScienceComputer Science (R0)