Skip to main content

Learning to Act: A Reinforcement Learning Approach to Recommend the Best Next Activities

  • Conference paper
  • First Online:
Business Process Management Forum (BPM 2022)

Abstract

The rise of process data availability has recently led to the development of data-driven learning approaches. However, most of these approaches restrict the use of the learned model to predict the future of ongoing process executions. The goal of this paper is moving a step forward and leveraging available data to learning to act, by supporting users with recommendations derived from an optimal strategy (measure of performance). We take the optimization perspective of one process actor and we recommend the best activities to execute next, in response to what happens in a complex external environment, where there is no control on exogenous factors. To this aim, we investigate an approach that learns, by means of Reinforcement Learning, the optimal policy from the observation of past executions and recommends the best activities to carry on for optimizing a Key Performance Indicator of interest. The validity of the approach is demonstrated on two scenarios taken from real-life data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    In this paper we set \(\gamma =1\), hence equally weighting the reward obtained at each action points of the target actor.

  2. 2.

    The information on the average interest rate is extracted from the BPI2017 [5] dataset which contains data from the same financial institution.

  3. 3.

    We estimate the average salary of a bank employed in the Netherlands from https://www.salaryexpert.com/salary/job/banking-disbursement-clerk/netherlands.

  4. 4.

    The complete MDP description is available at tinyurl.com/2p8aytrb.

  5. 5.

    The MDP actions in this scenario take into account, besides the activity name, also the 2-month interval (since the creation of the fine) in which the activity has been carried out (2months).

References

  1. De Giacomo, G., Iocchi, L., Favorito, M., Patrizi, F.: Foundations for restraining bolts: Reinforcement learning with LTLF/LDLF restraining specifications. In: Proceedings of the 29th International Conference on Automated Planning and Scheduling, ICAPS 2018, pp. 128–136. AAAI Press (2019)

    Google Scholar 

  2. de Leoni, M., Dees, M., Reulink, L.: Design and evaluation of a process-aware recommender system based on prescriptive analytics. In: 2nd International Conference on Process Mining (ICPM 2020), pp. 9–16. IEEE (2020)

    Google Scholar 

  3. Di Francescomarino, C., Ghidini, C., Maggi, F.M., Milani, F.: Predictive process monitoring methods: which one suits me best? In: Weske, M., Montali, M., Weber, I., vom Brocke, J. (eds.) BPM 2018. LNCS, vol. 11080, pp. 462–479. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98648-7_27

    Chapter  Google Scholar 

  4. van Dongen, B.: Bpi challenge 2012, April 2012. https://doi.org/10.4121/uuid:3926db30-f712-4394-aebc-75976070e91f

  5. van Dongen, B.: Bpi challenge 2017, February 2017. https://doi.org/10.4121/uuid:5f3067df-f10b-45da-b98b-86ae4c7a310b

  6. Dumas, M.: Constructing digital twins for accurate and reliable what-if business process analysis. In: Proceedings of the International Workshop on BPM Problems to Solve Before We Die (PROBLEMS 2021). CEUR Workshop Proceedings, vol. 2938, pp. 23–27. CEUR-WS.org (2021)

    Google Scholar 

  7. Dumas, M., et al.: Augmented business process management systems: a research manifesto. CoRR abs/2201.12855 (2022). https://arxiv.org/abs/2201.12855

  8. Fahrenkrog-Petersen, S.A., et al.: Fire now, fire later: alarm-based systems for prescriptive process monitoring. arXiv preprint arXiv:1905.09568 (2019)

  9. Gröger, C., Schwarz, H., Mitschang, B.: Prescriptive analytics for recommendation-based business process optimization. In: International Conference on Business Information Systems. pp. 25–37. Springer (2014). https://doi.org/10.1007/978-3-319-06695-0_3

  10. Hu, J., Niu, H., Carrasco, J., Lennox, B., Arvin, F.: Voronoi-based multi-robot autonomous exploration in unknown environments via deep reinforcement learning. IEEE Trans. Vehicul. Technol. 69(12), 14413–14423 (2020)

    Article  Google Scholar 

  11. Huang, Z., van der Aalst, W., Lu, X., Duan, H.: Reinforcement learning based resource allocation in business process management. Data Know. Eng. 70(1), 127–145 (2011)

    Article  Google Scholar 

  12. Kubrak, K., Milani, F., Nolte, A., Dumas, M.: Prescriptive process monitoring: Quo vadis? CoRR abs/2112.01769 (2021). https://arxiv.org/abs/2112.01769

  13. de Leoni, M.M., Mannhardt, F.: Road traffic fine management process (2015). https://doi.org/10.4121/uuid:270fd440-1057-4fb9-89a9-b699b47990f5

  14. Maggi, F.M., Di Francescomarino, C., Dumas, M., Ghidini, C.: Predictive monitoring of business processes. In: Jarke, M., et al. (eds.) CAiSE 2014. LNCS, vol. 8484, pp. 457–472. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07881-6_31

  15. Mannhardt, F., de Leoni, M., Reijers, H.A., van der Aalst, W.M.P.: Balanced multi-perspective checking of process conformance. Computing 98(4), 407–437 (2015). https://doi.org/10.1007/s00607-015-0441-1

    Article  MathSciNet  MATH  Google Scholar 

  16. Massimo, D., Ricci, F.: Harnessing a generalised user behaviour model for next-poi recommendation. In: Proceedings of the 12th ACM Conference on Recommender Systems, RecSys 2018, pp. 402–406. ACM (2018)

    Google Scholar 

  17. Metzger, A., Kley, T., Palm, A.: Triggering proactive business process adaptations via online reinforcement learning. In: International Conference on Business Process Management, pp. 273–290. Springer (2020). https://doi.org/10.1007/978-3-030-58666-9_16

  18. Metzger, A., Neubauer, A., Bohn, P., Pohl, K.: Proactive process adaptation using deep learning ensembles. In: International Conference on Advanced Information Systems Engineering, pp. 547–562. Springer (2019). https://doi.org/10.1007/978-3-030-21290-2_34

  19. Park, G., Song, M.: Prediction-based resource allocation using LSTM and minimum cost and maximum flow algorithm. In: International Conference on Process Mining, ICPM 2019, pp. 121–128. IEEE (2019)

    Google Scholar 

  20. Shoush, M., Dumas, M.: Prescriptive process monitoring under resource constraints: a causal inference approach. CoRR abs/2109.02894 (2021). arxiv.org/abs/2109.02894

  21. Silvander, J.: Business process optimization with reinforcement learning. In: Shishkov, B. (ed.) BMSD 2019. LNBIP, vol. 356, pp. 203–212. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24854-3_13

    Chapter  Google Scholar 

  22. Sindhgatta, R., Ghose, A., Dam, H.K.: Context-aware analysis of past process executions to aid resource allocation decisions. In: Nurcan, S., Soffer, P., Bajec, M., Eder, J. (eds.) CAiSE 2016. LNCS, vol. 9694, pp. 575–589. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39696-5_35

    Chapter  Google Scholar 

  23. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press, 2nd edn. (2018)

    Google Scholar 

  24. Teinemaa, I., Tax, N., de Leoni, M., Dumas, M., Maggi, F.M.: Alarm-based prescriptive process monitoring. In: Weske, M., Montali, M., Weber, I., vom Brocke, J. (eds.) BPM 2018. LNBIP, vol. 329, pp. 91–107. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98651-7_6

    Chapter  Google Scholar 

  25. Weinzierl, S., Dunzer, S., Zilker, S., Matzner, M.: Prescriptive business process monitoring for recommending next best actions. In: Fahland, D., Ghidini, C., Becker, J., Dumas, M. (eds.) BPM 2020. LNBIP, vol. 392, pp. 193–209. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58638-6_12

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Massimiliano Ronzani .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Branchi, S., Di Francescomarino, C., Ghidini, C., Massimo, D., Ricci, F., Ronzani, M. (2022). Learning to Act: A Reinforcement Learning Approach to Recommend the Best Next Activities. In: Di Ciccio, C., Dijkman, R., del Río Ortega, A., Rinderle-Ma, S. (eds) Business Process Management Forum. BPM 2022. Lecture Notes in Business Information Processing, vol 458. Springer, Cham. https://doi.org/10.1007/978-3-031-16171-1_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16171-1_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16170-4

  • Online ISBN: 978-3-031-16171-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics