Abstract
Software has become a rapidly evolving artifact and Workflow Management Systems (WfMSs) are not an exception. WfMSs’ changes may impact key performance indicators or resource consumption levels may change among different versions. Thus, users considering a WfMS upgrade need to evaluate the extent of such changes for frequently issued workload. Deriving such information requires running performance experiments with appropriate workloads. In this paper, we propose a novel method for deriving a structurally representative workload from a given business process collection, which we later use to evaluate the performance and resource consumption over four versions of two open-source WfMSs, for different numbers of simulated users. In our case study scenario the results reveal relevant variations in the WfMSs’ performance and resource consumption, indicating a decrease in performance for newer versions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
We do not have an explicit vendors’ consent to publish WfMSs’ names.
- 6.
References
Bianculli, D., et al.: Automated performance assessment for service-oriented middleware: a case study on BPEL engines. In: Proceedings of WWW, pp. 141–150 (2010)
Duan, S., et al.: Apples and oranges: a comparison of RDF benchmarks and real RDF datasets. In: Proceedings of SIGMOD. Association for Computing Machinery (2011)
Feitelson, D.G.: Workload Modeling for Computer Systems Performance Evaluation. Cambridge University Press, New York (2015)
Ferme, V., Ivanchikj, A., Pautasso, C.: A framework for benchmarking BPMN 2.0 workflow management systems. In: Motahari-Nezhad, H.R., Recker, J., Weidlich, M. (eds.) BPM 2015. LNCS, vol. 9253, pp. 251–259. Springer, Cham (2015). doi:10.1007/978-3-319-23063-4_18
Ferme, V., Ivanchikj, A., Pautasso, C., Skouradaki, M., Leymann, F.: A Container-centric methodology for benchmarking workflow management systems. In: CLOSER 2016 - Proceedings of the 6th International Conference on Cloud Computing and Services Science, vol. 2, Rome, Italy, April 23-25, pp. 74–84 (2016). doi:10.5220/0005908400740084
Gupta, A.: Generating large-scale heterogeneous graphs for benchmarking. In: Rabl, T., Poess, M., Baru, C., Jacobsen, H.-A. (eds.) WBDB -2012. LNCS, vol. 8163, pp. 113–128. Springer, Heidelberg (2014). doi:10.1007/978-3-642-53974-9_11
Hollingsworth, D.: The workflow reference model. WfMC 68 (1995)
Jordan, D., et al.: Business process model and notation (BPMN) version 2.0. Object Management Group, Inc. (2011). http://www.omg.org/spec/BPMN/2.0/
Molyneaux, I.: The Art of Application Performance Testing: From Strategy to Tools, 2nd edn. O’Reilly Media, Sebastopol (2014)
Montgomery, D.C., Runger, G.C.: Applied Statistics and Probability for Engineers. Wiley, New York (2003)
Muehlen, M., Recker, J.: How much language is enough? Theoretical and practical use of the business process modeling notation. In: Bellahsène, Z., Léonard, M. (eds.) CAiSE 2008. LNCS, vol. 5074, pp. 465–479. Springer, Heidelberg (2008). doi:10.1007/978-3-540-69534-9_35
Pietsch, P., Wenzel, S.: Comparison of BPMN2 diagrams. In: Mendling, J., Weidlich, M. (eds.) BPMN 2012. LNBIP, vol. 125, pp. 83–97. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33155-8_7
Röck, C., et al.: Performance benchmarking of BPEL engines: a comparison framework, status quo evaluation and challenges. In: Proceedings of SEKE, pp. 31–34 (2014)
Skouradaki, M., Andrikopoulos, V., Leymann, F.: Representative BPMN 2.0 process model generation from recurring structures. In: Proceedings of ICWS 2016 (2016)
Skouradaki, M., Ferme, V., Pautasso, C., Leymann, F., Hoorn, A.: Micro-benchmarking BPMN 2.0 workflow management systems with workflow patterns. In: Nurcan, S., Soffer, P., Bajec, M., Eder, J. (eds.) CAiSE 2016. LNCS, vol. 9694, pp. 67–82. Springer, Cham (2016). doi:10.1007/978-3-319-39696-5_5
Skouradaki, M., Andrikopoulos, V., Kopp, O., Leymann, F.: RoSE: reoccurring structures detection in bpmn 2.0 process model collections. In: Debruyne, C. et al. (eds) On the Move to Meaningful Internet Systems: OTM 2016 Conferences. OTM 2016. LNCS, vol 10033. Springer, Cham (2016). doi:10.1007/978-3-319-48472-3_15
Vicknair, C., et al.: A comparison of a graph database and a relational database. In: Proceedings of ACM SE 2010. Association for Computing Machinery (ACM) (2010)
Wetzstein, B., et al.: Monitoring and analyzing influential factors of business process performance. In: Proceedings of EDOC 2009, pp. 141–150 (2009)
Acknowledgements
This work is partially funded by the Swiss National Science Foundation and the German Research Foundation through the BenchFlow: A Benchmark for Workflow Management Systems (DACH Grant Nr. 200021E-145062/1) project.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Ferme, V., Skouradaki, M., Ivanchikj, A., Pautasso, C., Leymann, F. (2017). Performance Comparison Between BPMN 2.0 Workflow Management Systems Versions. In: Reinhartz-Berger, I., Gulden, J., Nurcan, S., Guédria, W., Bera, P. (eds) Enterprise, Business-Process and Information Systems Modeling. BPMDS EMMSAD 2017 2017. Lecture Notes in Business Information Processing, vol 287. Springer, Cham. https://doi.org/10.1007/978-3-319-59466-8_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-59466-8_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-59465-1
Online ISBN: 978-3-319-59466-8
eBook Packages: Business and ManagementBusiness and Management (R0)