Abstract
Today, more and more companies model their business processes using languages such as Business Process Model and Notation (BPMN) or Business Process Execution Language (BPEL), and automate their processes by executing these models on appropriate workflow engines. To help choose the best fitting engine among different alternatives, several benchmarking initiatives have emerged, e.g., the BPEL/BPMN Engine Test System (betsy) that benchmarks conformance and expressiveness, and BenchFlow that benchmarks performance. However, their results are hard to analyze and compare for the typical end-user, developer or researcher. This paper tries to solve this issue by introducing (a) a common data model which can hold data of both betsy and BenchFlow tests and their results, (b) a transformer that can automatically transform the results of betsy and BenchFlow runs into the common data model, and (c) an interactive dashboard that visualizes the results according to the most important use cases. Hence, it enables the end-users, developers and researchers to analyze and compare the engines in a straight-forward manner.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
The license is given according to the enumeration at http://spdx.org/spdx-license-list/license-list-overview.
- 11.
See http://www.json.org/.
- 12.
- 13.
- 14.
References
Delgado, A., Calegari, D., Milanese, P., Falcon, R., GarcÃa, E.: A systematic approach for evaluating BPM systems: case studies on open source and proprietary tools. In: Damiani, E., Frati, F., Riehle, D., Wasserman, A.I. (eds.) OSS 2015. IFIP AICT, vol. 451, pp. 81–90. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-17837-0_8
Elias, M., Bezerianos, A.: Exploration views: understanding dashboard creation and customization for visualization novices. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011. LNCS, vol. 6949, pp. 274–291. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23768-3_23
Ferme, V., Ivanchikj, A., Pautasso, C.: A framework for benchmarking BPMN 2.0 workflow management systems. In: Motahari-Nezhad, H.R., Recker, J., Weidlich, M. (eds.) BPM 2015. LNCS, vol. 9253, pp. 251–259. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-23063-4_18
Few, S.: Information Dashboard Design: The Effective Visual Communication of Data. O’Reilly, Massachusetts (2006)
Geiger, M., Harrer, S., Lenhard, J.: Process engine benchmarking with betsy – current status and future directions. In: ZEUS, pp. 37–44, January 2016
Geiger, M., Harrer, S., Lenhard, J., Casar, M., Vorndran, A., Wirtz, G.: BPMN conformance in open source engines. In: SOSE, March 2015
Geiger, M., Harrer, S., Lenhard, J., Wirtz, G.: On the evolution of BPMN 2.0 support and implementation. In: SOSE, pp. 120–128, March 2016
Harrer, S., Lenhard, J., Wirtz, G.: BPEL conformance in open source engines. In: SOCA, pp. 237–244, December 2012
Harrer, S., Lenhard, J., Wirtz, G.: Open source versus proprietary software in service-orientation: the case of BPEL engines. In: Basu, S., Pautasso, C., Zhang, L., Fu, X. (eds.) ICSOC 2013. LNCS, vol. 8274, pp. 99–113. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-45005-1_8
Harrer, S., Nizamic, F., Wirtz, G., Lazovik, A.: Towards a robustness evaluation framework for BPEL engines. In: SOCA, pp. 199–206, November 2014
Harrer, S., Preißinger, C., Wirtz, G.: BPEL conformance in open source engines: the case of static analysis. In: SOCA, pp. 33–40, November 2014
ISO/IEC. ISO/IEC 25010:2011; Systems and software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – System and software quality models (2011)
ISO/IEC. ISO/IEC 19510:2013 – Information technology - Object Management Group Business Process Model and Notation, November 2013. v2.0.2
OASIS. Web Services Business Process Execution Language, April 2007. v2.0
Oppenheimer, D., Brown, A.B., Traupman, J., Broadwell, P., Patterson, D.A.: Practical issues in dependability benchmarking. In: EASY, p. 7 (2002)
Pauwels, K., Ambler, T., Clark, B.H., LaPointe, P., Reibstein, D., Skiera, B., Wierenga, B., Wiesel, T.: Dashboards as a service: why, what, how, and what research is needed? J. Serv. Res. 12, 175–189 (2009)
Röck, C., Harrer, S., Wirtz, G.: Performance benchmarking of BPEL engines: a comparison framework, status quo evaluation and challenges. In: SEKE, pp. 31–34, July 2014
Skouradaki, M., Ferme, V., Pautasso, C., Leymann, F., van Hoorn, A.: Micro-benchmarking BPMN 2.0 workflow management systems with workflow patterns. In: Nurcan, S., Soffer, P., Bajec, M., Eder, J. (eds.) CAiSE 2016. LNCS, vol. 9694, pp. 67–82. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39696-5_5
van der Aalst, W.M.P., ter Hofstede, A.H.M., Kiepuszewski, B., Barros, A.P.: Workflow patterns. Distribu. Parallel Databases 14(1), 5–51 (2003)
WfMC. Terminology & Glossary. WfMC, February 1999. v3.0
Yigitbasioglu, O.M., Velcu, O.: A review of dashboards in performance management: implications for design and research. IJAIS 13(1), 41–59 (2012)
Acknowledgment
We would like to express our gratitude to Jörg Lenhard and Matthias Geiger for fruitful discussions and feedback regarding the dashboard, and both, Marigianna Skouradaki and Vincenco Ferme, for helping in bringing the BenchFlow data into this dashboard.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this paper
Cite this paper
Bimamisa, D., Müller, M., Harrer, S., Wirtz, G. (2018). Interactive Dashboard for Workflow Engine Benchmarks. In: Lazovik, A., Schulte, S. (eds) Advances in Service-Oriented and Cloud Computing. ESOCC 2016. Communications in Computer and Information Science, vol 707. Springer, Cham. https://doi.org/10.1007/978-3-319-72125-5_14
Download citation
DOI: https://doi.org/10.1007/978-3-319-72125-5_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-72124-8
Online ISBN: 978-3-319-72125-5
eBook Packages: Computer ScienceComputer Science (R0)