Abstract
Experimental performance evaluation of stream processing engines (SPE) can be a great challenge. Aiming to make fair comparisons of different SPEs raises this bar even higher. One important reason for this challenge is the fact that these systems often use concepts that require expert knowledge for each SPE. To address this issue, we present Expose, a distributed performance evaluation framework for SPEs that enables a user through a declarative approach to specify experiments and conduct them on multiple SPEs in a fair way and with low effort. Experimenters with few technical skills can define and execute distributed experiments that can easily be replicated. We demonstrate Expose by defining a set of experiments based on the existing NEXMark benchmark and conduct a performance evaluation of Flink, Beam with the Flink runner, Siddhi, T-Rex, and Esper, on powerful and resource-constrained hardware.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
GitHub repository available at https://github.com/espv/expose.
References
Apache Beam. https://beam.apache.org. Accessed 6 Aug 2020
Esper. http://www.espertech.com/esper. Accessed 6 Aug 2020
Arasu, A., et al.: Linear road: a stream data management benchmark. In: Proceedings of the Thirtieth international Conference on Very Large Data Bases, vol. 30. pp. 480–491. VLDB Endowment (2004)
Begoli, E., Akidau, T., Hueske, F., Hyde, J., Knight, K., Knowles, K.: One SQL to rule them all-an efficient and syntactically idiomatic approach to management of streams and tables. In: Proceedings of the 2019 International Conference on Management of Data, pp. 1757–1772 (2019)
Boden, C., Alexandrov, A., Kunft, A., Rabl, T., Markl, V.: PEEL: a framework for benchmarking distributed systems and algorithms. In: Nambiar, R., Poess, M. (eds.) TPCTC 2017. LNCS, vol. 10661, pp. 9–24. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-72401-0_2
Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., Tzoumas, K.: Apache Flink: Stream and batch processing in a single engine. In: Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, vol. 36, no. 4 (2015)
Chintapalli, S., et al.: Benchmarking streaming computation engines: Storm, Flink and Spark streaming. In: 2016 IEEE international parallel and distributed processing symposium workshops (IPDPSW), pp. 1789–1792. IEEE (2016)
Cugola, G., Margara, A.: Complex event processing with T-REX. J. Syst. Softw. 85(8), 1709–1728 (2012)
Folkerts, E., Alexandrov, A., Sachs, K., Iosup, A., Markl, V., Tosun, C.: Benchmarking in the cloud: what it should, can, and cannot be. In: Nambiar, R., Poess, M. (eds.) TPCTC 2012. LNCS, vol. 7755, pp. 173–188. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-36727-4_12
Hanif, M., Yoon, H., Lee, C.: Benchmarking tool for modern distributed stream processing engines. In: 2019 International Conference on Information Networking (ICOIN), pp. 393–395. IEEE (2019)
Hesse, G., Matthies, C., Glass, K., Huegle, J., Uflacker, M.: Quantitative impact evaluation of an abstraction layer for data stream processing systems. In: 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), pp. 1381–1392. IEEE (2019)
Hesse, G., Reissaus, B., Matthies, C., Lorenz, M., Kraus, M., Uflacker, M.: Senska – towards an enterprise streaming benchmark. In: Nambiar, R., Poess, M. (eds.) TPCTC 2017. LNCS, vol. 10661, pp. 25–40. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-72401-0_3
Hochstein, L., Moser, R.: Ansible: Up and Running: Automating Configuration Management and Deployment the Easy Way. O’Reilly Media Inc, Sebastopol (2017)
Huppler, K.: The art of building a good benchmark. In: Nambiar, R., Poess, M. (eds.) Performance Evaluation and Benchmarking. TPCTC 2009. LNCS, vol. 5895, pp. 18–30. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-10424-4_3
Jepsen, T., Moshref, M., Carzaniga, A., Foster, N., Soulé, R.: Life in the fast lane: A line-rate linear road. In: Proceedings of the Symposium on SDN Research, pp. 1–7 (2018)
Karimov, J., Rabl, T., Katsifodimos, A., Samarev, R., Heiskanen, H., Markl, V.: Benchmarking distributed stream data processing systems. In: 2018 IEEE 34th International Conference on Data Engineering (ICDE), pp. 1507–1518. IEEE (2018)
Kiatipis, A., et al.: A survey of benchmarks to evaluate data analytics for smart-* applications. arXiv preprint arXiv:1910.02004 (2019)
Lu, R., Wu, G., Xie, B., Hu, J.: Stream bench: towards benchmarking modern distributed stream computing frameworks. In: 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing, pp. 69–78. IEEE (2014)
Mendes, M.R., Bizarro, P., Marques, P.: A framework for performance evaluation of complex event processing systems. In: Proceedings of the Second International Conference on Distributed Event-Based Systems, pp. 313–316 (2008)
Mendes, M.R., Bizarro, P., Marques, P.: Fincos: benchmark tools for event processing systems. In: Proceedings of the 4th ACM/SPEC International Conference on Performance Engineering, pp. 431–432 (2013)
Rabl, T., Frank, M., Danisch, M., Jacobsen, H.A., Gowda, B.: The vision of bigbench 2.0. In: Proceedings of the Fourth Workshop on Data analytics in the Cloud, pp. 1–4 (2015)
Shukla, A., Chaturvedi, S., Simmhan, Y.: RioTBench: an IoT benchmark for distributed stream processing systems. Concurrency Comput. Pract. Exp. 29(21), e4257 (2017)
Stonebraker, M., Çetintemel, U.: “One size fits all” an idea whose time has come and gone. In: Making Databases Work: The Pragmatic Wisdom of Michael Stonebraker, pp. 441–462 (2018)
Suhothayan, S., Gajasinghe, K., Loku Narangoda, I., Chaturanga, S., Perera, S., Nanayakkara, V.: Siddhi: a second look at complex event processing architectures. In: Proceedings of the 2011 ACM workshop on Gateway Computing Environments, pp. 43–50 (2011)
Tucker, P., Tufte, K., Papadimos, V., Maier, D.: NEXMark-a benchmark for queries over data streams (draft). OGI School of Science & Engineering at OHSU, September, Technical report (2008)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Volnes, E., Plagemann, T., Goebel, V., Kristiansen, S. (2021). EXPOSE: Experimental Performance Evaluation of Stream Processing Engines Made Easy. In: Nambiar, R., Poess, M. (eds) Performance Evaluation and Benchmarking. TPCTC 2020. Lecture Notes in Computer Science(), vol 12752. Springer, Cham. https://doi.org/10.1007/978-3-030-84924-5_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-84924-5_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-84923-8
Online ISBN: 978-3-030-84924-5
eBook Packages: Computer ScienceComputer Science (R0)