Abstract
Since NoSQL database schema evolution is an important cost and time factor in the development of software applications, a standardized benchmark is essential for the comparison and evaluation of different schema evolution management systems. This benchmark should be simple to be set up, its design and usage ergonomic, its results straightforward for interpretation and reproduction – even after decades. Therefore, we present the implementation of a benchmark using Docker containers. By using containers with databases that already contain the test data, containers with the schema evolution system to be measured and the possibility to use the benchmark system itself in a container, it is very convenient to get the benchmark up and running. We also provide a data generator that creates individual data sets and/or reproduces real data once the schema is known. We demonstrate the flexibility and easy application of our approach by means of several experiments and discuss their results.
This work has been funded by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 385808805.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
https://db-engines.com/de/ranking (accessed July 17, 2021).
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
References
Bermbach, D., Kuhlenkamp, J., Dey, A., Sakr, S., Nambiar, R.: Towards an extensible middleware for database benchmarking. In: Nambiar, R., Poess, M. (eds.) TPCTC 2014. LNCS, vol. 8904, pp. 82–96. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-15350-6_6
Braininger, D., Mauerer, W., Scherzinger, S.: Replicability and reproducibility of a schema evolution study in embedded databases. In: Grossmann, G., Ram, S. (eds.) ER 2020. LNCS, vol. 12584, pp. 210–219. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-65847-2_19
Curino, C., Moon, H.J., Tanca, L., Zaniolo, C.: Schema evolution in Wikipedia - toward a web information system benchmark. In: ICEIS 2008 (2008)
Felter, W., Ferreira, A., Rajamony, R., Rubio, J.: An updated performance comparison of virtual machines and Linux containers. In: ISPASS 2015. IEEE (2015)
Hillenbrand, A., Levchenko, M., Störl, U., Scherzinger, S., Klettke, M.: MigCast: putting a price tag on data model evolution in NoSQL data stores. In: SIGMOD 2019. ACM (2019)
Hillenbrand, A., Störl, U., Levchenko, M., Nabiyev, S., Klettke, M.: Towards self-adapting data migration in the context of schema evolution in NoSQL databases. In: ICDE 2020 (2020)
Klettke, M., Störl, U., Shenavai, M., Scherzinger, S.: NoSQL schema evolution and big data migration at scale. In: ICBD 2016. IEEE (2016)
Möller, M.L., Berton, N., Klettke, M., Scherzinger, S., Störl, U.: jHound: Large-Scale Profiling of Open JSON Data. In: BTW 2019. GI (2019)
Möller, M.L., Klettke, M., Störl, U.: EvoBench - a framework for benchmarking schema evolution in NoSQL. In: ICBD 2020. IEEE (2020)
Möller, M.L., Scherzinger, S., Klettke, M., Störl, U.: Why it is time for yet another schema evolution benchmark. In: Herbaut, N., La Rosa, M. (eds.) CAiSE 2020. LNBIP, vol. 386, pp. 113–125. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58135-0_10
Scherzinger, S., Cerqueus, T., de Almeida, E.C.: ControVol: a framework for controlled schema evolution in NoSQL application development. In: ICDE 2015. IEEE (2015)
Scherzinger, S., Klettke, M., Störl, U.: Managing schema evolution in NoSQL data stores. In: DBPL 2013 (2013)
Scherzinger, S., Sidortschuck, S.: An empirical study on the design and evolution of NoSQL database schemas. In: Dobbie, G., Frank, U., Kappel, G., Liddle, S.W., Mayr, H.C. (eds.) ER 2020. LNCS, vol. 12400, pp. 441–455. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-62522-1_33
Shirinbab, S., Lundberg, L., Casalicchio, E.: Performance evaluation of containers and virtual machines when running Cassandra workload concurrently. Concurr. Comput. Pract. Exp. 32, e5693 (2020)
Störl, U., et al.: Curating variational data in application development. In: ICDE 2018. IEEE (2018)
Wevers, L., Hofstra, M., Tammens, M., Huisman, M., van Keulen, M.: A benchmark for online non-blocking schema transformations. In: DATA 2015. SciTePress (2015)
Zhang, C., Lu, J.: Holistic evaluation in multi-model databases benchmarking. Distrib. Parallel Databases 39(1), 1–33 (2019). https://doi.org/10.1007/s10619-019-07279-6
Zhang, C., Lu, J., Xu, P., Chen, Y.: UniBench: a benchmark for multi-model database management systems. In: Nambiar, R., Poess, M. (eds.) TPCTC 2018. LNCS, vol. 11135, pp. 7–23. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11404-6_2
Acknowledgements
We want to thank Stefanie Scherzinger, Christina Ehlinger and Edson Lucas Filho for their constructive feedback and inspired discussions.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Conrad, A., Möller, M.L., Kreiter, T., Mair, JC., Klettke, M., Störl, U. (2022). EvoBench: Benchmarking Schema Evolution in NoSQL. In: Nambiar, R., Poess, M. (eds) Performance Evaluation and Benchmarking. TPCTC 2021. Lecture Notes in Computer Science(), vol 13169. Springer, Cham. https://doi.org/10.1007/978-3-030-94437-7_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-94437-7_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-94436-0
Online ISBN: 978-3-030-94437-7
eBook Packages: Computer ScienceComputer Science (R0)