skip to main content
10.1145/3297280.3297545acmconferencesArticle/Chapter ViewAbstractPublication PagessacConference Proceedingsconference-collections
poster

Is it safe to dockerize my database benchmark?

Published: 08 April 2019 Publication History

Abstract

Docker seems to be an attractive solution for cloud database benchmarking as it simplifies the setup process through pre-built images that are portable and simple to maintain. However, the usage of Docker for benchmarking is only valid if there is no effect on measurement results. Existing work has so far only focused on the performance overheads that Docker directly induces for specific applications. In this paper, we have studied indirect effects of dockerization on the results of database benchmarking. Among others, our results clearly show that containerization has a measurable and non-constant influence on measurement results and should, hence, only be used after careful analysis.

References

[1]
Q. Ali, B. Agrawal, and D. Bergamasco. Docker containers performance in vmware vsphere. https://blogs.vmware.com/performance/2014/10/docker-containers-performance-vmware-vsphere.html (accessed on Apr 21, 2018), 2014.
[2]
D. Bermbach, J. Kuhlenkamp, A. Dey, S. Sakr, and R. Nambiar. Towards an Extensible Middleware for Database Benchmarking. In Proc. of TPCTC. Springer, 2014.
[3]
D. Bermbach, E. Wittern, and S. Tai. Cloud Service Benchmarking: Measuring Quality of Cloud Services from a Client Perspective. Springer, 2017.
[4]
S. Ceesay, A. Barker, and B. Varghese. Plug and play bench: Simplifying big data benchmarking using containers. CoRR, abs/1711.09138, 2017.
[5]
M. T. Chung, N. Quang-Hung, M.-T. Nguyen, and N. Thoai. Using docker in high performance computing applications. In Proc. of ICCE. IEEE, 2016.
[6]
P. Di Tommaso, E. Palumbo, M. Chatzou, P. Prieto, M. L. Heuer, and C. Notredame. The impact of docker containers on the performance of genomic pipelines. PeerJ, 3, 2015.
[7]
W. Felter, A. Ferreira, R. Rajamony, and J. Rubio. An updated performance comparison of virtual machines and linux containers. In Technical Report RC25482. IBM Research, 2014.
[8]
V. Ferme, A. Ivanckikj, C. Pautasso, M. Skouradaki, and F. Leymann. A container-centric methodology for benchmarking workflow management systems. In Proc. of CLOSER 2017. SciTePress, 2016.
[9]
E. Folkerts, A. Alexandrov, K. Sachs, A. Iosup, V. Markl, and C. Tosun. Benchmarking in the cloud: What it should, can, and cannot be. In Proc. of TPCTC 2012. Springer, 2013.
[10]
M. Grambow, J. Hasenburg, T. Pfandzelter, and D. Bermbach. Dockerization impacts in database performance benchmarking. In Technical Report MCC.2018.1. TU Berlin & ECDF, Mobile Cloud Computing Research Group. https://github.com/martingrambow/dockerExperiments, 2018.
[11]
J. Gray. Database and transaction processing handbook. The Benchmark Handbook for Database and Transaction Systems, 1993.
[12]
K. Huppler. The art of building a good benchmark. In Proc. of TPCTC 2009. Springer, 2009.
[13]
D. Merkel. Docker: lightweight linux containers for consistent development and deployment. Linux Journal, 2014(vol. 239), 2014.
[14]
S. Müller, D. Bermbach, S. Tai, and F. Pallas. Benchmarking the performance impact of transport layer security in cloud database systems. In Proc. of IC2E. IEEE, 2014.
[15]
T. Palit, Y. Shen, and M. Ferdman. Demystifying cloud benchmarking. In Proc. of ISPASS. IEEE, 2016.
[16]
J. v. Kistowski, J. A. Arnold, K. Huppler, K.-D. Lange, J. L. Henning, and P. Cao. How to build a benchmark. In Proc. of ICPE 2015. ACM, 2015.
[17]
B. Varghese, L. T. Subba, L. Thai, and A. Barker. Container-based cloud virtual machine benchmarking. In Proc. of IC2E. IEEE, 2016.

Cited By

View all
  • (2023)Using Microbenchmark Suites to Detect Application Performance ChangesIEEE Transactions on Cloud Computing10.1109/TCC.2022.321794711:3(2575-2590)Online publication date: 1-Jul-2023
  • (2023)MockFog 2.0: Automated Execution of Fog Application Experiments in the CloudIEEE Transactions on Cloud Computing10.1109/TCC.2021.307498811:1(58-70)Online publication date: 1-Jan-2023
  • (2021)Using application benchmark call graphs to quantify and improve the practical relevance of microbenchmark suitesPeerJ Computer Science10.7717/peerj-cs.5487(e548)Online publication date: 28-May-2021
  • Show More Cited By

Index Terms

  1. Is it safe to dockerize my database benchmark?

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      SAC '19: Proceedings of the 34th ACM/SIGAPP Symposium on Applied Computing
      April 2019
      2682 pages
      ISBN:9781450359337
      DOI:10.1145/3297280
      Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 08 April 2019

      Check for updates

      Author Tags

      1. database benchmarking
      2. docker
      3. performance

      Qualifiers

      • Poster

      Conference

      SAC '19
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

      Upcoming Conference

      SAC '25
      The 40th ACM/SIGAPP Symposium on Applied Computing
      March 31 - April 4, 2025
      Catania , Italy

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)13
      • Downloads (Last 6 weeks)2
      Reflects downloads up to 18 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2023)Using Microbenchmark Suites to Detect Application Performance ChangesIEEE Transactions on Cloud Computing10.1109/TCC.2022.321794711:3(2575-2590)Online publication date: 1-Jul-2023
      • (2023)MockFog 2.0: Automated Execution of Fog Application Experiments in the CloudIEEE Transactions on Cloud Computing10.1109/TCC.2021.307498811:1(58-70)Online publication date: 1-Jan-2023
      • (2021)Using application benchmark call graphs to quantify and improve the practical relevance of microbenchmark suitesPeerJ Computer Science10.7717/peerj-cs.5487(e548)Online publication date: 28-May-2021
      • (2021)Enabling Containerized, Parametric and Distributed Database Deployment and Benchmarking as a ServiceCompanion of the ACM/SPEC International Conference on Performance Engineering10.1145/3447545.3451188(77-80)Online publication date: 19-Apr-2021
      • (2020)Benchmarking microservice performanceProceedings of the 35th Annual ACM Symposium on Applied Computing10.1145/3341105.3373875(232-241)Online publication date: 30-Mar-2020
      • (2019)MockFog: Emulating Fog Computing Infrastructure in the Cloud2019 IEEE International Conference on Fog Computing (ICFC)10.1109/ICFC.2019.00026(144-152)Online publication date: Jun-2019

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media