skip to main content
10.1145/3341105.3373875acmconferencesArticle/Chapter ViewAbstractPublication PagessacConference Proceedingsconference-collections
research-article

Benchmarking microservice performance: a pattern-based approach

Published: 30 March 2020 Publication History

Abstract

Benchmarking microservices serves to understand and check their non-functional properties for relevant workloads and over time. Performing benchmarks, however, can be costly: each microservice requires the design and implementation of a benchmark, possibly repeatedly as the service evolves. As microservice APIs differ, benchmarking tools that assume common interfaces - like ones for databases - do not exist.
In this work, we present a pattern-based approach to reduce the efforts for defining microservice benchmarks, while still allowing to measure qualities of complex interactions. It assumes that microservices expose a REST API, described in a machine-understandable way, and allows developers to model interaction patterns from abstract operations that can be mapped to that API. Possible data-dependencies between operations are resolved at runtime. We implement a prototype of our approach, which we use to demonstrate that it can be applied to open-source microservices with little effort. Our work shows that pattern-based benchmarking of microservices is feasible and opens up opportunities for microservice providers and tooling developers.

References

[1]
David Bermbach, Jörn Kuhlenkamp, Akon Dey, Arunmoezhi Ramachandran, Alan Fekete, and Stefan Tai. 2017. BenchFoundry: A Benchmarking Framework for Cloud Storage Services. In Proc. of the International Conference on Service Oriented Computing (ICSOC 2017). Springer.
[2]
David Bermbach, Jörn Kuhlenkamp, Akon Dey, Sherif Sakr, and Raghunath Nambiar. 2014. Towards an Extensible Middleware for Database Benchmarking. In Proc. of the TPC Technology Conference on Performance Evaluation and Benchmarking (TPCTC 2014). Springer.
[3]
David Bermbach and Erik Wittern. 2016. Benchmarking Web API Quality. In Proc. of the International Conference on Web Engineering (ICWE 2016). Springer.
[4]
David Bermbach, Erik Wittern, and Stefan Tai. 2017. Cloud Service Benchmarking: Measuring Quality of Cloud Services from a Client Perspective. Springer.
[5]
Amir Hossein Borhani, Philipp Leitner, Bu-Sung Lee, Xiaorong Li, and Terence Hung. 2014. WPress: An Application-Driven Performance Benchmark for Cloud-Based Virtual Machines. In Proc. of the International Enterprise Distributed Object Computing Conference (EDOC 2014). IEEE.
[6]
Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and Russell Sears. 2010. Benchmarking Cloud Serving Systems with YCSB. In Proc. of the Symposium on Cloud Computing (SOCC 2010). ACM.
[7]
Akon Dey, Alan Fekete, Raghunath Nambiar, and Uwe Röhm. 2014. YCSB+ T: Benchmarking web-scale transactional databases. In Proc. of the International Conference on Data Engineering Workshops (ICDE 2014). IEEE.
[8]
Djellel Eddine Difallah, Andrew Pavlo, Carlo Curino, and Philippe Cudre-Mauroux. 2013. OLTP-Bench: An extensible testbed for benchmarking relational databases. Proceedings of the VLDB Endowment 7, 4 (2013).
[9]
Dror G Feitelson, Eitan Frachtenberg, and Kent L Beck. 2013. Development and deployment at facebook. IEEE Internet Computing 17, 4 (2013).
[10]
Enno Folkerts, Alexander Alexandrov, Kai Sachs, Alexandru Iosup, Volker Markl, and Cafer Tosun. 2013. Benchmarking in the cloud: What it should, can, and cannot be. In Proc. of the TPC Technology Conference on Performance Evaluation and Benchmarking (TPCTC 2012). Springer.
[11]
Martin Fowler. 2010. Richardson Maturity Model. Retrieved February 18, 2019 from https://martinfowler.com/articles/richardsonMaturityModel.html
[12]
Martin Grambow, Jonathan Hasenburg, Tobias Pfandzelter, and David Bermbach. 2019. Is it Safe to Dockerize my Database Benchmark?. In Proc. of the ACM Symposium on Applied Computing, Posters Track (SAC 2019). ACM.
[13]
Martin Grambow, Fabian Lehmann, and David Bermbach. 2019. Continuous Benchmarking: Using System Benchmarking in Build Pipelines. In Proc. of the Workshop on Service Quality and Quantitative Evaluation in new Emerging Technologies (SQUEET 2019). IEEE.
[14]
Jez Humble and David Farley. 2010. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Pearson Education.
[15]
Karl Huppler. 2009. The Art of Building a Good Benchmark. In Proc. of the TPC Technology Conference on Performance Evaluation and Benchmarking (TPCTC 2009). Springer.
[16]
Ana Ivanchikj, Ilija Gjorgjiev, and Cesare Pautasso. 2018. RESTalk Miner: Mining RESTful Conversations, Pattern Discovery and Matching. In Proc. of International Conference on Service-Oriented Computing - Workshops (ICSOC 2018). Springer.
[17]
Alexander Keller and Heiko Ludwig. 2003. The WSLA framework: Specifying and monitoring service level agreements for web services. Journal of Network and Systems Management 11 (2003).
[18]
Markus Klems, David Bermbach, and Rene Weinert. 2012. A Runtime Quality Measurement Framework for Cloud Database Service Systems. In Proc. of the International Conference on the Quality of Information and Communications Technology (QUATIC 2012). IEEE.
[19]
Markus Klems, Michael Menzel, and Robin Fischer. 2010. Consistency Bench-marking: Evaluating the Consistency Behavior of Middleware Services in the Cloud. In Service-Oriented Computing, Paul Maglio, Mathias Weske, Jian Yang, and Marcelo Fantinato (Eds.). Lecture Notes in Computer Science, Vol. 6470. Springer.
[20]
James Lewis and Martin Fowler. 2014. Microservices. Retrieved February 15, 2019 from https://martinfowler.com/articles/microservices.html
[21]
Heiko Ludwig, Alexander Keller, Asit Dan, Richard P King, and Richard Franck. 2003. Web service level agreement (WSLA) language specification. Ibm corporation (2003).
[22]
Malcolm D McIlroy, EN Pinson, and BA Tague. 1978. UNIX Time-Sharing System: Foreword. Bell System Technical Journal 57, 6 (1978).
[23]
Steffen Müller, David Bermbach, Stefan Tai, and Frank Pallas. 2014. Benchmarking the Performance Impact of Transport Layer Security in Cloud Database Systems. In Proc. of the International Conference on Cloud Engineering (IC2E 2014). IEEE.
[24]
Ioannis Papapanagiotou and Vinay Chella. 2018. NDBench: Benchmarking Microservices at Scale. arXiv e-prints (2018).
[25]
Swapnil Patil, Milo Polte, Kai Ren, Wittawat Tantisiriroj, Lin Xiao, Julio López, Garth Gibson, Adam Fuchs, and Billie Rinaldi. 2011. YCSB++: Benchmarking and Performance Debugging Advanced Features in Scalable Table Stores. In Proc. of the Symposium on Cloud Computing (SOCC 2011). ACM.
[26]
Tilmann Rabl, Michael Frank, Hatem Mousselly Sergieh, and Harald Kosch. 2010. A data generator for cloud-scale benchmarking. In Proc. of the TPC Technology Conference on Performance Evaluation and Benchmarking (TPCTC 2010). Springer.
[27]
Alex Rodriguez. 2008. Restful web services: The basics. IBM developer Works 33 (2008).
[28]
Gerald Schermann, Dominik Schöni, Philipp Leitner, and Harald C Gall. 2016. Bifrost: supporting continuous deployment with automated enactment of multi-phase live testing strategies. In Proc. of the International Middleware Conference (Middleware 2016). ACM.
[29]
Joel Scheuner, Philipp Leitner, Jürgen Cito, and Harald Gall. 2014. Cloud Work-Bench - Infrastructure-as-Code Based Cloud Benchmarking. In Proc. of the International Conference on Cloud Computing Technology and Science (CloudCom 2014). IEEE.
[30]
Jóakim v. Kistowski, Jeremy A. Arnold, Karl Huppler, Klaus-Dieter Lange, John L. Henning, and Paul Cao. 2015. How to Build a Benchmark. In Proc. of the ACM/SPEC International Conference on Performance Engineering (ICPE 2015). ACM.
[31]
Jan Waller, Nils C Ehmke, and Wilhelm Hasselbring. 2015. Including performance benchmarks into continuous integration to enable DevOps. ACM SIGSOFT Software Engineering Notes 40, 2 (2015).
[32]
Qing Zheng, Haopeng Chen, Yaguang Wang, Jiangang Duan, and Zhiteng Huang. 2012. Cosbench: A benchmark tool for cloud object storage services. In Proc. of the International Conference on Cloud Computing (CLOUD 2012). IEEE.

Cited By

View all
  • (2025)Unveiling the microservices testing methods, challenges, solutions, and solutions gaps: A systematic mapping studyJournal of Systems and Software10.1016/j.jss.2024.112232220(112232)Online publication date: Feb-2025
  • (2024)Hyperion: Hardware-Based High-Performance and Secure System for Container NetworksIEEE Transactions on Cloud Computing10.1109/TCC.2024.340317512:3(844-858)Online publication date: Jul-2024
  • (2024)HardWhale: A Hardware-Isolated Network Security Enforcement System for Cloud Environments2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS60910.2024.00053(496-507)Online publication date: 23-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SAC '20: Proceedings of the 35th Annual ACM Symposium on Applied Computing
March 2020
2348 pages
ISBN:9781450368667
DOI:10.1145/3341105
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 March 2020

Permissions

Request permissions for this article.

Check for updates

Qualifiers

  • Research-article

Conference

SAC '20
Sponsor:
SAC '20: The 35th ACM/SIGAPP Symposium on Applied Computing
March 30 - April 3, 2020
Brno, Czech Republic

Acceptance Rates

Overall Acceptance Rate 1,650 of 6,669 submissions, 25%

Upcoming Conference

SAC '25
The 40th ACM/SIGAPP Symposium on Applied Computing
March 31 - April 4, 2025
Catania , Italy

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)111
  • Downloads (Last 6 weeks)6
Reflects downloads up to 18 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Unveiling the microservices testing methods, challenges, solutions, and solutions gaps: A systematic mapping studyJournal of Systems and Software10.1016/j.jss.2024.112232220(112232)Online publication date: Feb-2025
  • (2024)Hyperion: Hardware-Based High-Performance and Secure System for Container NetworksIEEE Transactions on Cloud Computing10.1109/TCC.2024.340317512:3(844-858)Online publication date: Jul-2024
  • (2024)HardWhale: A Hardware-Isolated Network Security Enforcement System for Cloud Environments2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS60910.2024.00053(496-507)Online publication date: 23-Jul-2024
  • (2023)The Early Microbenchmark Catches the Bug -- Studying Performance Issues Using Micro- and Application BenchmarksProceedings of the IEEE/ACM 16th International Conference on Utility and Cloud Computing10.1145/3603166.3632128(1-10)Online publication date: 4-Dec-2023
  • (2022)Approach to the Design of a Microservice Architecture Based on PraxemeSYSTEM THEORY, CONTROL AND COMPUTING JOURNAL10.52846/stccj.2022.2.2.402:2(25-31)Online publication date: 31-Dec-2022
  • (2022)Proposed Methodology for Designing a Microservice Architecture2022 23rd International Carpathian Control Conference (ICCC)10.1109/ICCC54292.2022.9805930(303-308)Online publication date: 29-May-2022
  • (2022)Automated test-based learning and verification of performance models for microservices systemsJournal of Systems and Software10.1016/j.jss.2022.111225187:COnline publication date: 12-May-2022
  • (2022)Modeling Performance of Microservices Systems with Growth TheoryEmpirical Software Engineering10.1007/s10664-021-10088-027:2Online publication date: 11-Jan-2022
  • (2022)MSDBench: Understanding the Performance Impact of Isolation Domains on Microservice-Based IoT DeploymentsBenchmarking, Measuring, and Optimizing10.1007/978-3-031-31180-2_3(35-52)Online publication date: 7-Nov-2022
  • (2022)Characterizing and synthesizing the workflow structure of microservices in ByteDance CloudJournal of Software: Evolution and Process10.1002/smr.246734:8Online publication date: Jun-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media