Article

Automated benchmarking and analysis tool

Authors:

Tomas Kalibera,

Jakub Lehotsky,

Branislav Repcek,

Michal Tomcanyi,

Antonin Tomecek,

Jaroslav UrbanAuthors Info & Claims

valuetools '06: Proceedings of the 1st international conference on Performance evaluation methodolgies and tools

Pages 5 - es

https://doi.org/10.1145/1190095.1190101

Published: 11 October 2006 Publication History

Abstract

Benchmarking is an important performance evaluation technique that provides performance data representative of real systems. Such data can be used to verify the results of performance modeling and simulation, or to detect performance changes. Automated benchmarking is an increasingly popular approach to tracking performance changes during software development, which gives developers a timely feedback on their work. In contrast with the advances in modeling and simulation tools, the tools for automated benchmarking are usually being implemented ad-hoc for each project, wasting resources and limiting functionality.We present the result of project BEEN, a generic tool for automated benchmarking in a heterogeneous distributed environment. BEEN automates all steps of a benchmark experiment from software building and deployment through measurement and load monitoring to the evaluation of results. The notable features include separation of measurement from the evaluation and ability to adaptively scale the benchmark experiment based on the evaluation. BEEN has been designed to facilitate automated detection of performance changes during software development (regression benchmarking).

References

[1]

Advanced Technology Labs. Agent and distributed objects quality of service. http://www.atl.external.lmco.com/projects/QoS, 2006.]]

[2]

BEEN Developers. Benchmarking environment (BEEN). http://nenya.ms.mff.cuni.cz/been, 2006.]]

[3]

L. Bulej, T. Kalibera, and P. Tuma. Repeated results analysis for middleware regression benchmarking. Performance Evaluation, 60(1--4):345--358, May 2005.]]

Digital Library

[4]

M. Courson, A. Mink, G. Marçais, and B. Traverse. An automated benchmarking toolset. In HPCN Europe, volume 1823 of LNCS, pages 497--506. Springer, 2000.]]

Digital Library

[5]

D. Grisby et al. Free high performance orb. http://omniorb.sourceforge.net, 2006.]]

[6]

B. Dillenseger and E. Cecchet. CLIF is a Load Injection Framework. In Workshop on Middleware Benchmarking: Approaches, Results, Experiences, OOPSLA 2003, Oct. 2003.]]

[7]

Distributed Systems Research Group. Mono regression benchmarking. http://nenya.ms.mff.cuni.cz/projects/mono, 2005.]]

[8]

Distributed Systems Research Group. Comprehensive CORBA benchmarking. http://nenya.ms.mff.cuni.cz/projects/corba/xampler.html, 2006.]]

[9]

DOC Group. TAO perf.scoreboard.http://www.dre.vanderbilt.edu/stats/performance.shtml, 2006.]]

[10]

Free Software Foundation. The R project for statistical computing. http://www.r-project.org, 2006.]]

[11]

T. Kalibera, L. Bulej, and P. Tuma. Generic environment for full automation of benchmarking. In SOQUA/TECOS, volume 58 of LNI, pages 125--132. GI, 2004.]]

[12]

T. Kalibera, L. Bulej, and P. Tuma. Benchmark precision and random initial state. In SPECTS 2005, pages 853--862, San Diego, CA, USA, July 2005. SCS.]]

[13]

T. Kalibera and P. Tuma. Precise regression benchmarking with random effects: Improving Mono benchmark results. In Formal Methods and Stochastic Models for Performance Evaluation, volume 4054 of LNCS, pages 63--77. Springer, June 2006.]]

Digital Library

[14]

A. M. Memon, A. A. Porter, C. Yilmaz, A. Nagarajan, D. C. Schmidt, and B. Natarajan. Skoll: Distributed continuous quality assurance. In ICSE, pages 459--468. IEEE Computer Society, 2004.]]

Digital Library

[15]

M. Prochazka, A. Madan, J. Vitek, and W. Liu. RTJBench: A Real-Time Java Benchmarking Framework. In Component And Middleware Performance Workshop, OOPSLA 2004, Oct. 2004.]]

[16]

Supercomputer Computations Research Institute, Florida State University. Distributed queueing system. http://packages.qa.debian.org/d/dqs.html, 1998.]]

[17]

University Corporation for Atmospheric Research. Network Common Data Form. http://www.unidata.ucar.edu/software/netcdf, 2006.]]

[18]

C. Yilmaz, A. S. Krishna, A. M. Memon, A. A. Porter, D. C. Schmidt, A. S. Gokhale, and B. Natarajan. Main effects screening: a distributed continuous quality assurance process for monitoring performance degradation in evolving software systems. In ICSE, pages 293--302. ACM, 2005.]]

Digital Library

Cited By

De Oliveira AFischmeister SDiwan AHauswirth MSweeney P(2017)Perphecy: Performance Regression Test Selection Made Simple but Effective2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST.2017.17(103-113)Online publication date: Mar-2017
https://doi.org/10.1109/ICST.2017.17
Waller JEhmke NHasselbring W(2015)Including Performance Benchmarks into Continuous Integration to Enable DevOpsACM SIGSOFT Software Engineering Notes10.1145/2735399.273541640:2(1-4)Online publication date: 3-Apr-2015
https://dl.acm.org/doi/10.1145/2735399.2735416
Hauck MKuperberg MHuber NReussner R(2014)Deriving performance-relevant infrastructure properties through model-based experiments with GinpexSoftware and Systems Modeling (SoSyM)10.1007/s10270-013-0335-713:4(1345-1365)Online publication date: 1-Oct-2014
https://dl.acm.org/doi/10.1007/s10270-013-0335-7
Show More Cited By

Index Terms

Automated benchmarking and analysis tool

Recommendations

Repeated results analysis for middleware regression benchmarking
Performance modelling and evaluation of high-performance parallel and distributed systems

The paper outlines the concept of regression benchmarking as a variant of regression testing focused at detecting performance regressions. Applying the regression benchmarking in the area of middleware development, the paper explains how the regression ...
Precise regression benchmarking with random effects: improving mono benchmark results
EPEW'06: Proceedings of the Third European conference on Formal Methods and Stochastic Models for Performance Evaluation

Benchmarking as a method of assessing software performance is known to suffer from random fluctuations that distort the observed performance. In this paper, we focus on the fluctuations caused by compilation. We show that the design of a benchmarking ...
Smart CloudBench -- Automated Performance Benchmarking of the Cloud
CLOUD '13: Proceedings of the 2013 IEEE Sixth International Conference on Cloud Computing

As the rate of cloud computing adoption grows, so does the need for consumption assistance. Enterprises that are looking to migrate their IT systems to the cloud, would like to quickly identify providers that offer resources with the most appropriate ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

valuetools '06: Proceedings of the 1st international conference on Performance evaluation methodolgies and tools

October 2006

638 pages

ISBN:1595935045

DOI:10.1145/1190095

General Chairs:
Luciano Lenzini
University of Pisa, Italy
,
Rene Cruz
UCSD, US

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 October 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Acceptance Rates

Overall Acceptance Rate 90 of 196 submissions, 46%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
842
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

De Oliveira AFischmeister SDiwan AHauswirth MSweeney P(2017)Perphecy: Performance Regression Test Selection Made Simple but Effective2017 IEEE International Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST.2017.17(103-113)Online publication date: Mar-2017
https://doi.org/10.1109/ICST.2017.17
Waller JEhmke NHasselbring W(2015)Including Performance Benchmarks into Continuous Integration to Enable DevOpsACM SIGSOFT Software Engineering Notes10.1145/2735399.273541640:2(1-4)Online publication date: 3-Apr-2015
https://dl.acm.org/doi/10.1145/2735399.2735416
Hauck MKuperberg MHuber NReussner R(2014)Deriving performance-relevant infrastructure properties through model-based experiments with GinpexSoftware and Systems Modeling (SoSyM)10.1007/s10270-013-0335-713:4(1345-1365)Online publication date: 1-Oct-2014
https://dl.acm.org/doi/10.1007/s10270-013-0335-7
Bulej LBureš TKeznikl JKoubková APodzimek ATůma PKaeli DRolia JJohn LKrishnamurthy D(2012)Capturing performance assumptions using stochastic performance logicProceedings of the 3rd ACM/SPEC International Conference on Performance Engineering10.1145/2188286.2188345(311-322)Online publication date: 22-Apr-2012
https://dl.acm.org/doi/10.1145/2188286.2188345
Hauck MKuperberg MHuber NReussner RCrnkovic IStafford JPetriu DHappe JInverardi P(2011)GinpexProceedings of the joint ACM SIGSOFT conference -- QoSA and ACM SIGSOFT symposium -- ISARCS on Quality of software architectures -- QoSA and architecting critical systems -- ISARCS10.1145/2000259.2000269(53-62)Online publication date: 20-Jun-2011
https://dl.acm.org/doi/10.1145/2000259.2000269
Vanhie-Van Gerwen JDe Poorter ELatré BMoerman IDemeester P(2010)Real-Life Performance of Protocol Combinations for Wireless Sensor NetworksProceedings of the 2010 IEEE International Conference on Sensor Networks, Ubiquitous, and Trustworthy Computing10.1109/SUTC.2010.49(189-196)Online publication date: 7-Jun-2010
https://dl.acm.org/doi/10.1109/SUTC.2010.49

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten