research-article

How to Detect Performance Changes in Software History: Performance Analysis of Software System Versions

Authors:

David Georg Reichelt,

Stefan KühneAuthors Info & Claims

ICPE '18: Companion of the 2018 ACM/SPEC International Conference on Performance Engineering

Pages 183 - 188

https://doi.org/10.1145/3185768.3186404

Published: 02 April 2018 Publication History

Abstract

Source code changes can affect the performance of software. Structured knowledge about classes of those changes could guide software developers in avoiding negative changes and improving the performance by positive changes. Neither a comprehensive overview nor a mature method for structured detection of those changes exists for this purpose. We address this research challenge by presenting Performance Analysis of Software Systems (PeASS). PeASS builds up a comprehensive knowledge base of changes affecting the performance of a software by analyzing the version history of a repository using its unit tests. It is based on a method for determining the significant performance changes between two unit tests by measurement and statistical analysis. Furthermore, PeASS uses regression test selection for saving measurement time and root cause isolation method for performance changes analysis. We demonstrate our methodology in the context of Java by analyzing the versions of Apache Commons IO.

References

[1]

J. P. S. Alcocer and A. Bergel. Tracking down performance variation against source code evolution. In Proceedings of the 11th Symposium on Dynamic Languages, DLS 2015, pages 129-139, New York, NY, USA, 2015. ACM.

Digital Library

[2]

E. Barrett, C. F. Bolz-Tereick, R. Killick, S. Mount, and L. Tratt. Virtual machine warmup blows hot and cold. Proceedings of the ACM on Programming Languages, 1(OOPSLA):52, 2017.

Digital Library

[3]

S. Becker, H. Koziolek, and R. Reussner. The palladio component model for model-driven performance prediction. Journal of Systems and Software, 82(1):3-22, 2009.

Digital Library

[4]

L. Bulej, T. Bureš, V. Horký, J. Kotrč, L. Marek, T. Trojánek, and P. Tuma. Unit testing performance with stochastic performance logic. Automated Software Engineering, 24(1):139-187, Mar 2017.

Digital Library

[5]

J. Chen and W. Shang. An exploratory study of performance regression introducing code changes. In Software Maintenance and Evolution (ICSME), 2017 IEEE International Conference on, pages 341-352. IEEE, 2017.

[6]

A. Georges, D. Buytaert, and L. Eeckhout. Statistically rigorous java performance evaluation. ACM SIGPLAN Notices, 42(10):57-76, 2007.

Digital Library

[7]

C. Heger, J. Happe, and R. Farahbod. Automated root cause isolation of performance regressions during software development. In ICPE 13, pages 27-38, New York, USA, 2013. ACM.

Digital Library

[8]

A. Hindle, A. Wilson, K. Rasmussen, E. J. Barlow, J. C. Campbell, and S. Romansky. Greenminer: A hardware based mining software repositories software energy consumption framework. In MSR 2014, pages 12-21, New York, USA, 2014. ACM.

Digital Library

[9]

G. Jin, L. Song, X. Shi, J. Scherpelz, and S. Lu. Understanding and detecting real-world performance bugs. In Proceedings of the 33rd ACM SIGPLAN PLDI, PLDI '12, pages 77-88, New York, USA, 2012. ACM.

Digital Library

[10]

T. Kalibera and R. Jones. Rigorous benchmarking in reasonable time. In ACM SIGPLAN Notices, volume 48, pages 63-74. ACM, 2013.

Digital Library

[11]

C. Laaber and P. Leitner. (h, g)opper: Performance history mining and analysis. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering, ICPE '17, pages 167-168, New York, NY, USA, 2017. ACM.

Digital Library

[12]

Y. Liu, C. Xu, and S.-C. Cheung. Characterizing and detecting performance bugs for smartphone applications. In Proceedings of the 36th ICPE, pages 1013-1024. ACM, 2014.

Digital Library

[13]

Q. Luo, D. Poshyvanyk, and M. Grechanik. Mining performance regression inducing code changes in evolving software. In Proceedings of the 13th International Conference on Mining Software Repositories, pages 25-36. ACM, 2016.

Digital Library

[14]

T. H. Nguyen, B. Adams, Z. M. Jiang, A. E. Hassan, M. Nasser, and P. Flora. Automated detection of performance regressions using statistical process control techniques. In ICPE, pages 299-310, New York, USA, 2012. ACM.

Digital Library

[15]

A. Nistor, T. Jiang, and L. Tan. Discovering, reporting, and fixing performance bugs. In MSR 2013, pages 237-246. IEEE Press, 2013.

Digital Library

[16]

M. Pradel, M. Huggler, and T. R. Gross. Performance regression testing of concurrent classes. In Proceedings of the 2014 International Symposium on Software Testing and Analysis, pages 13-25. ACM, 2014.

Digital Library

[17]

D. G. Reichelt and L. Braubach. Sicherstellung von performanzeigenschaften durch kontinuierliche performanztests mit dem kopeme framework. In Software Engineering, pages 119-124, 2014.

[18]

J. P. Sandoval Alcocer, A. Bergel, and M. T. Valente. Learning from source code history to identify performance failures. In Proceedings of the 7th ACM/SPEC on International Conference on Performance Engineering, ICPE '16, pages 37-48, New York, NY, USA, 2016. ACM.

Digital Library

[19]

M. Selakovic and M. Pradel. Performance issues and optimizations in javascript: an empirical study. In Proceedings of the 38th International Conference on Software Engineering, pages 61-72. ACM, 2016.

Digital Library

[20]

P. Stefan, V. Horky, L. Bulej, and P. Tuma. Unit testing performance in java projects: Are we there yet? In Proceedings of ACM/SPEC ICPE 2017, pages 401-412. ACM, 2017.

Digital Library

[21]

A. van Hoorn, J. Waller, and W. Hasselbring. Kieker: A framework for application performance monitoring and dynamic software analysis. In Proceedings of the 3rd joint ACM/SPEC International Conference on Performance Engineering (ICPE 2012), pages 247-248. ACM, April 2012.

Digital Library

[22]

S. Zaman, B. Adams, and A. E. Hassan. Security versus performance bugs: a case study on firefox. In MSR 2011, pages 93-102. ACM, 2011.

Digital Library

Cited By

dos Santos Lde Souza ÉEndo ATrubiani CPinciroli RVijaykumar N(2025)Performance regression testing initiativesInformation and Software Technology10.1016/j.infsof.2024.107641179:COnline publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1016/j.infsof.2024.107641
Kaltenecker CMühlbauer SGrebhahn ASiegmund NApel S(2023)Performance evolution of configurable software systems: an empirical studyEmpirical Software Engineering10.1007/s10664-023-10338-328:6Online publication date: 13-Nov-2023
https://doi.org/10.1007/s10664-023-10338-3
Wu JDong JFang RZhang WWang WZuo DDwyer MDamian DZeller A(2022)FADATestProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510169(896-908)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510169
Show More Cited By

Index Terms

How to Detect Performance Changes in Software History: Performance Analysis of Software System Versions
1. Software and its engineering

Recommendations

An Exploratory Study of the State of Practice of Performance Testing in Java-Based Open Source Projects
ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

The usage of open source (OS) software is wide-spread across many industries. While the functional quality of OS projects is considered to be similar to closed-source software, much is unknown about the quality in terms of performance. One challenge for ...
Software performance in the real world: personal lessons from the performance trauma team
WOSP '07: Proceedings of the 6th international workshop on Software and performance

In the nine years that we have been involved in software performance engineering (SPE) and performance testing engagements we have learned several things. Across numerous eCommerce applications and an enterprise CRM product suite, our knowledge base ...
The Maintenance and Evolution of Field-Representative Performance Tests
ICSME '14: Proceedings of the 2014 IEEE International Conference on Software Maintenance and Evolution

The rise of large-scale software systems poses new challenges for the software performance engineering field. Failures in these systems are typically associated with performance issues, rather than with feature bugs. Therefore, performance testing has ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPE '18: Companion of the 2018 ACM/SPEC International Conference on Performance Engineering

April 2018

212 pages

ISBN:9781450356299

DOI:10.1145/3185768

General Chairs:
Katinka Wolter
Free University of Berlin, Germany
,
Will Knottenbelt
Imperial College London, UK
,
Program Chairs:
André van Hoorn
University of Stuttgart, Germany
,
Manoj Nambiar
Tata Consultancy Services, India
,
Heiko Koziolek
ABB, Germany

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 April 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICPE '18

Sponsor:

ICPE '18: ACM/SPEC International Conference on Performance Engineering

April 9 - 13, 2018

Berlin, Germany

Acceptance Rates

Overall Acceptance Rate 252 of 851 submissions, 30%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
357
Total Downloads

Downloads (Last 12 months)28
Downloads (Last 6 weeks)1

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

dos Santos Lde Souza ÉEndo ATrubiani CPinciroli RVijaykumar N(2025)Performance regression testing initiativesInformation and Software Technology10.1016/j.infsof.2024.107641179:COnline publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1016/j.infsof.2024.107641
Kaltenecker CMühlbauer SGrebhahn ASiegmund NApel S(2023)Performance evolution of configurable software systems: an empirical studyEmpirical Software Engineering10.1007/s10664-023-10338-328:6Online publication date: 13-Nov-2023
https://doi.org/10.1007/s10664-023-10338-3
Wu JDong JFang RZhang WWang WZuo DDwyer MDamian DZeller A(2022)FADATestProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510169(896-908)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510169
Reichelt DKühne SHasselbring W(2022)Automated Identification of Performance Changes at Code Level2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS57517.2022.00096(916-925)Online publication date: Dec-2022
https://doi.org/10.1109/QRS57517.2022.00096
Delgado‐Pérez PSánchez ASegura SMedina‐Bulo I(2020)Performance mutation testingSoftware Testing, Verification and Reliability10.1002/stvr.172831:5Online publication date: 29-Jan-2020
https://doi.org/10.1002/stvr.1728
Chen JYu DHu HLi ZHu HGuéhéneuc YKhomh FSarro F(2019)Analyzing performance-aware code changes in software development processProceedings of the 27th International Conference on Program Comprehension10.1109/ICPC.2019.00049(300-310)Online publication date: 25-May-2019
https://dl.acm.org/doi/10.1109/ICPC.2019.00049
Reichelt DKühne SHasselbring WZimmermann TLawall JMarinov D(2019)PeASSProceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE.2019.00123(1146-1149)Online publication date: 10-Nov-2019
https://dl.acm.org/doi/10.1109/ASE.2019.00123
Mühlbauer SApel SSiegmund NZimmermann TLawall JMarinov D(2019)Accurate modeling of performance histories for evolving software systemsProceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE.2019.00065(640-652)Online publication date: 10-Nov-2019
https://dl.acm.org/doi/10.1109/ASE.2019.00065
Huerta-Guevara OAyala-Rivera VMurphy LPortillo-Dominguez A(2019)DYNAMOJM: A JMeter Tool for Performance Testing Using Dynamic Workload AdaptationTesting Software and Systems10.1007/978-3-030-31280-0_14(234-241)Online publication date: 8-Oct-2019
https://doi.org/10.1007/978-3-030-31280-0_14
Huerta-Guevara OAyala-Rivera VMurphy LPortillo-Dominguez A(2019)Towards an Efficient Performance Testing Through Dynamic Workload AdaptationTesting Software and Systems10.1007/978-3-030-31280-0_13(215-233)Online publication date: 8-Oct-2019
https://doi.org/10.1007/978-3-030-31280-0_13

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten