skip to main content
10.1145/2851553.2851571acmconferencesArticle/Chapter ViewAbstractPublication PagesicpeConference Proceedingsconference-collections
research-article

Learning from Source Code History to Identify Performance Failures

Published: 12 March 2016 Publication History

Abstract

Source code changes may inadvertently introduce performance regressions. Benchmarking each software version is traditionally employed to identify performance regressions. Although effective, this exhaustive approach is hard to carry out in practice. This paper contrasts source code changes against performance variations. By analyzing 1,288 software versions from 17 open source projects, we identified 10 source code changes leading to a performance variation (improvement or regression). We have produced a cost model to infer whether a software commit introduces a performance variation by analyzing the source code and sampling the execution of a few versions. By profiling the execution of only 17% of the versions, our model is able to identify 83% of the performance regressions greater than 5% and 100% of the regressions greater than 50%.

References

[1]
Juan Pablo Sandoval Alcocer and Alexandre Bergel. Tracking down performance variation against source code evolution. In Proceedings of the 11th Symposium on Dynamic Languages, DLS 2015. ACM.
[2]
Len Bass, Ingo Weber, and Liming Zhu. DevOps: A Software Architect's Perspective. Addison-Wesley Professional, jun 2015.
[3]
L. A. Belady and M. M. Lehman. A model of large program development. IBM Syst. J., 15(3):225--252, September 1976.
[4]
Alexandre Bergel. Counting messages as a proxy for average execution time in pharo. In Proceedings of ECOOP'11.
[5]
C. Bezemer, E. Milon, A. Zaidman, and J. Pouwelse. Detecting and analyzing i/o performance regressions. Journal of Software: Evolution and Process, 26(12):1193--1212, 2014.
[6]
C. Bezemer, E. Milon, A. Zaidman, and J. Pouwelse. Detecting and analyzing i/o performance regressions. Journal of Software: Evolution and Process, 26(12):1193--1212, 2014.
[7]
Jennifer Black, Emanuel Melachrinoudis, and David Kaeli. Bi-criteria models for all-uses test suite reduction. In Proceedings of ICSE '04. IEEE.
[8]
Paul Duvall, Steve Matyas, and Andrew Glover. Continuous Integration: Improving Software Quality and Reducing Risk. Addison-Wesley Professional, first edition, 2007.
[9]
Sebastian Elbaum, Alexey G. Malishevsky, and Gregg Rothermel. Prioritizing test cases for regression testing. SIGSOFT Softw. Eng. Notes, 25(5):102--112, August 2000.
[10]
Shi Han, Yingnong Dang, Song Ge, Dongmei Zhang, and Tao Xie. Performance debugging in the large via mining millions of stack traces. In Proceedings of ICSE 2012.
[11]
Christoph Heger, Jens Happe, and Roozbeh Farahbod. Automated root cause isolation of performance regressions during software development. In Proceedings of ICPE '13.
[12]
Peng Huang, Xiao Ma, Dongcai Shen, and Yuanyuan Zhou. Performance regression testing target prioritization via performance risk analysis. In Proceedings of ICSE '14.
[13]
Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz, and Shan Lu. Understanding and detecting real-world performance bugs. SIGPLAN Not., 47(6):77--88, June 2012.
[14]
Jung-Min Kim and Adam Porter. A history-based test prioritization technique for regression testing in resource constrained environments. In Proceedings of ICSE '02.
[15]
M M. Lehman, J F. Ramil, P D. Wernick, D E. Perry, and W M. Turski. Metrics and laws of software evolution - the nineties view. In Proceedings of the 4th International Symposium on Software Metrics, METRICS '97, pages 20--32. IEEE Computer Society.
[16]
Zheng Li, M. Harman, and R.M. Hierons. Search algorithms for regression test case prioritization. IEEE Transactions on Software Engineering, 33(4):225--237, April 2007.
[17]
Nazim H. Madhavji, Juan Fernandez-Ramil, and Dewayne E. Perry. Software Evolution and Feedback: Theory and Practice. Wiley, Chichester, UK, 2006.
[18]
H. Malik, Zhen Ming Jiang, B. Adams, A.E. Hassan, P. Flora, and G. Hamann. Automatic comparison of load tests to support the performance analysis of large enterprise systems. In 14th European Conference on Software Maintenance and Reengineering (CSMR), 2010.
[19]
D. Maplesden, E. Tempero, J. Hosking, and J.C. Grundy. Performance analysis for object-oriented software: A systematic mapping. IEEE Transactions on Software Engineering, 41(7):691--710, July 2015.
[20]
Ian Molyneaux. The Art of Application Performance Testing: Help for Programmers and Quality Assurance. O'Reilly Media, Inc., 1st edition, 2009.
[21]
Thanh H. D. Nguyen, Meiyappan Nagappan, Ahmed E. Hassan, Mohamed Nasser, and Parminder Flora. An industrial case study of automatically identifying performance regression-causes. In Proceedings of MSR '14.
[22]
Thanh H.D. Nguyen, Bram Adams, Zhen Ming Jiang, Ahmed E. Hassan, Mohamed Nasser, and Parminder Flora. Automated detection of performance regressions using statistical process control techniques. In Proceedings of ICPE '12.
[23]
Adrian Nistor, Tian Jiang, and Lin Tan. Discovering, reporting, and fixing performance bugs. In Proceedings of MSR '13.
[24]
Adrian Nistor, Linhai Song, Darko Marinov, and Shan Lu. Toddler: Detecting performance problems via similar memory-access patterns. In Proceedings of ICSE '13.
[25]
G. Rothermel, R.H. Untch, Chengyun Chu, and M.J. Harrold. Test case prioritization: an empirical study. In Proceedings of ICSM '99.
[26]
Wei Shang, Ahmed E. Hassan, Mohamed Nasser, and Parminder Flora. Automated detection of performance regressions using regression models on clustered performance counters. In Proceedings of ICPE '15.
[27]
Du Shen, Qi Luo, Denys Poshyvanyk, and Mark Grechanik. Automating performance bottleneck detection using search-based application profiling. In Proceedings of ISSTA '15.
[28]
Reinhard Wilhelm, Jakob Engblom, Andreas Ermedahl, Niklas Holsti, Stephan Thesing, David Whalley, Guillem Bernat, Christian Ferdinand, Reinhold Heckmann, Tulika Mitra, Frank Mueller, Isabelle Puaut, Peter Puschner, Jan Staschulat, and Per Stenström. The worst-case execution-time problem; overview of methods and survey of tools. ACM Trans. Embed. Comput. Syst., 7(3):36:1--36:53, May 2008.
[29]
Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. Experimentation in Software Engineering. Kluwer Academic Publishers, 2000.
[30]
Shahed Zaman, Bram Adams, and Ahmed E. Hassan. A qualitative study on performance bugs. In Proceedings of MSR '12.
[31]
Hao Zhong, Lu Zhang, and Hong Mei. An experimental comparison of four test suite reduction techniques. In Proceedings of ICSE '06.

Cited By

View all
  • (2024)An Adaptive Logging System (ALS): Enhancing Software Logging with Reinforcement Learning TechniquesProceedings of the 15th ACM/SPEC International Conference on Performance Engineering10.1145/3629526.3645033(37-47)Online publication date: 7-May-2024
  • (2023)Adapting Performance Analytic Techniques in a Real-World Database-Centric System: An Industrial Experience ReportProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613893(1855-1866)Online publication date: 30-Nov-2023
  • (2022)On the Detection of Performance Regression Introducing Code Changes: Experience from the Git Project2022 IEEE 29th Annual Software Technology Conference (STC)10.1109/STC55697.2022.00036(206-217)Online publication date: Oct-2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICPE '16: Proceedings of the 7th ACM/SPEC on International Conference on Performance Engineering
March 2016
346 pages
ISBN:9781450340809
DOI:10.1145/2851553
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 March 2016

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. performance analysis
  2. performance evolution
  3. performance variation

Qualifiers

  • Research-article

Funding Sources

  • CONICYT (Comision Nacional de Investigacion Cientifica y Tecnologica)
  • STICAmSud
  • FONDECYT(Fondo Nacional de Desarrollo Cientifico y Tecnologico)

Conference

ICPE'16

Acceptance Rates

ICPE '16 Paper Acceptance Rate 23 of 74 submissions, 31%;
Overall Acceptance Rate 252 of 851 submissions, 30%

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)26
  • Downloads (Last 6 weeks)0
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)An Adaptive Logging System (ALS): Enhancing Software Logging with Reinforcement Learning TechniquesProceedings of the 15th ACM/SPEC International Conference on Performance Engineering10.1145/3629526.3645033(37-47)Online publication date: 7-May-2024
  • (2023)Adapting Performance Analytic Techniques in a Real-World Database-Centric System: An Industrial Experience ReportProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613893(1855-1866)Online publication date: 30-Nov-2023
  • (2022)On the Detection of Performance Regression Introducing Code Changes: Experience from the Git Project2022 IEEE 29th Annual Software Technology Conference (STC)10.1109/STC55697.2022.00036(206-217)Online publication date: Oct-2022
  • (2022)Automated Identification of Performance Changes at Code Level2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS57517.2022.00096(916-925)Online publication date: Dec-2022
  • (2022)To test, or not to test: A proactive approach for deciding complete performance test initiation2022 IEEE International Conference on Big Data (Big Data)10.1109/BigData55660.2022.10020543(4758-4767)Online publication date: 17-Dec-2022
  • (2022)On the Effectiveness of Bisection in Performance Regression LocalizationEmpirical Software Engineering10.1007/s10664-022-10152-327:4Online publication date: 30-Apr-2022
  • (2022)TEP-GNN: Accurate Execution Time Prediction of Functional Tests Using Graph Neural NetworksProduct-Focused Software Process Improvement10.1007/978-3-031-21388-5_32(464-479)Online publication date: 14-Nov-2022
  • (2021)Using application benchmark call graphs to quantify and improve the practical relevance of microbenchmark suitesPeerJ Computer Science10.7717/peerj-cs.5487(e548)Online publication date: 28-May-2021
  • (2021)A classification of code changes and test types dependencies for improving machine learning based test selectionProceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering10.1145/3475960.3475987(40-49)Online publication date: 19-Aug-2021
  • (2021)Locating Performance Regression Root Causes in the Field Operations of Web-based Systems: An Experience ReportIEEE Transactions on Software Engineering10.1109/TSE.2021.3131529(1-1)Online publication date: 2021
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media