skip to main content
10.1145/2489280.2489286acmconferencesArticle/Chapter ViewAbstractPublication PagesisstaConference Proceedingsconference-collections
Article

Automatic, load-independent detection of performance regressions by transaction profiles

Published: 15 July 2013 Publication History

Abstract

Performance regression testing is an important step in the production process of enterprise applications. Yet, analysing this type of testing data is mainly conducted manually and depends on the load applied during the test. To ease such a manual task we present an automated, load-independent technique to detect performance regression anomalies based on the analysis of performance testing data using a concept known as Transaction Profile. The approach can be automated and it utilises data already available to the performance testing along with the queueing network model of the testing system.
The presented ``Transaction Profile Run Report'' was able to automatically catch performance regression anomalies ca-used by software changes and isolate them from those caused by load variations with a precision of 80% in a case study conducted against an open source application. Hence, by deploying our system, the testing teams are able to detect performance regression anomalies by avoiding the manual approach and eliminating the need to do extra runs with varying load.

References

[1]
J. Z. Ming. Automated analysis of load testing results. In Proceedings of the 19th international symposium on Software testing and analysis, New York, NY, USA, 2010. ACM.
[2]
Hewlett-Packard Development Company. Loadrunner. http://www8.hp.com/ie/en/softwaresolutions/software.html?compURI=1175451#.UQ-2QWfuo-A, 2013.
[3]
The Apache Software Foundation. Apache jmeter. http://jmeter.apache.org/, 2013.
[4]
R. Jain. The art of computer systems performance analysis, volume 182. John Wiley & Sons New York, 1991.
[5]
Z. M. Jiang A. Hassan M. Nasser T. H.D. Nguyen, B. Adams and P. Flora. Automated detection of performance regressions using statistical process control techniques. In Proceedings of the third joint WOSP/SIPEW international conference on Performance Engineering, New York, NY, USA, 2012.
[6]
T. Gao, Y. Ge, and J. Ni G. Wu. A reactivity-based framework of automated performance testing for web applications. In Proceedings of DCABES, pages 593–597. IEEE, 2010.
[7]
F. M. Bereznay. Did something change? using statistical techniques to interpret service and resource metrics. In 32nd International Computer Measurement Group Conference, December 3-6, 2006, Reno, Nevada, USA, Proceedings. Computer Measurement Group, 2006.
[8]
L. Grinshpan. Solving Enterprise Applications Performance Puzzles, pages 5–57. John Wiley and Sons, Inc., Hoboken, New Jersey, 2012.
[9]
S. Ghaith, M. Wang, P. Perry, and J. Murphy. Profile-based, load-independent anomaly detection and analysis in performance regression testing of software systems. In 17th European Conference on Software Maintenance and Reengineering (CSMR’13), Genova, Italy, 2013.
[10]
D.A. Menascé and V.A.F. Almeida. Capacity Planning for Web Services: metrics, models, and methods. Prentice Hall, 2002.
[11]
J. Walrand. An introduction to queueing networks, volume 165. Prentice Hall Englewood Cliffs, NJ, 1988.
[12]
G. Franks, T. Al-Omari, M. Woodside, O. Das, and S. Derisavi. Enhanced modeling and solution of layered queueing networks. Software Engineering, IEEE Transactions on, 35(2):148–161, 2009.
[13]
R.O. Baldwin, N.J. Davis Iv, S.F. Midkiff, and J.E. Kobza. Queueing network analysis: concepts, terminology, and methods. Journal of Systems and Software, 66(2):99–117, 2003.
[14]
F. Baskett, K.M. Chandy, R.R. Muntz, and F.G. Palacios. Open, closed, and mixed networks of queues with different classes of customers. Journal of the ACM, 22(2), pages 248–260, 1975.
[15]
G. Serazzi. Performance evaluation modelling with jmt: learning by examples. technical report 366, Politecnico di Milano - DEI, Milano, 2008.
[16]
D.A. Menasce, L.W. Dowdy, and V.A.F. Almeida. Performance by Design: Computer Capacity Planning By Example. Prentice Hall PTR, Upper Saddle River, NJ, USA, 2004.
[17]
E.D. Lazowska, J. Zahorjan, G.S. Graham, and K.C. Sevcik. Quantitative System Performance. Prentice-Hall, 1984.
[18]
M.Bertoli, G.Casale, and G.Serazzi. Java modelling tools: an open source suite for queueing network modelling and workload analysis. In Proceedings of QEST 2006 Conference. IEEE Press, Sep 2006.
[19]
MyBatis. Jpetstore 6. http://www.mybatis.org/spring/sample.html/, 2013.
[20]
T H. D. Nguyen. Using control charts for detecting and understanding performance regressions in large software. In Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, Washington, DC, USA, 2012. IEEE Computer Society.
[21]
G. Casale, P. Cremonesi, and R. Turrin. Robust workload estimation in queueing network performance models. In Proceedings of PDP, Washington, DC, USA, 2008. IEEE Computer Society.

Cited By

View all
  • (2015)Anomaly detection in performance regression testing by transaction profile estimationSoftware Testing, Verification and Reliability10.1002/stvr.157326:1(4-39)Online publication date: 9-Mar-2015
  • (2014)Software contention aware queueing network model of three-tier web systemsProceedings of the 5th ACM/SPEC international conference on Performance engineering10.1145/2568088.2576760(273-276)Online publication date: 22-Mar-2014
  • (2014)Transaction Profile Estimation of Queueing Network Models for IT Systems Using a Search-Based TechniqueSearch-Based Software Engineering10.1007/978-3-319-09940-8_18(234-239)Online publication date: 2014

Index Terms

  1. Automatic, load-independent detection of performance regressions by transaction profiles

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    JAMAICA 2013: Proceedings of the 2013 International Workshop on Joining AcadeMiA and Industry Contributions to testing Automation
    July 2013
    76 pages
    ISBN:9781450321617
    DOI:10.1145/2489280
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    In-Cooperation

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 15 July 2013

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Application change
    2. performance models
    3. regression testing

    Qualifiers

    • Article

    Conference

    ISSTA '13
    Sponsor:

    Upcoming Conference

    ISSTA '25

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)0
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 19 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2015)Anomaly detection in performance regression testing by transaction profile estimationSoftware Testing, Verification and Reliability10.1002/stvr.157326:1(4-39)Online publication date: 9-Mar-2015
    • (2014)Software contention aware queueing network model of three-tier web systemsProceedings of the 5th ACM/SPEC international conference on Performance engineering10.1145/2568088.2576760(273-276)Online publication date: 22-Mar-2014
    • (2014)Transaction Profile Estimation of Queueing Network Models for IT Systems Using a Search-Based TechniqueSearch-Based Software Engineering10.1007/978-3-319-09940-8_18(234-239)Online publication date: 2014

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media