Shadi Ghaith
ABSTRACT
Performance regression testing is an important step in the production process of enterprise applications. Yet, analysing this type of testing data is mainly conducted manually and depends on the load applied during the test. To ease such a manual task we present an automated, load-independent technique to detect performance regression anomalies based on the analysis of performance testing data using a concept known as Transaction Profile. The approach can be automated and it utilises data already available to the performance testing along with the queueing network model of the testing system.
The presented ``Transaction Profile Run Report'' was able to automatically catch performance regression anomalies ca-used by software changes and isolate them from those caused by load variations with a precision of 80% in a case study conducted against an open source application. Hence, by deploying our system, the testing teams are able to detect performance regression anomalies by avoiding the manual approach and eliminating the need to do extra runs with varying load.
- J. Z. Ming. Automated analysis of load testing results. In Proceedings of the 19th international symposium on Software testing and analysis, New York, NY, USA, 2010. ACM. Google Scholar
Digital Library
- Hewlett-Packard Development Company. Loadrunner. http://www8.hp.com/ie/en/softwaresolutions/software.html?compURI=1175451#.UQ-2QWfuo-A, 2013.Google Scholar
- The Apache Software Foundation. Apache jmeter. http://jmeter.apache.org/, 2013.Google Scholar
- R. Jain. The art of computer systems performance analysis, volume 182. John Wiley & Sons New York, 1991.Google Scholar
- Z. M. Jiang A. Hassan M. Nasser T. H.D. Nguyen, B. Adams and P. Flora. Automated detection of performance regressions using statistical process control techniques. In Proceedings of the third joint WOSP/SIPEW international conference on Performance Engineering, New York, NY, USA, 2012. Google ScholarDigital Library
- T. Gao, Y. Ge, and J. Ni G. Wu. A reactivity-based framework of automated performance testing for web applications. In Proceedings of DCABES, pages 593–597. IEEE, 2010. Google ScholarDigital Library
- F. M. Bereznay. Did something change? using statistical techniques to interpret service and resource metrics. In 32nd International Computer Measurement Group Conference, December 3-6, 2006, Reno, Nevada, USA, Proceedings. Computer Measurement Group, 2006.Google Scholar
- L. Grinshpan. Solving Enterprise Applications Performance Puzzles, pages 5–57. John Wiley and Sons, Inc., Hoboken, New Jersey, 2012. Google ScholarDigital Library
- S. Ghaith, M. Wang, P. Perry, and J. Murphy. Profile-based, load-independent anomaly detection and analysis in performance regression testing of software systems. In 17th European Conference on Software Maintenance and Reengineering (CSMR’13), Genova, Italy, 2013. Google ScholarDigital Library
- D.A. Menascé and V.A.F. Almeida. Capacity Planning for Web Services: metrics, models, and methods. Prentice Hall, 2002. Google ScholarDigital Library
- J. Walrand. An introduction to queueing networks, volume 165. Prentice Hall Englewood Cliffs, NJ, 1988.Google Scholar
- G. Franks, T. Al-Omari, M. Woodside, O. Das, and S. Derisavi. Enhanced modeling and solution of layered queueing networks. Software Engineering, IEEE Transactions on, 35(2):148–161, 2009. Google ScholarDigital Library
- R.O. Baldwin, N.J. Davis Iv, S.F. Midkiff, and J.E. Kobza. Queueing network analysis: concepts, terminology, and methods. Journal of Systems and Software, 66(2):99–117, 2003. Google ScholarDigital Library
- F. Baskett, K.M. Chandy, R.R. Muntz, and F.G. Palacios. Open, closed, and mixed networks of queues with different classes of customers. Journal of the ACM, 22(2), pages 248–260, 1975. Google ScholarDigital Library
- G. Serazzi. Performance evaluation modelling with jmt: learning by examples. technical report 366, Politecnico di Milano - DEI, Milano, 2008.Google Scholar
- D.A. Menasce, L.W. Dowdy, and V.A.F. Almeida. Performance by Design: Computer Capacity Planning By Example. Prentice Hall PTR, Upper Saddle River, NJ, USA, 2004. Google ScholarDigital Library
- E.D. Lazowska, J. Zahorjan, G.S. Graham, and K.C. Sevcik. Quantitative System Performance. Prentice-Hall, 1984. Google ScholarDigital Library
- M.Bertoli, G.Casale, and G.Serazzi. Java modelling tools: an open source suite for queueing network modelling and workload analysis. In Proceedings of QEST 2006 Conference. IEEE Press, Sep 2006. Google ScholarDigital Library
- MyBatis. Jpetstore 6. http://www.mybatis.org/spring/sample.html/, 2013.Google Scholar
- T H. D. Nguyen. Using control charts for detecting and understanding performance regressions in large software. In Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, Washington, DC, USA, 2012. IEEE Computer Society. Google ScholarDigital Library
- G. Casale, P. Cremonesi, and R. Turrin. Robust workload estimation in queueing network performance models. In Proceedings of PDP, Washington, DC, USA, 2008. IEEE Computer Society. Google ScholarDigital Library
Index Terms
- Automatic, load-independent detection of performance regressions by transaction profiles
Recommendations
Profile-Based, Load-Independent Anomaly Detection and Analysis in Performance Regression Testing of Software Systems
CSMR '13: Proceedings of the 2013 17th European Conference on Software Maintenance and ReengineeringPerformance evaluation through regression testing is an important step in the software production process. It aims to make sure that the performance of new releases do not regress under a field-like load. The main outputs of regression tests are the ...
Analysis of performance regression testing data by transaction profiles
ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and AnalysisPerformance regression testing is an important step in the software development lifecycle, especially for enterprise applications. Commonly the analysis of performance regression testing to find anomalies is carried out manually and therefore can be ...
Anomaly detection in performance regression testing by transaction profile estimation
As part of the process to test a new release of an application, the performance testing team need to confirm that the existing functionalities do not perform worse than those in the previous release, a problem known as performance regression anomaly. ...
Comments