Du Shen
Denys Poshyvanyk
ABSTRACT
Application profiling is an important performance analysis technique, when an application under test is analyzed dynamically to determine its space and time complexities and the usage of its instructions. A big and important challenge is to profile nontrivial web applications with large numbers of combinations of their input parameter values. Identifying and understanding particular subsets of inputs leading to performance bottlenecks is mostly manual, intellectually intensive and laborious procedure. We propose a novel approach for automating performance bottleneck detection using search-based input-sensitive application profiling. Our key idea is to use a genetic algorithm as a search heuristic for obtaining combinations of input parameter values that maximizes a fitness function that represents the elapsed execution time of the application. We implemented our approach, coined as Genetic Algorithm-driven Profiler (GA-Prof) that combines a search-based heuristic with contrast data mining of execution traces to accurately determine performance bottlenecks. We evaluated GA-Prof to determine how effectively and efficiently it can detect injected performance bottlenecks into three popular open source web applications. Our results demonstrate that GA-Prof efficiently explores a large space of input value combinations while automatically and accurately detecting performance bottlenecks, thus suggesting that it is effective for automatic profiling.
- W. Afzal and R. Torkar. On the application of genetic programming for software engineering predictive modeling: A systematic review. Expert Syst. Applications, 38(9):11984–11997, 2011. Google Scholar
Digital Library
- M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In SOSP ’03, pages 74–89, 2003. Google ScholarDigital Library
- S. Ali, L. C. Briand, H. Hemmati, and R. K. Panesar-Walawege. A systematic review of the application and empirical investigation of search-based test case generation. TSE, 36(6):742–762, 2010. Google ScholarDigital Library
- N. Alshahwan and M. Harman. Automated web application testing using search based software engineering. In ASE ’11, pages 3–12, 2011. Google ScholarDigital Library
- G. Ammons, J.-D. Choi, M. Gupta, and N. Swamy. Finding and removing performance bottlenecks in large systems. In ECOOP ’04, pages 172–196. 2004.Google Scholar
- A. Arcuri and L. Briand. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In ICSE ’11, pages 1–10, 2011. Google ScholarDigital Library
- A. Arcuri and L. Briand. A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. STVR, 2012.Google Scholar
- S. Artzi, J. Dolby, F. Tip, and M. Pistoia. Directed test generation for effective fault localization. In ISSTA ’10, pages 49–60, 2010. Google ScholarDigital Library
- S. Artzi, J. Dolby, F. Tip, and M. Pistoia. Practical fault localization for dynamic web applications. In ICSE ’10, pages 49–60, 2010. Google ScholarDigital Library
- A. Baars, M. Harman, Y. Hassoun, K. Lakhotia, P. McMinn, P. Tonella, and T. Vos. Symbolic search-based testing. In ASE ’11, pages 53–62, 2011. Google ScholarDigital Library
- J. Bach. What is exploratory testing? stickyminds.com.Google Scholar
- L. C. Briand, Y. Labiche, and M. Shousha. Stress testing real-time systems with genetic algorithms. In GECCO ’05, pages 1021–1028, 2005. Google ScholarDigital Library
- J. Burnim, S. Juvekar, and K. Sen. Wise: Automated test generation for worst-case complexity. In ICSE ’09, pages 463–473, 2009. Google ScholarDigital Library
- Y. Cai, J. Grundy, and J. Hosking. Synthesizing client load models for performance engineering via web crawling. In ASE ’07, pages 353–362, 2007. Google ScholarDigital Library
- H. W. Cain, B. P. Miller, and B. J. Wylie. A callgraph-based search strategy for automated performance diagnosis. In Euro-Par ’00, pages 108–122, 2000. Google ScholarDigital Library
- N. Chapin, J. E. Hale, K. M. Kham, J. F. Ramil, and W.-G. Tan. Types of software evolution and software maintenance. J. of Softw. Maint. and Evo. R. P., 13(1):3–30, 2001. Google ScholarDigital Library
- T.-H. Chen, W. Shang, Z. M. Jiang, A. E. Hassan, M. Nasser, and P. Flora. Detecting performance anti-patterns for applications developed using object-relational mapping. In ICSE ’14, pages 1001–1012, 2014. Google ScholarDigital Library
- X. Chen, C. P. Ho, R. Osman, P. G. Harrison, and W. J. Knottenbelt. Understanding, modelling, and improving the performance of web applications in multicore virtualised environments. In ICPE ’14, pages 197–207, 2014. Google ScholarDigital Library
- T. M. Chilimbi, B. Liblit, K. Mehra, A. V. Nori, and K. Vaswani. Holmes: Effective statistical debugging via efficient path profiling. In ICSE ’09, pages 34–44, 2009. Google ScholarDigital Library
- E. Coppa, C. Demetrescu, and I. Finocchi. Input-sensitive profiling. In PLDI ’12, pages 89–98, 2012. Google ScholarDigital Library
- D. C. D’Elia, C. Demetrescu, and I. Finocchi. Mining hot calling contexts in small space. In PLDI ’11. Google ScholarDigital Library
- G. Dong and J. Bailey. Contrast Data Mining: Concepts, Algorithms, and Applications. 1st edition, 2012. Google ScholarDigital Library
- S. Elbaum and M. Hardojo. An empirical study of profiling strategies for released software and their impact on testing activities. In ISSTA ’04, pages 65–75, 2004. Google ScholarDigital Library
- I. F. Emilio Coppa, Camil Demetrescu. Input-sensitive profiling. TSE, 40(12):1185–1205, 2014.Google Scholar
- D. R. Faught. Exploratory load testing. stickyminds.com.Google Scholar
- S. Forrest, T. Nguyen, W. Weimer, and C. Le Goues. A genetic programming approach to automated software repair. In GECCO ’09, pages 947–954, 2009. Google ScholarDigital Library
- G. Fraser, A. Arcuri, and P. McMinn. A memetic algorithm for whole test suite generation. JSS, 2014.Google Scholar
- S. Frey, F. Fittkau, and W. Hasselbring. Search-based genetic optimization for deployment and reconfiguration of software in the cloud. In ICSE ’13, pages 512–521, 2013. Google ScholarDigital Library
- P. Godefroid and S. Khurshid. Exploring very large state spaces using genetic algorithms. STTT, 6(2):117–127, 2004. Google ScholarDigital Library
- C. L. Goues, T. Nguyen, S. Forrest, and W. Weimer. Genprog: A generic method for automatic software repair. TSE, 38:54–72, 2012. Google ScholarDigital Library
- S. Grant, J. R. Cordy, and D. Skillicorn. Automated concept location using independent component analysis. In WCRE’08, pages 138–142, 2008. Google ScholarDigital Library
- M. Grechanik, C. Fu, and Q. Xie. Automatically finding performance problems with feedback-directed learning software testing. In ICSE ’12, pages 156–166, 2012. Google ScholarDigital Library
- B. Gregg. Systems Performance: Enterprise and the Cloud. Prentice Hall Press, 2013. Google ScholarDigital Library
- Y. Group. Enterprise application management survey. 2005.Google Scholar
- S. Han, Y. Dang, S. Ge, D. Zhang, and T. Xie. Performance debugging in the large via mining millions of stack traces. In ICSE ’12, pages 145–155, 2012. Google ScholarDigital Library
- M. Harman. Search based software engineering for program comprehension. In ICPC ’07, pages 3–13, 2007. Google ScholarDigital Library
- M. Harman, Y. Jia, and W. B. Langdon. Strong higher order mutation-based test data generation. In FSE ’11, pages 212–222, 2011. Google ScholarDigital Library
- M. Harman, K. Lakhotia, J. Singer, D. R. White, and S. Yoo. Cloud engineering is search based software engineering too. JSS, 86(9):2225–2241, 2013. Google ScholarDigital Library
- M. Harman, S. A. Mansouri, and Y. Zhang. Search-based software engineering: Trends, techniques and applications. CSUR, 45(1):11:1–11:61, Dec. 2012. Google ScholarDigital Library
- M. Harman and P. McMinn. A theoretical and empirical study of search-based testing: Local, global, and hybrid search. TSE, 36(2):226–247, Mar. 2010. Google ScholarDigital Library
- J. H. Holland. Adaptation in natural and artificial systems: An introductory analysis with applications to biology, control, and artificial intelligence. U Michigan Press, 1975. Google ScholarDigital Library
- A. Hyvärinen and E. Oja. Independent component analysis: algorithms and applications. Neural networks, 13(4):411–430, 2000.Google ScholarDigital Library
- M. Z. Iqbal, A. Arcuri, and L. Briand. Empirical investigation of search algorithms for environment model-based testing of real-time embedded software. In ISSTA ’12, pages 199–209. ACM, 2012. Google ScholarDigital Library
- L. Jiang and Z. Su. Profile-guided program simplification for effective testing and analysis. In FSE ’08, pages 48–58, 2008. Google ScholarDigital Library
- Z. M. Jiang, A. E. Hassan, G. Hamann, and P. Flora. Automatic identification of load testing problems. In ICSM ’08, pages 307–316, 2008.Google ScholarCross Ref
- Z. M. Jiang, A. E. Hassan, G. Hamann, and P. Flora. Automated performance analysis of load tests. In ICSM ’09, pages 125–134, 2009.Google ScholarCross Ref
- G. Jin, L. Song, X. Shi, J. Scherpelz, and S. Lu. Understanding and detecting real-world performance bugs. PIDI ’12, pages 77–88, 2012. Google ScholarDigital Library
- B. F. Jones, H.-H. Sthamer, and D. E. Eyres. Automatic structural testing using genetic algorithms. Softw. Eng. J., 11(5):299–306, 1996.Google ScholarCross Ref
- P. Joshi, C.-S. Park, K. Sen, and M. Naik. A randomized dynamic program analysis technique for detecting real deadlocks. SIGPLAN Not., 44(6):110–120, 2009. Google ScholarDigital Library
- B. Korel. Automated software test data generation. TSE, 16(8):870–879, 1990. Google ScholarDigital Library
- T. Küstner, J. Weidendorfer, and T. Weinzierl. Argument controlled profiling. In Euro-Par ’09, pages 177–184. Google ScholarDigital Library
- W. B. Langdon, M. Harman, and Y. Jia. Multi objective higher order mutation testing with genetic programming. In TAIC PART ’09, pages 21–29, 2009. Google ScholarDigital Library
- C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. In ICSE ’12, pages 3–13, 2012. Google ScholarDigital Library
- Z. Li, M. Harman, and R. M. Hierons. Search algorithms for regression test case prioritization. TSE, 33(4):225–237, 2007. Google ScholarDigital Library
- B. P. Lientz and E. B. Swanson. Software Maintenance Management. Addison-Wesley, 1980. Google ScholarDigital Library
- X. Liu, J. Zhan, K. Zhan, W. Shi, L. Yuan, D. Meng, and L. Wang. Automatic performance debugging of spmd-style parallel programs. JPDC, 71(7):925–937, 2011. Google ScholarDigital Library
- P. McMinn. Search-based software test data generation: A survey: Research articles. STVR, 14(2):105–156, June 2004. Google ScholarDigital Library
- P. McMinn. Search-based software testing: Past, present and future. In ICSTW ’11, pages 153–163, 2011. Google ScholarDigital Library
- P. McMinn, M. Harman, K. Lakhotia, Y. Hassoun, and J. Wegener. Input domain reduction through irrelevant variable removal and its effect on local, global, and hybrid search-based structural test data generation. TSE, 38(2):453–477, 2012. Google ScholarDigital Library
- A. Memon, A. Porter, C. Yilmaz, A. Nagarajan, D. Schmidt, and B. Natarajan. Skoll: Distributed continuous quality assurance. In ICSE ’04, pages 459–468, 2004. Google ScholarDigital Library
- B. S. Mitchell and S. Mancoridis. Using heuristic search techniques to extract design abstractions from source code. In GECCO ’02, pages 1375–1382, 2002. Google ScholarDigital Library
- T. Mytkowicz, A. Diwan, M. Hauswirth, and P. F. Sweeney. Evaluating the accuracy of java profilers. In PLDI ’10, pages 187–197, 2010. Google ScholarDigital Library
- T. H. D. Nguyen, M. Nagappan, A. E. Hassan, M. Nasser, and F. Parminder. An industrial case study of automatically identifying performance regression-causes. In MSR ’14. Google ScholarDigital Library
- M. O’Keeffe and M. Ó. Cinnéide. Search-based software maintenance. In CSMR ’06, pages 249–260. IEEE, 2006. Google ScholarDigital Library
- M. Pradel and T. R. Gross. Leveraging test generation and specification mining for automated bug detection without false positives. In ICSE ’12, pages 288–298, 2012. Google ScholarDigital Library
- M. Pradel, P. Schuh, G. Necula, and K. Sen. Eventbreak: Analyzing the responsiveness of user interfaces through performance-guided test generation. In OOPSLA ’14, pages 33–47, 2014. Google ScholarDigital Library
- M. Schur, A. Roth, and A. Zeller. Mining behavior models from enterprise web applications. In FSE ’13, pages 422–432, 2013. Google ScholarDigital Library
- M. Schur, A. Roth, and A. Zeller. Procrawl: Mining test models from multi-user web applications. In ISSTA ’14, pages 413–416, 2014. Google ScholarDigital Library
- C. Schwaber, C. Mines, and L. Hogan. Performance-driven software development: How it shops can more efficiently meet performance requirements. Forrester Research, 2006.Google Scholar
- B. Schwarz, D. Schuler, and A. Zeller. Breeding high-impact mutations. In ICSTW ’11, pages 382–387. IEEE, 2011. Google ScholarDigital Library
- A. Shankar, M. Arnold, and R. Bodik. Jolt: Lightweight dynamic analysis and removal of object churn. In OOPSLA ’08, pages 127–142, 2008. Google ScholarDigital Library
- A. Srivastava and A. Eustace. Atom: A system for building customized program analysis tools. In PLDI ’94, pages 196–205. ACM, 1994. Google ScholarDigital Library
- V. Vangala, J. Czerwonka, and P. Talluri. Test case comparison and clustering using program profiles and static execution. In FSE ’09, pages 293–294, 2009. Google ScholarDigital Library
- A. L. Watkins. The automatic generation of test data using genetic algorithms. In SQC, pages 300–309, 1995.Google Scholar
- J. Wegener, K. Grimm, M. Grochtmann, H. Sthamer, and B. Jones. Systematic testing of real-time systems. In EuroSTAR ’96, 1996.Google Scholar
- J. Wegener and M. Grochtmann. Verifying timing constraints of real-time systems by means of evolutionary testing. Real-Time Systems, 15(3):275–298, 1998. Google ScholarDigital Library
- J. Wegener, H. Sthamer, B. F. Jones, and D. E. Eyres. Testing real-time systems using genetic algorithms. Software Quality Journal, 6(2):127–135, 1997. Google ScholarDigital Library
- W. Weimer, T. Nguyen, C. Le Goues, and S. Forrest. Automatically finding patches using genetic programming. In ICSE ’09, pages 364–374, 2009. Google ScholarDigital Library
- X. Xiao, S. Han, D. Zhang, and T. Xie. Context-sensitive delta inference for identifying workload-dependent performance bottlenecks. In ISSTA ’13, pages 90–100, 2013. Google ScholarDigital Library
- G. Xu and A. Rountev. Precise memory leak detection for java software using container profiling. In ICSE ’08, pages 151–160, 2008. Google ScholarDigital Library
- Z. Xu, M. B. Cohen, and G. Rothermel. Factors affecting the use of genetic algorithms in test suite augmentation. In GECCO ’10, pages 1365–1372, 2010. Google ScholarDigital Library
- D. Yan, G. Xu, and A. Rountev. Uncovering performance problems in java applications with reference propagation profiling. In ICSE ’12, pages 134–144, 2012. Google ScholarDigital Library
- N. Yuhanna. Dbms selection: Look beyond basic functions. Forrester Research, 2009.Google Scholar
- S. Zaman, B. Adams, and A. E. Hassan. A qualitative study on performance bugs. In MSR ’12, pages 199–208, 2012. Google ScholarDigital Library
- D. Zaparanuks and M. Hauswirth. Algorithmic profiling. In PLDI ’12, pages 67–76, 2012. Google ScholarDigital Library
- P. Zhang, S. Elbaum, and M. B. Dwyer. Automatic generation of load tests. In ASE ’11, pages 43–52, 2011. Google ScholarDigital Library
- S. Zhang and M. D. Ernst. Automated diagnosis of software configuration errors. In ICSE ’13, pages 312–321, 2013. Google ScholarDigital Library
Index Terms
- Automating performance bottleneck detection using search-based application profiling
Recommendations
Synthesizing programs that expose performance bottlenecks
CGO 2018: Proceedings of the 2018 International Symposium on Code Generation and OptimizationSoftware often suffers from performance bottlenecks, e.g., because some code has a higher computational complexity than expected or because a code change introduces a performance regression. Finding such bottlenecks is challenging for developers and for ...
Performance Anomaly Detection and Bottleneck Identification
In order to meet stringent performance requirements, system administrators must effectively detect undesirable performance behaviours, identify potential root causes, and take adequate corrective measures. The problem of uncovering and understanding ...
Adaptive Profiling for Root-Cause Analysis of Performance Anomalies in Web-Based Applications
NCA '11: Proceedings of the 2011 IEEE 10th International Symposium on Network Computing and ApplicationsThe most important factor in the assessment of the availability of a system is the mean-time to repair (MTTR). The lower the MTTR the higher the availability. A significant portion of the MTTR is spent in the detection and localization of the cause of ...
Comments