ABSTRACT
Regression test selection speeds up regression testing by re-running only the tests that can be affected by the most recent code changes. Much progress has been made on research in automated test selection over the last three decades, but it has not translated into practical tools that are widely adopted. Therefore, developers either re-run all tests after each change or perform manual test selection. Re-running all tests is expensive, while manual test selection is tedious and error-prone. Despite such a big trade-off, no study assessed how developers perform manual test selection and compared it to automated test selection.
This paper reports on our study of manual test selection in practice and our comparison of manual and automated test selection. We are the first to conduct a study that (1) analyzes data from manual test selection, collected in real time from 14 developers during a three-month study and (2) compares manual test selection with an automated state-of-the-research test-selection tool for 450 test sessions.
Almost all developers in our study performed manual test selection, and they did so in mostly ad-hoc ways. Comparing manual and automated test selection, we found the two approaches to select different tests in each and every one of the 450 test sessions investigated. Manual selection chose more tests than automated selection 73% of the time (potentially wasting time) and chose fewer tests 27% of the time (potentially missing bugs). These results show the need for better automated test-selection techniques that integrate well with developers' programming environments.
- T. Ball. On the limit of control flow analysis for regression test selection. In ISSTA, 1998. Google ScholarDigital Library
- J. Bible, G. Rothermel, and D. S. Rosenblum. A comparative study of coarse- and fine-grained safe regression test-selection techniques. TOSEM, 2001. Google ScholarDigital Library
- L. Briand, Y. Labiche, and S. He. Automating regression test selection based on UML designs. IST, 2009. Google ScholarDigital Library
- CodingTracker. http://codingtracker.web.engr. illinois.edu/.Google Scholar
- R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test data selection: Help for the practicing programmer. Computer, 1978. Google ScholarDigital Library
- S. G. Elbaum, G. Rothermel, S. Kanduri, and A. G. Malishevsky. Selecting a cost-effective test case prioritization technique. SQJ, 2004. Google ScholarDigital Library
- E. Engström and P. Runeson. A qualitative survey of regression testing practices. In PROFES , 2010.Google ScholarDigital Library
- K. Fischer, F. Raji, and A. Chruscicki. A methodology for retesting modified software. In NTC, 1981.Google Scholar
- Use a gated check-in build process to validate changes. http://msdn.microsoft.com/en-us/library/dd787631.aspx.Google Scholar
- M. Gligoric, R. Majumdar, R. Sharma, L. Eloussi, and D. Marinov. Regression test selection for distributed software histories. In CAV, 2014.Google Scholar
- J. Goodenough and S. Gerhart. Toward a theory of test data selection. TOSEM, 1975.Google ScholarDigital Library
- T. L. Graves, M. J. Harrold, J.-M. Kim, A. Porter, and G. Rothermel. An empirical study of regression test selection techniques. In ICSE, 1998. Google ScholarDigital Library
- M. Greiler, A. van Deursen, and M. Storey. Test confessions: A study of testing practices for plug-in systems. In ICSE, 2012. Google ScholarDigital Library
- M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pennings, S. Sinha, S. A. Spoon, and A. Gujarathi. Regression test selection for Java software. In OOPSLA, 2001. Google ScholarDigital Library
- M. J. Harrold and M. L. Soffa. An incremental approach to unit testing during maintenance. In ICSM, 1988.Google ScholarCross Ref
- J. Hartmann. Applying selective revalidation techniques at Microsoft. In PNSQC, 2007.Google Scholar
- J. Hartmann. 30 years of regression testing: Past, present and future. In PNSQC, 2012.Google Scholar
- J. L. Hintze and R. D. Nelson. Violin plots: A box plot-density trace synergism. The American Statistician, 1998.Google Scholar
- Java, Java everywhere, 2012. http://sdtimes.com/content/article.aspx?ArticleID=36362.Google Scholar
- Q. Luo, F. Hariri, L. Eloussi, and D. Marinov. An empirical analysis of flaky tests. In FSE, 2014. to appear.Google ScholarDigital Library
- S. Negara, N. Chen, M. Vakilian, R. E. Johnson, and D. Dig. A comparative study of manual and automated refactorings. In ECOOP, 2013. Google ScholarDigital Library
- S. Negara, M. Codoban, D. Dig, and R. E. Johnson. Mining fine-grained code changes to detect unknown change patterns. In ICSE, 2014. Google ScholarDigital Library
- S. Negara, M. Vakilian, N. Chen, R. E. Johnson, and D. Dig. Is it dangerous to use version control histories to study source code evolution? In ECOOP, 2012. Google ScholarDigital Library
- A. K. Onoma, W.-T. Tsai, M. Poonawala, and H. Suganuma. Regression testing in an industrial environment. Communications, 1998. Google ScholarDigital Library
- A. Orso, T. Apiwattanapong, and M. J. Harrold. Leveraging field data for impact analysis and regression testing. In FSE, 2003. Google ScholarDigital Library
- D. Parsons, T. Susnjak, and M. Lange. Influences on regression testing strategies in agile software development environments. SQJ, 2013.Google Scholar
- X. Qu, M. B. Cohen, and G. Rothermel. Configuration-aware regression testing: An empirical study of sampling and prioritization. In ISSTA, 2008. Google ScholarDigital Library
- X. Ren, F. Shah, F. Tip, B. G. Ryder, and O. Chesley. Chianti: A tool for change impact analysis of Java programs. In OOPSLA, 2004. Google ScholarDigital Library
- G. Rothermel and M. Harrold. Selecting regression tests for object-oriented software. In ICSM, 1994. Google ScholarDigital Library
- G. Rothermel and M. Harrold. Empirical studies of a safe regression test selection technique. TOSEM, 1998.Google ScholarDigital Library
- G. Rothermel and M. J. Harrold. A safe, efficient regression test selection technique. TOSEM, 1997. Google ScholarDigital Library
- G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Test case prioritization: An empirical study. In ICSM, 1999. Google ScholarDigital Library
- D. Saff and M. D. Ernst. An experimental evaluation of continuous testing during development. In ISSTA, 2004. Google ScholarDigital Library
- A. Srivastava and J. Thiagarajan. Effectively prioritizing tests in development environment. In ISSTA, 2002. Google ScholarDigital Library
- D. Talby, A. Keren, O. Hazzan, and Y. Dubinsky. Agile software testing in a large-scale project. Software, 2006. Google ScholarDigital Library
- Streamline testing process with test impact analysis, 2013. http://msdn.microsoft.com/en-us/library/ff576128%28v=vs.100%29.aspx.Google Scholar
- Testing at the speed and scale of Google, 2011. http://goo.gl/OKqBk.Google Scholar
- Tools for continuous integration at Google scale, 2011. http://www.youtube.com/watch?v=b52aXZ2yi08.Google Scholar
- M. Vakilian, N. Chen, S. Negara, B. A. Rajkumar, B. P. Bailey, and R. E. Johnson. Use, disuse, and misuse of automated refactorings. In ICSE, 2012. Google ScholarDigital Library
- D. Willmor and S. M. Embury. A safe regression test selection technique for database-driven applications. In ICSM, 2005. Google ScholarDigital Library
- W. E. Wong, J. R. Horgan, S. London, and H. Agrawal. A study of effective regression testing in practice. In ISSRE, 1997. Google ScholarDigital Library
- S. Yau and Z. Kishimoto. A method for revalidating modified programs in the maintenance phase. In COMPSAC, 1987.Google Scholar
- S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: A survey. STVR, 2012. Google ScholarDigital Library
- L. Zhang, M. Kim, and S. Khurshid. Localizing failure-inducing program edits based on spectrum information. In ICSM, 2011. Google ScholarDigital Library
Index Terms
- An empirical evaluation and comparison of manual and automated test selection
Recommendations
Empirical Studies of a Safe Regression Test Selection Technique
Regression testing is an expensive testing procedure utilized to validate modified software. Regression test selection techniques attempt to reduce the cost of regression testing by selecting a subset of a program's existing test suite. Safe regression ...
Automatic test selection based on CEFSM specifications
Mutation analysis is a fault based testing method used initially for code based software testing. In this paper, this method is applied to formal specifications and used for automatic conformance test selection. This paper defines formally a set of ...
Towards Tool-Support for Test Case Selection in Manual Regression Testing
ICSTW '13: Proceedings of the 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation WorkshopsManual regression testing can be a time-intensive and costly activity. Required efforts can be reduced by selecting only the tests for re-testing that verify actually modified system parts. However, if testers are not familiar with the system ...
Comments