skip to main content
10.1145/2642937.2643019acmconferencesArticle/Chapter ViewAbstractPublication PagesaseConference Proceedingsconference-collections
research-article

An empirical evaluation and comparison of manual and automated test selection

Authors Info & Claims
Published:15 September 2014Publication History

ABSTRACT

Regression test selection speeds up regression testing by re-running only the tests that can be affected by the most recent code changes. Much progress has been made on research in automated test selection over the last three decades, but it has not translated into practical tools that are widely adopted. Therefore, developers either re-run all tests after each change or perform manual test selection. Re-running all tests is expensive, while manual test selection is tedious and error-prone. Despite such a big trade-off, no study assessed how developers perform manual test selection and compared it to automated test selection.

This paper reports on our study of manual test selection in practice and our comparison of manual and automated test selection. We are the first to conduct a study that (1) analyzes data from manual test selection, collected in real time from 14 developers during a three-month study and (2) compares manual test selection with an automated state-of-the-research test-selection tool for 450 test sessions.

Almost all developers in our study performed manual test selection, and they did so in mostly ad-hoc ways. Comparing manual and automated test selection, we found the two approaches to select different tests in each and every one of the 450 test sessions investigated. Manual selection chose more tests than automated selection 73% of the time (potentially wasting time) and chose fewer tests 27% of the time (potentially missing bugs). These results show the need for better automated test-selection techniques that integrate well with developers' programming environments.

References

  1. T. Ball. On the limit of control flow analysis for regression test selection. In ISSTA, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. J. Bible, G. Rothermel, and D. S. Rosenblum. A comparative study of coarse- and fine-grained safe regression test-selection techniques. TOSEM, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. L. Briand, Y. Labiche, and S. He. Automating regression test selection based on UML designs. IST, 2009. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. CodingTracker. http://codingtracker.web.engr. illinois.edu/.Google ScholarGoogle Scholar
  5. R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test data selection: Help for the practicing programmer. Computer, 1978. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. S. G. Elbaum, G. Rothermel, S. Kanduri, and A. G. Malishevsky. Selecting a cost-effective test case prioritization technique. SQJ, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. E. Engström and P. Runeson. A qualitative survey of regression testing practices. In PROFES , 2010.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. K. Fischer, F. Raji, and A. Chruscicki. A methodology for retesting modified software. In NTC, 1981.Google ScholarGoogle Scholar
  9. Use a gated check-in build process to validate changes. http://msdn.microsoft.com/en-us/library/dd787631.aspx.Google ScholarGoogle Scholar
  10. M. Gligoric, R. Majumdar, R. Sharma, L. Eloussi, and D. Marinov. Regression test selection for distributed software histories. In CAV, 2014.Google ScholarGoogle Scholar
  11. J. Goodenough and S. Gerhart. Toward a theory of test data selection. TOSEM, 1975.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. T. L. Graves, M. J. Harrold, J.-M. Kim, A. Porter, and G. Rothermel. An empirical study of regression test selection techniques. In ICSE, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. M. Greiler, A. van Deursen, and M. Storey. Test confessions: A study of testing practices for plug-in systems. In ICSE, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. M. J. Harrold, J. A. Jones, T. Li, D. Liang, A. Orso, M. Pennings, S. Sinha, S. A. Spoon, and A. Gujarathi. Regression test selection for Java software. In OOPSLA, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. M. J. Harrold and M. L. Soffa. An incremental approach to unit testing during maintenance. In ICSM, 1988.Google ScholarGoogle ScholarCross RefCross Ref
  16. J. Hartmann. Applying selective revalidation techniques at Microsoft. In PNSQC, 2007.Google ScholarGoogle Scholar
  17. J. Hartmann. 30 years of regression testing: Past, present and future. In PNSQC, 2012.Google ScholarGoogle Scholar
  18. J. L. Hintze and R. D. Nelson. Violin plots: A box plot-density trace synergism. The American Statistician, 1998.Google ScholarGoogle Scholar
  19. Java, Java everywhere, 2012. http://sdtimes.com/content/article.aspx?ArticleID=36362.Google ScholarGoogle Scholar
  20. Q. Luo, F. Hariri, L. Eloussi, and D. Marinov. An empirical analysis of flaky tests. In FSE, 2014. to appear.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. S. Negara, N. Chen, M. Vakilian, R. E. Johnson, and D. Dig. A comparative study of manual and automated refactorings. In ECOOP, 2013. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. S. Negara, M. Codoban, D. Dig, and R. E. Johnson. Mining fine-grained code changes to detect unknown change patterns. In ICSE, 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. S. Negara, M. Vakilian, N. Chen, R. E. Johnson, and D. Dig. Is it dangerous to use version control histories to study source code evolution? In ECOOP, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. A. K. Onoma, W.-T. Tsai, M. Poonawala, and H. Suganuma. Regression testing in an industrial environment. Communications, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. A. Orso, T. Apiwattanapong, and M. J. Harrold. Leveraging field data for impact analysis and regression testing. In FSE, 2003. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. D. Parsons, T. Susnjak, and M. Lange. Influences on regression testing strategies in agile software development environments. SQJ, 2013.Google ScholarGoogle Scholar
  27. X. Qu, M. B. Cohen, and G. Rothermel. Configuration-aware regression testing: An empirical study of sampling and prioritization. In ISSTA, 2008. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. X. Ren, F. Shah, F. Tip, B. G. Ryder, and O. Chesley. Chianti: A tool for change impact analysis of Java programs. In OOPSLA, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. G. Rothermel and M. Harrold. Selecting regression tests for object-oriented software. In ICSM, 1994. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. G. Rothermel and M. Harrold. Empirical studies of a safe regression test selection technique. TOSEM, 1998.Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. G. Rothermel and M. J. Harrold. A safe, efficient regression test selection technique. TOSEM, 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Test case prioritization: An empirical study. In ICSM, 1999. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. D. Saff and M. D. Ernst. An experimental evaluation of continuous testing during development. In ISSTA, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. A. Srivastava and J. Thiagarajan. Effectively prioritizing tests in development environment. In ISSTA, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. D. Talby, A. Keren, O. Hazzan, and Y. Dubinsky. Agile software testing in a large-scale project. Software, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Streamline testing process with test impact analysis, 2013. http://msdn.microsoft.com/en-us/library/ff576128%28v=vs.100%29.aspx.Google ScholarGoogle Scholar
  37. Testing at the speed and scale of Google, 2011. http://goo.gl/OKqBk.Google ScholarGoogle Scholar
  38. Tools for continuous integration at Google scale, 2011. http://www.youtube.com/watch?v=b52aXZ2yi08.Google ScholarGoogle Scholar
  39. M. Vakilian, N. Chen, S. Negara, B. A. Rajkumar, B. P. Bailey, and R. E. Johnson. Use, disuse, and misuse of automated refactorings. In ICSE, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. D. Willmor and S. M. Embury. A safe regression test selection technique for database-driven applications. In ICSM, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. W. E. Wong, J. R. Horgan, S. London, and H. Agrawal. A study of effective regression testing in practice. In ISSRE, 1997. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. S. Yau and Z. Kishimoto. A method for revalidating modified programs in the maintenance phase. In COMPSAC, 1987.Google ScholarGoogle Scholar
  43. S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: A survey. STVR, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. L. Zhang, M. Kim, and S. Khurshid. Localizing failure-inducing program edits based on spectrum information. In ICSM, 2011. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. An empirical evaluation and comparison of manual and automated test selection

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in
    • Published in

      cover image ACM Conferences
      ASE '14: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering
      September 2014
      934 pages
      ISBN:9781450330138
      DOI:10.1145/2642937

      Copyright © 2014 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 15 September 2014

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article

      Acceptance Rates

      ASE '14 Paper Acceptance Rate82of337submissions,24%Overall Acceptance Rate82of337submissions,24%

      Upcoming Conference

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader