Matthew J. Rummel
ABSTRACT
Regression test prioritization techniques re-order the execution of a test suite in an attempt to ensure that defects are revealed earlier in the test execution phase. In prior work, test suites were prioritized with respect to their ability to satisfy control flow-based and mutation-based test adequacy criteria. In this paper, we propose an approach to regression test prioritization that leverages the all-DUs test adequacy criterion that focuses on the definition and use of variables within the program under test. Our prioritization scheme is motivated by empirical studies that have shown that (i) tests fulfilling the all-DUs test adequacy criteria are more likely to reveal defects than those that meet the control flow-based criteria, (ii) there is an unclear relationship between all-DUs and mutation-based criteria, and (iii) mutation-based testing is significantly more expensive than testing that relies upon all-DUs.In support of our prioritization technique, we provide a formal statement of the algorithms and equations that we use to instrument the program under test, perform test suite coverage monitoring, and calculate test adequacy. Furthermore, we examine the architecture of a tool that implements our novel prioritization scheme and facilitates experimentation. The use of this tool in a preliminary experimental evaluation indicates that, for three case study applications, our prioritization can be performed with acceptable time and space overheads. Finally, these experiments also demonstrate that the prioritized test suite can have an improved potential to identify defects earlier during the process of test execution.
- Boris Beizer. Software Testing Techniques, Van Nostrond Reinhold, New York, NY, 1990. Google Scholar
Digital Library
- James Bieman, Sudipto Ghosh, and Roger Alexander. A technique for mutation of Java objects. In Proceedings of the 16th IEEE International Conference on Automated Software Engineering, November 2001. Google ScholarDigital Library
- Evelyn Duesterwald, Rajiv Gupta, and Mary Lou Soffa. A demand-driven analyzer for data flow testing at the integration level. In Proceedings of the 18th International Conference on Software Engineering, pages 575--584, 1996. Google ScholarDigital Library
- Sebastian Elbaum, Alexey G. Malishevsky, and G. Rothermel. Prioritizing test cases for regression testing. In Proceedings of the International Symposium on Software Testing and Analysis, pages 102--112. ACM Press, August 2000. Google ScholarDigital Library
- Sebastian Elbaum, Gregg Rothermel, Satya Kanduri, and Alexey G. Malishevsky. Selecting a cost-effective test case prioritization technique. Technical Report 03-01-01, Department of Computer Science and Engineering, University of Nebraska - Lincoln, January 2003.Google Scholar
- Sebastian G. Elbaum, Alexey G. Malishevsky, and Gregg Rothermel. Incorporating varying test costs and fault severities into test case prioritization. In International Conference on Software Engineering, pages 329--338, 2001. Google ScholarDigital Library
- Phyllis G. Frankl, Stewart N. Weiss, and Cang Hu. All-uses vs mutation testing: an experimental comparison of effectiveness. J. Syst. Softw., 38(3):235--253, 1997. Google ScholarDigital Library
- Phyllis G. Frankl and Elaine J. Weyuker. An applicable family of data flow testing criteria. IEEE Transactions on Software Engineering, 14(10):1483--1498, October 1988. Google ScholarDigital Library
- Dick Hamlet and Joe Maybee. The Engineering of Software. Addison Wesley, Boston, MA, 2001.Google Scholar
- Michael Harder, Jeff Mellen, and Michael D. Ernst. Improving test suites via operational abstraction. In Proceedings of the 24th International Conference on Software Engineering, pages 60--71, 2003. Google ScholarDigital Library
- Monica Hutchins, Herb Foster, Tarak Goradia, and Thomas Ostrand. Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In Proceedings of the 16th International Conference on Software Engineering, pages 191--200. IEEE Computer Society Press, 1994. Google ScholarDigital Library
- Gregory M. Kapfhammer. The Computer Science Handbook, chapter Software Testing. CRC Press, June 2004.Google Scholar
- Gregory M. Kapfhammer and Mary Lou Soffa. A family of test adequacy criteria for database-driven applications. In Proceedings of the 9th European Software Engineering Conference and the 11th ACM SIGSOFT Symposium on Foundations of Software Engineering, 2003. Google ScholarDigital Library
- Yu-Seung Ma, Yong-Rae Kwon, and Jeff Offutt. Inter-class mutation operators for Java. In Proceedings of the Twelfth International Symposium on Software Reliability Engineering, November 2002. Google ScholarDigital Library
- G. Rothermel and M. J. Harrold. A framework for evaluating regression test selection techniques. In Proceedings of the 16th International Conference on Software Engineering, pages 201--210. IEEE Computer Society Press, May 1994. Google ScholarDigital Library
- G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Test case prioritization: An empirical study. In Proceedings of the International Conference on Software Maintenance, pages 179--188, August 1999. Google ScholarDigital Library
- G. Rothermel, Roland H. Untch, Chengyun Chu, and M. J. Harrold. Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering, 27(10):929--948, October 2001. Google ScholarDigital Library
- Raja Vallée-Rai, Laurie Hendren, Vijay Sundaresan, Patrick Lam, Etienne Gagnon, and Phong Co. Soot - a Java optimization framework. In Proceedings of CASCON 1999, pages 125--135, 1999.Google ScholarDigital Library
Recommendations
Adaptive random prioritization for interaction test suites
SAC '14: Proceedings of the 29th Annual ACM Symposium on Applied ComputingCombinatorial interaction testing (CIT), a black-box testing method, has been well studied in recent years. It aims at constructing an effective interaction test suites, so as to identify the faults that are caused by interactions among parameters. ...
Regression test suite prioritization using system models
During regression testing, a modified system is often retested using an existing test suite. Since the size of the test suite may be very large, testers are interested in detecting faults in the modified system as early as possible during this retesting ...
Comments