Alessandro Orso
Tao Xie
ABSTRACT
During maintenance, it is common to run the new version of a program against its existing test suite to check whether the modifications in the program introduced unforeseen side effects. Although this kind of regression testing can be effective in identifying some change-related faults, it is limited by the quality of the existing test suite. Because generating tests for real programs is expensive, developers build test suites by finding acceptable tradeoffs between cost and thoroughness of the tests. Such test suites necessarily target only a small subset of the program's functionality and may miss many regression faults. To address this issue, we introduce the concept of behavioral regression testing, whose goal is to identify behavioral differences between two versions of a program through dynamic analysis. Intuitively, given a set of changes in the code, behavioral regression testing works by (1) generating a large number of test cases that focus on the changed parts of the code, (2) running the generated test cases on the old and new versions of the code and identifying differences in the tests' outcome, and (3) analyzing the identified differences and presenting them to the developers. By focusing on a subset of the code and leveraging differential behavior, our approach can provide developers with more (and more focused) information than traditional regression testing techniques. This paper presents our approach and performs a preliminary assessment of its feasibility.
- T. Apiwattanapong, A. Orso, and M. J. Harrold. A differencing algorithm for object-oriented programs. In Proc. IEEE International Conference on Automated Software Engineering (ASE 2004), pages 2--13, 2004. Google Scholar
Digital Library
- T. Apiwattanapong, R. Santelices, P. K. Chittimalli, A. Orso, and M. J. Harrold. Matrix: Maintenance-oriented testing requirements identifier and examiner. In Proc. Testing: Academic & Industrial Conference on Practice And Research Techniques (TAIC-PART 2006), pages 137--146, 2006. Google ScholarDigital Library
- M. Boshernitsan, R. Doong, and A. Savoia. From Daikon to Agitator: lessons and challenges in building a commercial tool for developer testing. In Proc. ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2006), pages 169--180, 2006. Google ScholarDigital Library
- B. Daniel, D. Dig, K. Garcia, and D. Marinov. Automated testing of refactoring engines. In Proc. joint meeting of European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE 2007), pages 185--194, 2007. Google ScholarDigital Library
- R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on test data selection: Help for the practicing programmer. IEEE Computer, 11(4): 34--41, 1978. Google ScholarDigital Library
- R. A. DeMillo and A. J. Offutt. Constraint-based automatic test data generation. IEEE Transactions on Software Engineering, 17(9): 900--910, 1991. Google ScholarDigital Library
- S. Elbaum, H. N. Chin, M. Dwyer, and J. Dokulil. Carving differential unit test cases from system test cases. In Proc. ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE 2006), pages 253--264, 2006. Google ScholarDigital Library
- R. B. Evans and A. Savoia. Differential testing: a new approach to change detection. In Proc. joint meeting of European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE 2007), pages 549--552, 2007. Google ScholarDigital Library
- E. Gamma and K. Beck. JUnit: a regression testing framework. http://www.junit.org, 2002.Google Scholar
- T. Graves, M. J. Harrold, J.-M. Kim, A. Porter, and G. Rothermel. An empirical study of regression test selection techniques. In Proc. International Conference on Software Engineering (ICSE 1998), pages 188--197, 1998. Google ScholarDigital Library
- A. Groce, G. Holzmann, and R. Joshi. Randomized differential testing as a prelude to formal verification. In Proc. International Conference on Software Engineering (ICSE 2007), pages 621--631, 2007. Google ScholarDigital Library
- JUnit Factory website, 2008. http://www.junitfactory.com/.Google Scholar
- B. Korel and A. M. Al-Yami. Automated regression test generation. In Proc. ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 1998), pages 143--152, 1998. Google ScholarDigital Library
- R. Lämmel and W. Schulte. Controllable combinatorial coverage in grammar-based testing. In Proc. IFIP International Conference on Testing Communicating Systems (TestCom 2006), pages 19--38, 2006. Google ScholarDigital Library
- G. T. Leavens, A. L. Baker, and C. Ruby. Preliminary design of JML: A behavioral interface specification language for Java. Technical Report TR 98-06i, Department of Computer Science, Iowa State University, 1998.Google Scholar
- H. K. N. Leung and L. White. Insights into regression testing. In Proc. International Conference on Software Maintenance (ICSM 1989), pages 60--69, 1989.Google ScholarCross Ref
- W. M. McKeeman. Differential testing for software. Digital Technical Journal of Digital Equipment Corporation, 10(1): 100--107, 1998.Google Scholar
- A. Orso, S. Joshi, M. Burger, and A. Zeller. Isolating relevant component interactions with JINSI. In Proc. International ICSE Workshop on Dynamic Analysis (WODA 2006), pages 3--9, 2006. Google ScholarDigital Library
- A. Orso and B. Kennedy. Selective capture and replay of program executions. In Proc. International ICSE Workshop on Dynamic Analysis (WODA 2005), pages 29--35, 2005. Google ScholarDigital Library
- A. Orso, N. Shi, and M. J. Harrold. Scaling regression testing to large software systems. In Proc. ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE 2004), pages 241--252, 2004. Google ScholarDigital Library
- C. Pacheco and M. D. Ernst. Randoop: Feedback-directed random testing for Java. In Companion to the 22nd ACM SIGPLAN Conference on Object Oriented Programming Systems and Applications (OOPSLA Companion 2007), pages 815--816, 2007. Google ScholarDigital Library
- Parasoft Jtest manuals version 4.5. Online manual, 2003. http://www.parasoft.com/.Google Scholar
- A. Podgurski and E. J. Weyuker. Re-estimation of software reliability after maintenance. In Proc. International Conference on Software Engineering (ICSE 1997), pages 79--85, 1997. Google ScholarDigital Library
- X. Ren, F. Shah, F. Tip, B. Ryder, and O. Chesley. Chianti: A tool for change impact analysis of Java programs. In Proc. ACM Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA 2004), pages 432--448, 2004. Google ScholarDigital Library
- G. Rothermel, R. Untch, C. Chu, and M. Harrold. Test case prioritization. IEEE Transactions on Software Engineering, 27(10): 929--948, October 2001. Google ScholarDigital Library
- D. Saff, S. Artzi, J. H. Perkins, and M. D. Ernst. Automatic test factoring for Java. In Proc. IEEE International Conference on Automated Software Engineering (ASE 2005), pages 114--123, 2005. Google ScholarDigital Library
- E. J. Weyuker. On testing non-testable programs. Compuer Journal, 25: 465--470, 1982.Google ScholarCross Ref
- T. Xie. Augmenting automatically generated unit-test suites with regression oracle checking. In Proc. European Conference on Object-Oriented Programming (ECOOP 2006), pages 380--403, 2006. Google ScholarDigital Library
- T. Xie, K. Taneja, S. Kale, and D. Marinov. Towards a framework for differential unit testing of object-oriented programs. In Proc. International Workshop on Automation of Software Test (AST 2007), pages 5--11, 2007. Google ScholarDigital Library
Index Terms
- BERT: BEhavioral Regression Testing
Recommendations
Automated Behavioral Regression Testing
ICST '10: Proceedings of the 2010 Third International Conference on Software Testing, Verification and ValidationWhen a program is modified during software evolution, developers typically run the new version of the program against its existing test suite to validate that the changes made on the program did not introduce unintended side effects (i.e., regression ...
BERT: a tool for behavioral regression testing
FSE '10: Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineeringDuring maintenance, software is modified and evolved to enhance its functionality, eliminate faults, and adapt it to changed or new platforms. In this demo, we present BERT, a tool for helping developers identify regression faults that they may have ...
Scaling regression testing to large software systems
SIGSOFT '04/FSE-12: Proceedings of the 12th ACM SIGSOFT twelfth international symposium on Foundations of software engineeringWhen software is modified, during development and maintenance, it is <i>regression tested</i> to provide confidence that the changes did not introduce unexpected errors and that new features behave as expected. One important problem in regression ...
Comments