Yingfei Xiong
Dan Hao
ABSTRACT
Traditional test oracles are defined on the outputs of test executions, and cannot assert internal states of executions. Traditional assertions are common to all test execution, and are usually more difficult to construct than on oracle for one test input. In this paper we propose the concept of inner oracles, which are assertions on internal states that are specific to one test input. We first motivate the necessity of inner oracles, and then show that it can be implemented easily using the available programming mechanisms. Next, we report two initial empirical studies on inner oracles, showing that inner oracles have a significant impact on both the fault-detection capability of tests and the performance of test suite reduction. Finally, we highlight the implications of inner oracles on several research and practical problems.
- E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo. The oracle problem in software testing: A survey. IEEE TSE, 41(5):507–525, 2014.Google Scholar
- K. Beck. Test-driven development: by example. Addison-Wesley Professional, 2003. Google ScholarDigital Library
- H. Do, S. Elbaum, and G. Rothermel. Supporting controlled experimentation with testing techniques: An infrastructure and its potential impact. ESE, 10(4):405–435, 2005. Google ScholarDigital Library
- C. Fang, Z. Chen, and B. Xu. Comparing logic coverage criteria on test case prioritization. SCIENCE CHINA Information Sciences, 55(12):2826–2840, 2012.Google ScholarCross Ref
- R. G. Hamlet. Testing programs with the aid of a compiler. IEEE TSE, 3(4):279–290, 1977. Google ScholarDigital Library
- D. Hao, L. Zhang, M. Liu, H. Li, and J. Sun. Test-data generation guided by static defect detection. JCST, 24(2):284–293, 2009. Google ScholarDigital Library
- D. Hao, L. Zhang, L. Zhang, G. Rothermel, and H. Mei. A unified test-case prioritization approach. ACM TOSEM, 24(2):1–31, 2014. Google ScholarDigital Library
- W. E. Howden. Weak mutation testing and completeness of test sets. IEEE TSE, 8(4):371–379, 1982. Google ScholarDigital Library
- G. Kiczales, J. Lamping, A. Mendhekar, C. Maeda, C. Lopes, J.-M. Loingtier, and J. Irwin. Aspect-oriented programming. Springer, 1997.Google ScholarCross Ref
- P. McMinn. Search-based software test data generation: A survey. STVR, 14(2):105–156, 2004. Google ScholarDigital Library
- H. Mei, D. Hao, L. Zhang, L. Zhang, J. Zhou, and G. Rothermel. A static approach to prioritizing JUnit test cases. IEEE TSE, 38(6):1258–1275, 2012. Google ScholarDigital Library
- L. Pinto, S. Sinha, and A. Orso. Understanding myths and realities of test-suite evolution. In Proc. FSE, pages 1–11, 2012. Google ScholarDigital Library
- D. Schuler and A. Zeller. Javalanche: Efficient mutation testing for Java. In Proc. FSE, pages 297–298, 2009. Google ScholarDigital Library
- M. Staats, M. W. Whalen, and M. P. Heimdahl. Programs, tests, and oracles: The foundations of testing revisited. In Proc. ICSE, pages 391–400, 2011. Google ScholarDigital Library
- J. M. Voas. PIE: A dynamic failure-based technique. IEEE TSE, 18(8):717–727, 1992. Google ScholarDigital Library
- T. Xie. Augmenting automatically generated unit-test suites with regression oracle checking. In Proc. ECOOP, pages 380–403. Springer, 2006. Google ScholarDigital Library
- J. Zhang, J. Chen, D. Hao, Y. Xiong, B. Xie, L. Zhang, and H. Mei. Search-based inference of polynomial metamorphic relations. In ASE, pages 701–712, 2014. Google ScholarDigital Library
- L. Zhang, D. Marinov, L. Zhang, and S. Khurshid. An empirical study of junit test-suite reduction. In Proc. ISSRE, pages 170–179, 2011. Google ScholarDigital Library
- H. Zhong, L. Zhang, and H. Mei. An experimental study of four typical test suite reduction techniques. IST, 50(6):534–546, 2008. Google ScholarDigital Library
- H. Zhu. A formal analysis of the subsume relation between software test adequacy criteria. IEEE TSE, 22(4):248–255, 1996. Google ScholarDigital Library
Index Terms
- Inner oracles: input-specific assertions on internal states
Recommendations
Mutation-driven generation of unit tests and oracles
ISSTA '10: Proceedings of the 19th international symposium on Software testing and analysisTo assess the quality of test suites, mutation analysis seeds artificial defects (mutations) into programs; a non-detected mutation indicates a weakness in the test suite. We present an automated approach to generate unit tests that detect these ...
Mutation-Driven Generation of Unit Tests and Oracles
To assess the quality of test suites, mutation analysis seeds artificial defects (mutations) into programs; a nondetected mutation indicates a weakness in the test suite. We present an automated approach to generate unit tests that detect these ...
Assertions are strongly correlated with test suite effectiveness
ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software EngineeringCode coverage is a popular test adequacy criterion in practice. Code coverage, however, remains controversial as there is a lack of coherent empirical evidence for its relation with test suite effectiveness. More recently, test suite size has been ...
Comments