ABSTRACT
Testing consumes significant human and machine resources, especially for large, complex systems such as database servers. While a variety of testing approaches have been proposed to improve the efficiency of the testing process, it is difficult to evaluate these approaches. Mutation testing has been proposed as a way to assess the adequacy of a test suite, assigning a score that can be used to compare testing approaches. While promising, serious obstacles appear to prevent mutation testing with large software systems. Recent advances in mutation testing have scaled to medium-sized programs of around 100,000 lines of code but to our knowledge there are no reported studies working with large systems with millions of lines of code and other features of database systems that complicate testing. In this paper we explore using mutation testing on a database server to evaluate its suitability for comparing test suites or testing approaches.
- Hardik Bati, Leo Giakoumakis, Steve Herbert, and Aleksandras Surna. 2007. A genetic approach for random testing of database systems. In Proceedings of the 33rd international conference on Very large data bases (VLDB '07). VLDB Endowment 1243--1251. Google ScholarDigital Library
- R. A. DeMillo, R. J. Lipton, and F. G. Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer 11, 4 (April 1978), 34-41. DOI=http://dx.doi.org/10.1109/C-M.1978.218136 Google ScholarDigital Library
- R. J. Lipton, "Fault Diagnosis of Computer Programs," student report, Carnegie Mellon Univ., 1971.Google Scholar
- Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Trans. Softw. Eng. 37, 5 (September 2011), 649-678. DOI= http://dx.doi.org/10.1109/TSE.2010.62 Google ScholarDigital Library
- David Schuler, Valentin Dallmeier, and Andreas Zeller. 2009. Efficient mutation testing by checking invariant violations. In Proceedings of the eighteenth international symposium on Software testing and analysis (ISSTA '09). ACM, New York, NY, USA, 69-80. DOI= http://doi.acm.org/10.1145/1572272.1572282 Google ScholarDigital Library
- Roland H. Untch. 1992. Mutation-based software testing using program schemata. In Proceedings of the 30th annual Southeast regional conference (ACM-SE 30). ACM, New York, NY, USA, 285-291. DOI=http://doi.acm.org/10.1145/503720.503749 Google ScholarDigital Library
Index Terms
Mutatis mutandis: evaluating DBMS test adequacy with mutation testing
Recommendations
Faster mutation testing inspired by test prioritization and reduction
ISSTA 2013: Proceedings of the 2013 International Symposium on Software Testing and AnalysisMutation testing is a well-known but costly approach for determining test adequacy. The central idea behind the approach is to generate mutants, which are small syntactic transformations of the program under test, and then to measure for a given test ...
Mitigating the effects of flaky tests on mutation testing
ISSTA 2019: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and AnalysisMutation testing is widely used in research as a metric for evaluating the quality of test suites. Mutation testing runs the test suite on generated mutants (variants of the code under test), where a test suite kills a mutant if any of the tests fail ...
Checked Coverage and Object Branch Coverage: New Alternatives for Assessing Student-Written Tests
SIGCSE '15: Proceedings of the 46th ACM Technical Symposium on Computer Science EducationMany educators currently use code coverage metrics to assess student-written software tests. While test adequacy criteria such as statement or branch coverage can also be used to measure the thoroughness of a test suite, they have limitations. Coverage ...
Comments