ABSTRACT
The configuration spaces of modern software systems are too large to test exhaustively. Combinatorial interaction testing (CIT) approaches, such as covering arrays, systematically sample the configuration space and test only the selected configurations. The basic justification for CIT approaches is that they can cost-effectively exercise all system behaviors caused by the settings of t or fewer options. We conjecture, however, that in practice many such behaviors are not actually tested because of masking effects -- failures that perturb execution so as to prevent some behaviors from being exercised. In this work we present a feedback-driven, adaptive, combinatorial testing approach aimed at detecting and working around masking effects. At each iteration we detect potential masking effects, heuristically isolate their likely causes, and then generate new covering arrays that allow previously masked combinations to be tested in the subsequent iteration. We empirically assess the effectiveness of the proposed approach on two large widely used open source software systems. Our results suggest that masking effects do exist and that our approach provides a promising and efficient way to work around them.
- Advanced Combinatorial Testing System (ACTS), 2010. http://csrc.nist.gov/groups/SNS/acts/documents/comparison-report.html.Google Scholar
- R. Brownlie, J. Prowse, and M. S. Phadke. Robust testing of AT&T PMX/StarMAIL using OATS. AT&T Technical Journal, 71(3):41--7, 1992.Google ScholarCross Ref
- R. C. Bryce and C. J. Colbourn. Prioritized interaction testing for pair-wise coverage with seeding and constraints. Journal of Information and Software Technology, 48(10):960--970, 2006.Google ScholarCross Ref
- R. C. Bryce, A. Rajan, and M. P. Heimdahl. Interaction testing in model-based development: Effect on model-coverage. In Asia Pacific Software Engineering Conference, pages 259--268, 2006. Google ScholarDigital Library
- K. Burr and W. Young. Combinatorial test techniques: Table-based automation, test generation and code coverage. In Proceedings of the International Conference on Software Testing Analysis & Review, 1998.Google Scholar
- A. Calvagna and A. Gargantini. A logic-based approach to combinatorial testing with constraints. In Tests and Proofs, Lecture Notes in Computer Science, 4966, pages 66--83, 2008. Google ScholarDigital Library
- D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton. The AETG system: an approach to testing based on combinatorial design. IEEE Transactions on Software Engineering, 23(7):437--44, 1997. Google ScholarDigital Library
- M. B. Cohen, M. B. Dwyer, and J. Shi. Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach. IEEE Transactions on Software Engineering, 34(5):633--650, 2008. Google ScholarDigital Library
- S. R. Dalal, A. Jain, N. Karunanithi, J. M. Leaton, C. M. Lott, G. C. Patton, and B. M. Horowitz. Model-based testing in practice. In Proceedings of the International Conference on Software Engineering, pages 285--294, 1999. Google ScholarDigital Library
- S. Fouché, M. B. Cohen, and A. Porter. Incremental covering array failure characterization in large configuration spaces. In Proceedings of the International Symposium on Software Testing and Analysis, pages 177--188, New York, NY, USA, 2009. ACM. Google ScholarDigital Library
- D. Kuhn, D. R. Wallace, and A. M. Gallo. Software fault interactions and implications for software testing. IEEE Transactions on Software Engingeering, 30(6):418--421, 2004. Google ScholarDigital Library
- T. M. Mitchell. Machine Learning. McGraw Hill, 1997. Google ScholarDigital Library
- MySQL, 2006. http://www.mysql.com.Google Scholar
- T. J. Ostrand and M. J. Balcer. The category-partition method for specifying and generating functional tests. Communications of the ACM, 31:678--686, 1988. Google ScholarDigital Library
- X. Qu, M. B. Cohen, and G. Rothermel. Configuration-aware regression testing: An empirical study of sampling and prioritization. In International Symposium on Software Testing and Analysis, pages 75--85, July 2008. Google ScholarDigital Library
- X. Qu, M. B. Cohen, and K. M. Woolf. Combinatorial interaction regression testing: A study of test case generation and prioritization. In International Conference on Software Maintenance, pages 255--264, October 2007.Google ScholarCross Ref
- I. H. Witten and E. Frank. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, 1999. Google ScholarDigital Library
- C. Yilmaz, M. B. Cohen, and A. Porter. Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Transactions on Software Engineering, 31(1):20--34, Jan 2006. Google ScholarDigital Library
- X. Yuan, M. Cohen, and A. M. Memon. Covering array sampling of input event sequences for automated GUI testing. In International Conference on Automated Software Engineering, pages 405--408, 2007. Google ScholarDigital Library
Index Terms
- Feedback driven adaptive combinatorial testing
Recommendations
Characterizing failure-causing parameter interactions by adaptive testing
ISSTA '11: Proceedings of the 2011 International Symposium on Software Testing and AnalysisCombinatorial testing is a widely used black-box testing technique, which is used to detect failures caused by parameter interactions (we call them faulty interactions). Traditional combinatorial testing techniques provide fault detection, but most of ...
Test Case-Aware Combinatorial Interaction Testing
The configuration spaces of modern software systems are too large to test exhaustively. Combinatorial interaction testing (CIT) approaches, such as covering arrays, systematically sample the configuration space and test only the selected configurations ...
Adaptive Combinatorial Testing
QSIC '13: Proceedings of the 2013 13th International Conference on Quality SoftwareCombinatorial Testing (CT) has been proven to be effective in detecting and locating the interaction triggered failure in the last 20 years. But CT still suffers from many challenges, such as modeling for CT, sampling mechanisms for test generation, ...
Comments