Myra B. Cohen
Matthew B. Dwyer
ABSTRACT
Combinatorial interaction testing (CIT) is a method to sample configurations of a software system systematically for testing. Many algorithms have been developed that create CIT samples, however few have considered the practical concerns that arise when adding constraints between combinations of options. In this paper, we survey constraint handling techniques in existing algorithms and discuss the challenges that they present. We examine two highly-configurable software systems to quantify the nature of constraints in real systems. We then present a general constraint representation and solving technique that can be integrated with existing CIT algorithms and compare two constraint-enhanced algorithm implementations with existing CIT tools to demonstrate feasibility.
- R. Brownlie, J. Prowse, and M. S. Phadke. Robust testing of AT&T PMX/StarMAIL using OATS. AT&T Technical Journal, 71(3):41--47, 1992.Google Scholar
Cross Ref
- R. C. Bryce and C. J. Colbourn. Prioritized interaction testing for pair-wise coverage with seeding and constraints. Journal of Information and Software Technology, 48(10):960--970, 2006.Google ScholarCross Ref
- R. C. Bryce, C. J. Colbourn, and M. B. Cohen. A framework of greedy methods for constructing interaction test suites. In Proceedings of the International Conference on Software Engineering, pages 146--155, May 2005. Google ScholarDigital Library
- D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton. The AETG system: an approach to testing based on combinatorial design. IEEE Transactions on Software Engineering, 23(7):437--444, 1997. Google ScholarDigital Library
- M. B. Cohen, C. J. Colbourn, P. B. Gibbons, and W. B. Mugridge. Constructing test suites for interaction testing. In Proceedings of the International Conference on Software Engineering, pages 38--48, May 2003. Google ScholarDigital Library
- M. B. Cohen, C. J. Colbourn, and A. C. H. Ling. Augmenting simulated annealing to build interaction test suites. In 14th IEEE International Symposium on Software Reliability Engineering, pages 394--405, November 2003. Google ScholarDigital Library
- M. B. Cohen, M. B. Dwyer, and J.Shi. Coverage and adequacy in software product line testing. In Proceedings of the Workshop on the Role of Architecture for Testing and Analysis, pages 53--63, July 2006. Google ScholarDigital Library
- C. J. Colbourn, M. B. Cohen, and R. C. Turban. A deterministic density algorithm for pairwise interaction coverage. In IASTED Proceedings of the International Conference on Software Engineering, pages 345--352, February 2004.Google Scholar
- J. Czerwonka. Pairwise testing in real world. In Pacific Northwest Software Quality Conference, pages 419--430, October 2006.Google Scholar
- I. S. Dunietz, W. K. Ehrlich, B. D. Szablak, C. L. Mallows, and A. Iannino. Applying design of experiments to software testing. In Proceedings of the International Conference on Software Engineering, pages 205--215, 1997. Google ScholarDigital Library
- D. Dvorak, R. Rasmussen, G. Reeves, and A. Sacks. Software architecture themes in JPL's mission data system. In Proceedings of the 2000 IEEE Aerospace Conference, Mar. 2000.Google ScholarCross Ref
- Free Software Foundation. GNU 4.1.1 manpages. http://gcc.gnu.org/onlinedocs/gcc-4.1.1/gcc/, 2005.Google Scholar
- A. Hartman. Software and hardware testing using combinatorial covering suites. In Graph Theory, Combinatorics and Algorithms: Interdisciplinary Applications, pages 327--266, 2005.Google Scholar
- A. Hartman and L. Raskin. Problems and algorithms for covering arrays. Discrete Math, 284:149--156, 2004.Google ScholarDigital Library
- B. Hnich, S. Prestwich, E. Selensky, and B. Smith. Constraint models for the covering test problem. Constraints, 11:199--219, 2006. Google ScholarDigital Library
- G. J. Holzmann. The model checker SPIN. IEEE Transactions on Software Engineering, 23(5):279--295, 1997. Google ScholarDigital Library
- G. J. Holzmann. On-the-fly, LTL model checking with SPIN: Man pages. http://spinroot.com/spin/Man/index.html, 2006.Google Scholar
- IBM alphaWorks. IBM Intelligent Test Case Handler. http://www.alphaworks.ibm.com/tech/whitch, 2005.Google Scholar
- D. Kuhn, D. R. Wallace, and A. M. Gallo. Software fault interactions and implications for software testing. IEEE Transactions on Software Engineering, 30(6):418--421, 2004. Google ScholarDigital Library
- C. Lott, A. Jain, and S. Dalal. Modeling requirements for combinatorial software testing. In Workshop on Advances in Model-based Software Testing, pages 1--7, May 2005. Google ScholarDigital Library
- S. Malik. zchaff. http://www.princeton.edu/~chaff/zchaff.html, 2004.Google Scholar
- R. Mandl. Orthogonal latin squares: An application of experiment design to compiler testing. Communications of the ACM, 28(10):1054--1058, October 1985. Google ScholarDigital Library
- J. D. McGregor. Testing a software product line. Technical report, Carnegie Mellon, Software Engineering Institute, December 2001.Google Scholar
- Nokia Corporation. Nokia mobile phone line. http://www.nokiausa.com/phones, 2007.Google Scholar
- K. Nurmela. Upper bounds for covering arrays by tabu search. Discrete Applied Mathematics, 138(1-2):143--152, 2004. Google ScholarDigital Library
- D. L. Parnas. On the design and development of program families. IEEE Transactions on Software Engineering, 2(1):1--9, 1976.Google ScholarDigital Library
- G. Sherwood. Testcover.com. http://testcover.com/pub/constex.php, 2006.Google Scholar
- G. Sherwood. Personal communication, 2007.Google Scholar
- K. C. Tai and Y. Lei. A test generation strategy for pairwise testing. IEEE Transactions on Software Engineering, 28(1):109--111, 2002. Google ScholarDigital Library
- Y. Tung and W. S. Aldiwan. Automating test case generation for the new generation mission software system. In Proceedings of the IEEE Aerospace Conference, pages 431--437, 2000.Google Scholar
- A. W. Williams and R. L. Probert. A measure for component interaction test coverage. In Proceedings of the ACS/IEEE International Conference on Computer Systems and Applications, pages 301--311, Los Alamitos, CA, October 2001. IEEE. Google ScholarDigital Library
- C. Yilmaz, M. B. Cohen, and A. Porter. Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Transactions on Software Engineering, 31(1):20--34, 2006. Google ScholarDigital Library
Index Terms
- Interaction testing of highly-configurable systems in the presence of constraints
Recommendations
Test Case-Aware Combinatorial Interaction Testing
The configuration spaces of modern software systems are too large to test exhaustively. Combinatorial interaction testing (CIT) approaches, such as covering arrays, systematically sample the configuration space and test only the selected configurations ...
Test suite prioritization by interaction coverage
DOSTA '07: Workshop on Domain specific approaches to software test automation: in conjunction with the 6th ESEC/FSE joint meetingEvent-driven software (EDS) is a widely used class of software that takes sequences of events as input, changes state, and outputs new event sequences. Managing the size of tests suites for EDS is difficult as the number of event combinations and ...
Numerical Constraints for Combinatorial Interaction Testing
ICST '12: Proceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and ValidationConstraints can be found in many specifications of a software system. The impact of constraints varies with the test problem, but their presence causes problems for many existing combinatorial interaction testing (CIT) tools. Of the numerous existing ...
Comments