Skip to main content
SpringerLink
Log in

A Formal Logic Approach to Constrained Combinatorial Testing

  • Published:
Journal of Automated Reasoning Aims and scope Submit manuscript

Abstract

Combinatorial testing is as an effective testing technique to reveal failures in a given system, based on input combinations coverage and combinatorial optimization. Combinatorial testing of strength t (t ≥ 2) requires that each t-wise tuple of values of the different system input parameters is covered by at least one test case. Combinatorial test suite generation algorithms aim at producing a test suite covering all the required tuples in a small (possibly minimal) number of test cases, in order to reduce the cost of testing. The most used combinatorial technique is the pairwise testing (t = 2) which requires coverage of all pairs of input values. Constrained combinatorial testing takes also into account constraints over the system parameters, for instance forbidden tuples of inputs, modeling invalid or not realizable input values combinations. In this paper a new approach to combinatorial testing, tightly integrated with formal logic, is presented. In this approach, test predicates are used to formalize combinatorial testing as a logical problem, and an external formal logic tool is applied to solve it. Constraints over the input domain are expressed as logical predicates too, and effectively handled by the same tool. Moreover, inclusion or exclusion of select tuples is supported, allowing the user to customize the test suite layout. The proposed approach is supported by a prototype tool implementation and results of experimental assessment are also presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Atlee, J.M., Buckley, M.A.: A logic-model semantics for SCR software requirements. In: International Symposium on Software Testing and Analysis. ACM (1996)

  2. Biere, A., Cimatti, A., Clarke, E.M., Zhu, Y.: Symbolic model checking without bdds. In: In TACAS 99: Proceedings of the 5th International Conference on Tools and Algorithms for Construction and Analysis of Systems, London, UK, 1999, pp. 193–207. Springer, Heidelberg, ISBN 3-540-65703-7 (1999)

    Google Scholar 

  3. Blass, A., Gurevich, Y.: Pairwise testing. In: Current Trends in Theoretical Computer Science, pp. 237–266. World Scientific Press, Singapore (2004)

    Chapter  Google Scholar 

  4. Bose, R.C., Bush, K.A.: Orthogonal arrays of strength two and three. Ann. Math. Stat. 23(4), 508–524 (1952)

    Article  MATH  MathSciNet  Google Scholar 

  5. Brownlie, R., Prowse, J., Phadke, M.: Robust testing of AT&T PMX/starMAIL using OATS. ATT Tech. J. 71(3), 41–47 (1992)

    Google Scholar 

  6. Bryce, R.C., Colbourn, C.J.: Prioritized interaction testing for pair-wise coverage with seeding and constraints. Inf. Softw. Technol. 48(10), 960–970 (2006)

    Article  Google Scholar 

  7. Burr, K., Young, W.: Combinatorial test techniques: table-based automation, test generation, and code coverage. In: Proceedings of the Intl. Conf. on Software Testing Analysis and Review, pp. 503–513 (1998)

  8. Calvagna, A., Gargantini, A.: A logic-based approach to combinatorial testing with constraints. In: Beckert, B., Hähnle, R. (eds.) Tests and Proofs, Second International Conference, TAP 2008, Prato, Italy, 9–11 April 2008. Proceedings. Lecture Notes in Computer Science, vol. 4966, pp. 66–83. Springer, Heidelberg (2008)

    Google Scholar 

  9. Calvagna, A., Gargantini, A.: Using SRI SAL model checker for combinatorial tests generation in the presence of temporal constraints. In: Rushby, J., Shankar, N. (eds.) AFM’08: Third Workshop on Automated Formal Methods (satellite of CAV), pp. 43–52 (2008)

  10. Chvatal, V.: A greedy heuristic for the set-covering problem. Math. Oper. Res. 4(3), 233–235 (1979)

    Article  MATH  MathSciNet  Google Scholar 

  11. Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: an approach to testing based on combinatorial design. IEEE Trans. Softw. Eng. 23(7), 437–444 (1997)

    Article  Google Scholar 

  12. Cohen, M., Dwyer, M., Shi, J.: Exploiting constraint solving history to construct interaction test suites. In: Testing: Academic and Industrial Conference-Practice and Research Techniques (TAIC PART), London, September 2007, pp. 121–130 (2007)

  13. Cohen, M.B., Colbourn, C.J., Gibbons, P.B., Mugridge, W.B.: Constructing test suites for interaction testing. In: ICSE, pp. 38–48 (2003)

  14. Cohen, M.B., Dwyer, M.B., Shi, J.: Interaction testing of highly-configurable systems in the presence of constraints. In: ISSTA International Symposium on Software Testing and Analysis, pp. 129–139. ACM Press, New York, NY, USA (2007)

    Chapter  Google Scholar 

  15. Cohen, M.B., Dwyer, M.B., Shi, J.: Constructing interaction test suites for highly-configurable systems in the presence of constraints: a greedy approach. IEEE Trans. Softw. Eng. 34, 633–650 (2010)

    Article  Google Scholar 

  16. Czerwonka, J.: Pairwise testing in real world. In: 24th Pacific Northwest Software Quality Conference (2006)

  17. Dalal, S., Jain, A., Karunanithi, N., Leaton, J., Lott, C.: Model-based testing of a highly programmable system. ISSRE Proceedings 1998, 5–7 November, pp. 174–178 (1998)

  18. Dalal, S.R., Jain, A., Karunanithi, N., Leaton, J.M., Lott, C.M., Patton, G.C., Horowitz, B.M.: Model-based testing in practice. In: International Conference on Software Engineering ICSE, pp. 285–295. Association for Computing Machinery, New York (1999)

    Chapter  Google Scholar 

  19. de Moura, L., Owre, S., Rueß, H., Rushby, J., Shankar, N., Sorea, M., Tiwari, A.: SAL 2. In: Alur, R., Peled, D. (eds.) Computer-Aided Verification, CAV 2004. Lecture Notes in Computer Science, vol. 3114, pp. 496–500. Springer, Boston, MA (2004)

    Chapter  Google Scholar 

  20. Dunietz, I.S., Ehrlich, W.K., Szablak, B., Mallows, C., Iannino, A.: Applying design of experiments to software testing. In: Society, I. (ed.) Proc. Int’l Conf. Software Eng. (ICSE), pp. 205–215 (1997)

  21. Fraser, G., Wotawa, F.: Using LTL rewriting to improve the performance of model-checker based test-case generation. In: Proceedings of the 3rd International Workshop on Advances in Model Based Testing (A-MOST 2007), pp. 64–74 (2007)

  22. Fraser, G., Wotawa, F.: Ordering coverage goals in model checker based testing. In: Proceedings of the Fourth International Workshop on Advances in Model Based Testing (A-MOST 2008) (2008)

  23. Gargantini, A.: Using model checking to generate fault detecting tests. In: International Conference on Tests And Proofs (TAP). Lecture Notes in Computer Science (LNCS), vol. 4454, pp. 189–206. Springer, Heidelberg (2007)

    Google Scholar 

  24. Gargantini, A., Heitmeyer, C.: Using model checking to generate tests from requirements specifications. In: Nierstrasz, O., Lemoine, M. (eds.) Software Engineering ESEC/FSE’99. LNCS, vol. 1687 (1999)

  25. Gargantini, A., Riccobene, E.: Asm-based testing: coverage criteria and automatic test sequence generation. JUCS 7(11), 1050–1067 (2001)

    Google Scholar 

  26. Gargantini, A., Riccobene, E., Rinzivillo, S.: Using spin to generate tests from ASM specifications. In: ASM 2003 - Taormina, Italy, March 2003. Proceedings. LNCS, vol. 2589 (2003)

  27. Gargantini, A., Riccobene, E., Scandurra, P.: A metamodel-based language and a simulation engine for abstract state machines. JUCS 14, 1949–1983 (2008)

    Google Scholar 

  28. Grindal, M., Offutt, J., Andler, S.F.: Combination testing strategies: a survey. Softw. Test. Verif. Reliab. 15(3), 167–199 (2005)

    Article  Google Scholar 

  29. Hamon, G., de Moura, L., Rushby, J.: Generating efficient test sets with a model checker. In: 2nd International Conference on Software Engineering and Formal Methods, pp. 261–270. IEEE Computer Society, Beijing, China (2004)

    Google Scholar 

  30. Hartman, A.: Ibm intelligent test case handler: whitch. http://www.alpha-works.ibm.com/tech/whitch

  31. Hartman, A.: Graph theory, combinatorics and algorithms interdisciplinary applications. In: chap. Software and Hardware Testing Using Combinatorial Covering Suites, pp. 237–266. Springer (2005)

  32. Hartman, A., Raskin, L.: Problems and algorithms for covering arrays. Discrete Math. 284(1–3), 149–156 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  33. Hnich, B., Prestwich, S.D., Selensky, E., Smith, B.M.: Constraint models for the covering test problem. Constraints 11(2–3), 199–219 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  34. Holzmann, G.J.: The model checker SPIN. Softw. Eng. 23(5), 279–295 (1997)

    Article  MathSciNet  Google Scholar 

  35. Jenny Combinatorial Tool: http://www.burtleburtle.net/bob/math/jenny.html

  36. Kobayashi, N., Tsuchiya, T., Kikuno, T.: Non-specification-based approaches to logic testing for software. J. Inf. Softw. Technol. 44(2), 113–121 (2002)

    Article  MathSciNet  Google Scholar 

  37. Kuhn, D.R., Okum, V.: Pseudo-exhaustive testing for software. In: SEW ’06: IEEE/NASA Software Engineering Workshop, vol. 0, pp. 153–158. IEEE Computer Society, Los Alamitos, CA, USA (2006)

    Chapter  Google Scholar 

  38. Kuhn, D.R., Reilly, M.J.: An investigation of the applicability of design of experiments to software testing. In: Society, I. (ed.) 27th NASA/IEEE Software Engineering Workshop, pp. 91–95 (2002)

  39. Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng. 30(6), 418–421 (2004)

    Article  Google Scholar 

  40. Lott, C., Jain, A., Dalal, S.: Modeling requirements for combinatorial software testing. In: A-MOST ’05: Proceedings of the 1st International Workshop on Advances in Model-Based Testing, pp. 1–7. ACM Press, New York, NY, USA (2005)

    Chapter  Google Scholar 

  41. McDowell, A.: All-pairs testing. http://www.mcdowella.demon.co.uk/allpairs.html

  42. McMillan, K.L.: Symbolic Model Checking. Kluwer Academic Publishers, Norwell, MA, USA, ISBN 0792393805 (1993)

    MATH  Google Scholar 

  43. Nurmela, K.: Upper bounds for covering arrays by tabu. Discrete Appl. Math. 138(1–2), 143–152 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  44. Pairwise web site: http://www.pairwise.org/

  45. Seroussi, G., Bshouty, N.H.: Vector sets for exhaustive testing of logic circuits. IEEE Trans. Inf. Theory 34(3), 513–522 (1988)

    Article  MATH  MathSciNet  Google Scholar 

  46. Smith, B.D., Feather, M.S., Muscettola, N.: Challenges and methods in validating the remote agent planner. In: Breckenridge, C. (ed.) Proceedings of the Fifth International Conference on Artificial Intelligence Planning Systems (AIPS) (2000)

  47. Tai, K.C., Lie, Y.: A test generation strategy for pairwise testing. IEEE Trans. Softw. Eng. 28(1), 109–111 (2002)

    Article  Google Scholar 

  48. Williams, A.: Determination of test configurations for pair-wise interaction coverage. In: Proceedings of the 13th International Conference on the Testing of Communicating Systems (TestCom 2000), pp. 59–74 (2000)

  49. Williams, A.W., Probert, R.L.: Formulation of the interaction test coverage problem as an integer program. In: Proceedings of the 14th International Conference on the Testing of Communicating Systems (TestCom) Berlin, Germany, pp. 283–298 (2002)

  50. Yilmaz, C., Cohen, M.B., Porter, A.A.: Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Trans. Softw. Eng. 32(1), 20–34 (2006)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dip. Ingegneria Informatica e delle Telecomunicazioni, University of Catania, Catania, Italy

    Andrea Calvagna

  2. Dip. Metodi Matematici e Ingegneria dell’Informazione, University of Bergamo, Bergamo, Italy

    Angelo Gargantini

Authors
  1. Andrea Calvagna
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Angelo Gargantini
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Angelo Gargantini.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Calvagna, A., Gargantini, A. A Formal Logic Approach to Constrained Combinatorial Testing. J Autom Reasoning 45, 331–358 (2010). https://doi.org/10.1007/s10817-010-9171-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10817-010-9171-4

Keywords

Search

Navigation