Skip to main content
SpringerLink
Log in

On a Combinatorial Framework for Fault Characterization

  • Published:
Mathematics in Computer Science Aims and scope Submit manuscript

Abstract

Covering arrays have been widely used to detect the presence of faults in large software and hardware systems. Indeed, finding failures that result from faulty interactions requires that all interactions that may cause faults be covered by a test case. However, finding the actual faults requires more, because the failures resulting from two potential sets of faults must not be the same. The combinatorial requirements on test suites to enable a tester to locate the faults are developed, and set in the context of similar combinatorial search questions. Test suites known as locating and detecting arrays to locate faults both in principle and in practice generalize covering arrays, thereby addressing combinatorial fault characterization. In common with covering arrays, these locating and detecting arrays scale logarithmically in size with the number of factors, but unlike covering arrays they support complete characterization of the interactions that underlie faults.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aldaco, A.N., Colbourn, C.J., Syrotiuk, V.R.: Locating arrays: a new experimental design for screening complex engineered systems. SIGOPS Oper. Syst. Rev. 49(1), 31–40 (2015)

    Article  Google Scholar 

  2. Alon, N., Spencer, J.H.: The Probabilistic Method. Wiley-Interscience Series in Discrete Mathematics and Optimization, 3rd edn. Wiley, Hoboken (2008)

    Book  Google Scholar 

  3. Bach, J., Schroeder, P.J.: Pairwise testing: a best practice that isn’t. In: 22nd Annual Pacific Northwest Software Quality Conference, pp. 180–196 (2004)

  4. van den Berg, E., Candès, E., Chinn, G., Levin, C., Olcott, P.D., Sing-Long, C.: Single-photon sampling architecture for solid-state imaging sensors. Proc. Natl. Acad. Sci. 110(30), E2752–E2761 (2013)

    Article  Google Scholar 

  5. Bryce, R.C., Colbourn, C.J.: A density-based greedy algorithm for higher strength covering arrays. Softw. Test. Verif. Reliab. 19, 37–53 (2009)

    Article  Google Scholar 

  6. Bryce, R.C., Colbourn, C.J.: Expected time to detection of interaction faults. J. Comb. Math. Comb. Comput. 86, 87–110 (2013)

    MathSciNet  MATH  Google Scholar 

  7. Bryce, R.C., Colbourn, C.J., Kuhn, D.R.: Finding interaction faults adaptively using distance-based strategies. In: 2011 18th IEEE International Conference and Workshops on Engineering of Computer Based Systems (ECBS), pp. 4–13 (2011)

  8. Bshouty, N.H., Costa, A.: Exact learning of juntas from membership queries. Lect. Notes Artif. Intell. 9925, 115–129 (2016)

    MathSciNet  MATH  Google Scholar 

  9. Chandrasekaran, J., Ghandehari, L.S., Lei, Y., Kacker, R., Kuhn, D.R.: Evaluating the effectiveness of BEN in localizing different types of software fault. In: 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 26–34 (2016)

  10. Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)

    Article  MathSciNet  Google Scholar 

  11. Chen, T.Y., Kuo, F.C., Merkel, R.G., Tse, T.: Adaptive random testing: the art of test case diversity. J. Syst. Softw. 83(1), 60–66 (2010)

    Article  Google Scholar 

  12. Chen, T.Y., Leung, H., Mak, I.K.: Adaptive random testing. In: 9th Asian Computing Science Conference Advances in Computer Science—ASIAN 2004, Higher-Level Decision Making, pp. 320–329 (2004)

    Chapter  Google Scholar 

  13. Cheng, C.S., Tang, B.: Upper bounds on the number of columns in supersaturated designs. Biometrika 88(4), 1169–1174 (2001)

    Article  MathSciNet  Google Scholar 

  14. Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: an approach to testing based on combinatorial design. IEEE Trans. Softw. Eng. 23, 437–44 (1997)

    Article  Google Scholar 

  15. Cohen, M.B., Dwyer, M.B., Shi, J.: Constructing interaction test suites for highly-configurable systems in the presence of constraints: a greedy approach. IEEE Trans. Softw. Eng. 34, 633–650 (2008)

    Article  Google Scholar 

  16. Colbourn, C.J., Fan, B.: Locating one pairwise interaction: three recursive constructions. J. Algebra Comb. Discrete Struct. Appl. 3, 125–134 (2016)

    MathSciNet  Google Scholar 

  17. Colbourn, C.J., Fan, B., Horsley, D.: Disjoint spread systems and fault location. SIAM J. Discrete Math. 30, 2011–2016 (2016)

    Article  MathSciNet  Google Scholar 

  18. Colbourn, C.J., Lanus, E., Sarkar, K.: Asymptotic and constructive methods for covering perfect hash families and covering arrays. Designs Codes Cryptogr. 86, 907–937 (2018)

    Article  MathSciNet  Google Scholar 

  19. Colbourn, C.J., McClary, D.W.: Locating and detecting arrays for interaction faults. J. Comb. Optim. 15, 17–48 (2008)

    Article  MathSciNet  Google Scholar 

  20. Colbourn, C.J., Syrotiuk, V.R.: Coverage, location, detection, and measurement. In: 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 19–25. IEEE Press (2016)

  21. Compton, R., Mehari, M.T., Colbourn, C.J., De Poorter, E., Syrotiuk, V.R.: Screening interacting factors in a wireless network testbed using locating arrays. In: IEEE INFOCOM International Workshop on Computer and Networking Experimental Research Using Testbeds (CNERT) (2016)

  22. Dalal, S.R., Mallows, C.L.: Factor-covering designs for testing software. Technometrics 40, 234–243 (1998)

    Article  Google Scholar 

  23. Damaschke, P.: Adaptive versus nonadaptive attribute-efficient learning. Mach. Learn. 41, 197–215 (2000)

    Article  Google Scholar 

  24. Donoho, D.L., Huo, X.: Uncertainty principles and ideal atomic decomposition. IEEE Trans. Inf. Theory 47, 2845–2862 (2001)

    Article  MathSciNet  Google Scholar 

  25. Du, D.Z., Hwang, F.K.: Combinatorial Group Testing and Its Applications, 2nd edn. World Scientific Publishing Co., Inc., River Edge (2000)

    MATH  Google Scholar 

  26. Erdős, P., Lovász, L.: Problems and results on 3-chromatic hypergraphs and some related questions. In: Infinite and finite sets (Colloq., Keszthely, 1973 Vol. II, pp. 609–627. Colloq. Math. Soc. János Bolyai, Vol. 10. North-Holland, Amsterdam (1975)

  27. Francetić, N., Stevens, B.: Asymptotic size of covering arrays: an application of entropy compression. J. Comb. Des 25, 243–257 (2017)

    Article  MathSciNet  Google Scholar 

  28. Frankl, P.G., Hamlet, R.G., Littlewood, B., Strigini, L.: Evaluating testing methods by delivered reliability. IEEE Trans. Softw. Eng. 24(8), 586–601 (1998)

    Article  Google Scholar 

  29. Ghandehari, L.S., Chandrasekaran, J., Lei, Y., Kacker, R., Kuhn, D.R.: BEN: a combinatorial testing-based fault localization tool. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–4 (2015)

  30. Ghosh, S., Shirakura, T., Srivastava, J.N.: Model identification using search linear models and search designs. In: Entropy, search, complexity, Bolyai Soc. Math. Stud., vol. 16, pp. 85–112. Springer, Berlin (2007)

  31. Godbole, A.P., Skipper, D.E., Sunley, R.A.: \(t\)-covering arrays: upper bounds and Poisson approximations. Comb. Probab. Comput. 5, 105–118 (1996)

    Article  MathSciNet  Google Scholar 

  32. Grindal, M., Offutt, J., Andler, S.F.: Combination testing strategies—a survey. Softw. Test. Verif. Reliab. 5, 167–199 (2005)

    Article  Google Scholar 

  33. Hamlet, D., Taylor, R.: Partition testing does not inspire confidence (program testing). IEEE Trans. Softw. Eng. 16(12), 1402–1411 (1990)

    Article  Google Scholar 

  34. Hartman, A.: Software and hardware testing using combinatorial covering suites. In: Golumbic, M.C., Hartman, I.B.A. (eds.) Interdisciplinary Applications of Graph Theory, Combinatorics, and Algorithms, pp. 237–266. Springer, Norwell (2005)

    Chapter  Google Scholar 

  35. Hartman, A., Raskin, L.: Problems and algorithms for covering arrays. Discrete Math. 284, 149–156 (2004)

    Article  MathSciNet  Google Scholar 

  36. Jin, H., Tsuchiya, T.: Constrained locating arrays for combinatorial interaction testing. CoRR arXiv:1801.06041 (2018)

  37. Johnson, D.S.: Approximation algorithms for combinatorial problems. J. Comput. Syst. Sci. 9, 256–278 (1974)

    Article  MathSciNet  Google Scholar 

  38. Jones, B., Majumdar, D.: Optimal supersaturated designs. J. Am. Stat. Assoc. 109(508), 1592–1600 (2014)

    Article  MathSciNet  Google Scholar 

  39. Konishi, T., Kojima, H., Nakagawa, H., Tsuchiya, T.: Finding minimum locating arrays using a SAT solver. In: 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops, ICST Workshops 2017, Tokyo, Japan, March 13–17, 2017, pp. 276–277 (2017)

  40. Kuhn, D.R., Bryce, R.C., Duan, F., Ghandehari, L.S.G., Lei, Y., Kacker, R.N.: Combinatorial testing: theory and practice. Adv. Comput. 99, 1–66 (2015)

    Article  Google Scholar 

  41. Kuhn, D.R., Kacker, R., Lei, Y.: Introduction to Combinatorial Testing. CRC Press, Boca Raton (2013)

    MATH  Google Scholar 

  42. Kuhn, D.R., Kacker, R., Lei, Y.: Combinatorial coverage as an aspect of test quality. Crosstalk pp. 19–23 (2015)

  43. Kuhn, D.R., Mendoza, I.D., Kacker, R., Lei, Y.: Combinatorial coverage measurement concepts and applications. In: Sixth IEEE International Conference on Software Testing, Verification and Validation, ICST 2013 Workshops Proceedings, Luxembourg, Luxembourg, March 18–22, 2013, pp. 352–361 (2013)

  44. Kuhn, D.R., Reilly, M.: An investigation of the applicability of design of experiments to software testing. In: Proceedings of 27th Annual NASA Goddard/IEEE Software Engineering Workshop, pp. 91–95. IEEE, Los Alamitos, CA (2002)

  45. Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng. 30, 418–421 (2004)

    Article  Google Scholar 

  46. Li, P.C., Meagher, K.: Sperner partition systems. J. Comb. Des. 21(7), 267–279 (2013)

    Article  MathSciNet  Google Scholar 

  47. Lovász, L.: On the ratio of optimal integral and fractional covers. Discrete Math. 13(4), 383–390 (1975)

    Article  MathSciNet  Google Scholar 

  48. Martínez, C., Moura, L., Panario, D., Stevens, B.: Locating errors using ELAs, covering arrays, and adaptive testing algorithms. SIAM J. Discrete Math. 23, 1776–1799 (2009-2010)

    Article  MathSciNet  Google Scholar 

  49. Meagher, K., Moura, L., Stevens, B.: A Sperner-type theorem for set-partition systems. Electron. J. Combin. 12, Note 20 (electronic) (2005)

  50. Montgomery, D.C.: Design and Analysis of Experiments, 8th edn. Wiley, New York (2012)

    Google Scholar 

  51. Moser, R.A., Tardos, G.: A constructive proof of the general Lovász local lemma. J. ACM 57(2), Art. 11, 15 (2010)

    Article  MathSciNet  Google Scholar 

  52. Nagamoto, T., Kojima, H., Nakagawa, H., Tsuchiya, T.: Locating a faulty interaction in pair-wise testing. In: 20th IEEE Pacific Rim International Symposium on Dependable Computing, PRDC 2014, Singapore, November 18–21, 2014, pp. 155–156 (2014)

  53. Nayeri, P., Colbourn, C.J., Konjevod, G.: Randomized postoptimization of covering arrays. Eur. J. Comb. 34, 91–103 (2013)

    Article  Google Scholar 

  54. Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. 43(2), #11 (2011)

    Article  Google Scholar 

  55. Nie, C., Wu, H., Niu, X., Kuo, F., Leung, H.K.N., Colbourn, C.J.: Combinatorial testing, random testing, and adaptive random testing for detecting interaction triggered failures. Inf. Softw. Technol. 62, 198–213 (2015)

    Article  Google Scholar 

  56. Petke, J.: Constraints: The future of combinatorial interaction testing. In: 2015 IEEE/ACM 8th International Workshop on Search-Based Software Testing (SBST), pp. 17–18 (2015)

  57. Petke, J., Cohen, M.B., Harman, M., Yoo, S.: Practical combinatorial interaction testing: empirical findings on efficiency and early fault detection. IEEE Trans. Softw. Eng. 41(9), 901–924 (2015)

    Article  Google Scholar 

  58. Sarkar, K., Colbourn, C.J.: Upper bounds on the size of covering arrays. SIAM J. Discrete Math. 31, 1277–1293 (2017)

    Article  MathSciNet  Google Scholar 

  59. Schroeder, P.J., Bolaki, P., Gopu, V.: Comparing the fault detection effectiveness of n-way and random test suites. In: Proceedings of International Symposium on Empirical Software Engineering (ISESE04), pp. 49–59 (2004)

  60. Seidel, S.A., Mehari, M.T., Colbourn, C.J., De Poorter, E., Moerman, I., Syrotiuk, V.R.: Analysis of large-scale experimental data from wireless networks. In: IEEE INFOCOM International Workshop on Computer and Networking Experimental Research Using Testbeds (CNERT) (2018)

  61. Seidel, S.A., Sarkar, K., Colbourn, C.J., Syrotiuk, V.R.: Separating interaction effects using locating and detecting arrays. In: International Workshop on Combinatorial Algorithms (2018)

    Google Scholar 

  62. Shi, C., Tang, Y., Yin, J.: Optimal locating arrays for at most two faults. Sci. China Math. 55(1), 197–206 (2012)

    Article  MathSciNet  Google Scholar 

  63. Stein, S.K.: Two combinatorial covering theorems. J. Comb. Theory Ser. A 16, 391–397 (1974)

    Article  MathSciNet  Google Scholar 

  64. Tai, K.C., Yu, L.: A test generation strategy for pairwise testing. IEEE Trans. Softw. Eng. 28, 109–111 (2002)

    Article  Google Scholar 

  65. Tang, Y., Colbourn, C.J., Yin, J.: Optimality and constructions of locating arrays. J. Stat. Theory Pract. 6(1), 20–29 (2012)

    Article  MathSciNet  Google Scholar 

  66. Torres-Jimenez, J., Rodriguez-Tello, E.: New upper bounds for binary covering arrays using simulated annealing. Inf. Sci. 185(1), 137–152 (2012)

    Article  Google Scholar 

  67. Yilmaz, C., Cohen, M.B., Porter, A.: Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Trans. Softw. Eng. 31, 20–34 (2006)

    Article  Google Scholar 

  68. Yilmaz, C., Dumlu, E., Cohen, M.B., Porter, A.: Reducing masking effects in combinatorial interaction testing: A feedback driven adaptive approach. IEEE Trans. Softw. Eng. 40(1), 43–66 (2014)

    Article  Google Scholar 

  69. Zhang, Y.: On theory of compressive sensing via \(\ell _1\)-minimization: Simple derivations and extensions. Tech. Rep. CAAM TR08-11, Rice University (2008)

  70. Zhou, W., Zhang, D.: Sole error locating array and approximate error locating array. In: 2012 IEEE International Conference on Computer Science and Automation Engineering, pp. 480–483 (2012)

Download references

Author information

Authors and Affiliations

  1. Computing, Informatics, and Decision Systems Engineering, Arizona State University, PO Box 878809, Tempe, AZ, 85287-8809, USA

    Charles J. Colbourn & Violet R. Syrotiuk

Authors
  1. Charles J. Colbourn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Violet R. Syrotiuk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Charles J. Colbourn.

Additional information

Research of CJC was supported in part by the Software Test & Analysis Techniques for Automated Software Test program supported by OPNAV N-84, United States Navy. Research of CJC and VRS was supported in part by the National Science Foundation under Grant No. 1421058.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Colbourn, C.J., Syrotiuk, V.R. On a Combinatorial Framework for Fault Characterization. Math.Comput.Sci. 12, 429–451 (2018). https://doi.org/10.1007/s11786-018-0385-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11786-018-0385-x

Keywords

Mathematics Subject Classification

Search

Navigation