Skip to main content
SpringerLink
Log in

How does combinatorial testing perform in the real world: an empirical study

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Studies have shown that combinatorial testing (CT) can be effective for detecting faults in software systems. By focusing on the interactions between different factors of a system, CT shows its potential for detecting faults, especially those that can be revealed only by the specific combinations of values of multiple factors (multi-factor faults). However, is CT practical enough to be applied in the industry? Can it be more effective than other industry-favored techniques? Are there any challenges when applying CT in practice? These research questions remain in the context of industrial settings. In this paper, we present an empirical study of CT on five industrial systems with real faults. The details of the input space model (ISM) construction, such as factor identification and value assignment, are included. We compared the faults detected by CT with those detected by the in-house testing teams using other methods, and the results suggest that despite some challenges, CT is an effective technique to detect real faults, especially multi-factor faults, of software systems in industrial settings. Observations and lessons learned are provided to further improve the fault detection effectiveness and overcome various challenges.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. Some description of case study 4 in this section and case study in section 4.5 appeared in (Li et al. 2016).

References

  • Andrews JH, Briand LC, Labiche Y 2005 May. Is mutation an appropriate tool for testing experiments? In Proceedings of the 27th international conference on Software engineering (pp. 402-411). ACM

  • “Automated combinatorial testing for software — csrc.” https://csrc.nist.gov/projects/automated-combinatorial-testing-for-software. [Online; accessed 12–25-2018]

  • Blue D, Raz O, Tzoref-Brill R, Wojciak P, Zalmanovici M (2018) “Proactive and pervasive combinatorial testing,” In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice, pp. 144–152, ACM

  • Borazjany MN, Yu L, Lei Y, Kacker R, Kuhn R (2012) “Combinatorial testing of acts: A case study,” in Software Testing, Verification and Validation (ICST), 2012 IEEE Fifth International Conference on, pp. 591–600, IEEE

  • Borazjany MN, Ghandehari LS, Lei Y, Kacker R, Kuhn R (2013) “An input space modeling methodology for combinatorial testing,” in Software Testing, Verification and Validation Workshops (ICSTW), 2013 IEEE Sixth International Conference on, pp. 372–381, IEEE

  • Bures M and Ahmed BS (2017) “On the effectiveness of combinatorial interaction testing: A case study,” in Software Quality, Reliability and Security Companion (QRS-C), 2017 IEEE International Conference on, pp. 69–76, IEEE

  • Burnstein, Practical software testing: a process-oriented approach. Springer Science & Business Media, 2006

  • Changhai N, Baowen X, Ziyuan W, Liang S (2006) “Generating optimal test set for neighbor factors combinatorial testing,” in Quality Software, 2006. QSIC 2006. Sixth international conference on, pp. 259–265, IEEE

  • Cohen M, Dalal SR, Parelius J, Patton GC (1996) The combinatorial design approach to automatic test generation. IEEE Softw 13(5):83–88

    Article  Google Scholar 

  • Cohen M, Dalal SR, Fredman ML, Patton GC (1997) The aetg system: an approach to testing based on combinatorial design. IEEE Trans Softw Eng 23(7):437–444

    Article  Google Scholar 

  • “Combinatorial interaction testing portal.” http://cse.unl.edu/~CTportal/. [Online; accessed 12–25-2018]

  • Condori-Fernández N, Vos T, Kruse PM, Brosse E, Bagnato A (2014) “Analyzing the applicability of a combinatorial testing tool in an industrial environment,” Technical Report UU-CS-2014-008

  • Dhadyalla G, Kumari N, Snell T (2014) “Combinatorial testing for an automotive hybrid electric vehicle control system: a case study,” in Software Testing, Verification and Validation Workshops (ICSTW), 2014 IEEE Seventh International Conference on, pp. 51–57, IEEE

  • Dumlu E, Yilmaz C, Cohen MB, Porter A (2011a) “Feedback driven adaptive combinatorial testing,” Proc Int’l Symp Software Testing and Analysis, pp. 243–253

  • Dumlu E, Yilmaz C, Cohen MB, Porter A (2011b) “Feedback driven adaptive combinatorial testing,” In Proceedings of the 2011 International Symposium on Software Testing and Analysis, pp. 243–253, ACM

  • Dunietz S, Ehrlich WK, Szablak B, Mallows CL, Iannino A (1997) “Applying design of experiments to software testing: experience report,” in Proceedings of the 19th international conference on Software engineering, pp. 205–215, ACM

  • Garvin BJ, Cohen MB, Dwyer MB (2011) Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empir Softw Eng 16(1):61–102

    Article  Google Scholar 

  • Ghandehari LSG, Bourazjany MN, Lei Y, Kacker RN, Kuhn DR (2013) “Applying combinatorial testing to the siemens suite,” In 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops, pp. 362–371, IEEE

  • Grindal M, Offutt J, Andler SF (2005) Combination testing strategies: a survey. Softw Test Verif Rel 15(3):167–199

    Article  Google Scholar 

  • Hagar JD, Wissink TL, Kuhn DR, Kacker RN (2015) “Introducing combinatorial testing in a large organization,” Computer, no. 4, pp. 64–72

  • Huller J (2000) “Reducing time to market with combinatorial design method testing,” in Proceedings of the 2000 international council on systems engineering (INCOSE) conference

  • Just, R., Jalali, D., Inozemtseva, L., Ernst, M.D., Holmes, R. and Fraser, G., 2014, November. Are mutants a valid substitute for real faults in software testing? In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (pp. 654-665). ACM

  • Krishnan R, Krishna SM, Nandhan PS (2007) Combinatorial testing: learnings from our experience. ACM SIGSOFT Softw Eng Notes 32(3):1–8

    Article  Google Scholar 

  • Kruse PM, Prasetya IWB, Hage J, Elyasov A (2013) “Loggingtofacilitatecombinatorial system testing,” In International Workshop on Future Internet Testing, pp. 48–58, Springer

  • Kruse PM, Shehory O, Tron DC, Fernandez NC, Vos T, Mendelson B (2014) “Assessing the applicability of a combinatorial testing tool within an industrial environment,” in 11th workshop on experimental software engineering (ESELAW 2014)

  • Kuhn R and Okum V (2006) “Pseudo-exhaustive testing for software,” in Software Engineering Workshop, 2006. SEW’06. 30th Annual IEEE/NASA, pp. 153–158, IEEE

  • R. Kuhn and M. J. Reilly, “An investigation of the applicability of design of experiments to software testing,” in Software Engineering Workshop, 2002. Proceedings. 27th Annual NASA Goddard/IEEE, pp. 91–95, IEEE, 2002

  • Kuhn R, Wallace DR, Gallo AM (2004) Software fault interactions and implications for software testing. IEEE Trans Softw Eng 30(6):418–421

    Article  Google Scholar 

  • Kuhn R, Lei Y, Kacker R (2008) “Practical combinatorial testing: Beyond pairwise,” It Professional, no. 3, pp. 19–23

  • Lei Y, Kacker R, Kuhn DR, Okun V, Lawrence J (2008) Ipog/ipog-d: efficient test generation for multi-way combinatorial testing. Softw Test Verif Rel 18(3):125–148

    Article  Google Scholar 

  • Li X, Gao R, Wong WE, Yang C, Li D (2016) “Applying Combinatorial Testing in Industrial Settings,” In Proceedings of 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS), pp. 53–60

  • Mehta M and Philip R (2013) “Applications of combinatorial testing methods for breakthrough results in software testing,” in Software Testing, Verification and Validation Workshops (ICSTW), 2013 IEEE Sixth International Conference on, pp. 348–351, IEEE

  • Patel S, Gupta P, Shah V (2013) “Combinatorial interaction testing with multi-perspective feature models,” in Software Testing, Verification and Validation Workshops (ICSTW), 2013 IEEE Sixth International Conference on, pp. 321–330, IEEE

  • Petke J, Cohen MB, Harman M, Yoo S (2015) Practical combinatorial interaction testing: empirical findings on efficiency and early fault detection. IEEE Trans Softw Eng 41(9):901–924

    Article  Google Scholar 

  • Puoskari T Vos E, Condori-Fernandez N, Kruse PM (2013) “Evaluating applicability of combinatorial testing in an industrial environment: A case study,” in Proceedings of the 2013 International Workshop on Joining AcadeMiA and Industry Contributions to testing Automation, pp. 7–12, ACM

  • Rao J, Guo N, Li Y, Lei Y, Zhang Y Li Y, Cao Y (2017) “Applying combinatorial testing to high-speed railway track circuit receiver,” in Software Testing, Verification and Validation Workshops (ICSTW), 2017 IEEE International Conference on, pp. 199–207, IEEE

  • Ratliff ZB, Kuhn DR, Kacker RN, Lei Y, Trivedi KS (2016) “The relationship between software bug type and number of factors involved in failures,” in Software Reliability Engineering Workshops (ISSREW), 2016 IEEE International Symposium on, pp. 119–124, IEEE

  • Raunak MS, Kuhn DR, Kacker R (2017) “Combinatorial testing of full text search in web applications”. In 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), pp. 100–107, IEEE

  • Reddy M (2015) “Combinatorial testing: A case study approach for software evaluation,” In Electrical, Computer and Communication Technologies (ICECCT), 2015 IEEE In- ternational Conference on, pp. 1–5, IEEE

  • Smith D, Feather MS, Muscettola N (2000) “Challenges and methods in testing the remote agent planner,” In AIPS, pp. 254–263

  • “TESTONA - Assystem Germany GmbH” – https://www.assystem-germany.com/en/products/testona/, [Online; accessed 02–22- 2019]

  • Vilkomir S and Amstutz B (2014) “Using combinatorial approaches for testing mobile applications,” in Software Testing, Verification and Validation Workshops (ICSTW), 2014 IEEE Seventh International Conference on, pp. 78–83, IEEE

  • Wang S, Wu T, Yao Y, Jin B, Ding L (2017) “Combinatorial testing on mp3 for audio players,” in Software Testing, Verification and Validation Workshops (ICSTW), 2017 IEEE International Conference on, pp. 272–275, IEEE

  • Wojciak P and Tzoref-Brill R (2014) “System level combinatorial testing in practice–the concurrent maintenance case study,” In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation (ICST), pp. 103–112, IEEE

  • Wong WE, Li X, Laplante PA (2017) Be more familiar with our enemies and pave the way forward: a review of the roles bugs played in software failures. J Syst Softw 133:68–94

    Article  Google Scholar 

  • Zhang Z, Liu X, Zhang J (2012) “Combinatorial testing on id3v2 tags of mp3 files,” in Proceedings of 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation (ICST), pp. 587–590, IEEE

Download references

Acknowledgements

This work was funded by the National Institute of Standards and Technology (Grant No. 60NANB17D322) and US National Science Foundation (Grant No. 1757828 and 1822137). The authors would like to thank their industrial associates for providing the SUTs, the testing environments, and the in-house testing data for this study: Yanzhong Gong and Dr. Jian Wang; Guokai He; Dr. Vidroha Debroy; Chunhui Yang, Dong Li, and Jun Lin. The authors would also like to express their appreciation and gratitude to Dr. Ruizhi Gao and Xuelin Li for helping with data collection.

Author information

Authors and Affiliations

  1. Computer Science Department, University of Texas at Dallas, 800 W Campbell Rd, Richardson, TX, 75080, USA

    Linghuan Hu & W. Eric Wong

  2. National Institute of Standards and Technology, Gaithersburg, MD, USA

    D. Richard Kuhn & Raghu N. Kacker

Authors
  1. Linghuan Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. W. Eric Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. D. Richard Kuhn
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Raghu N. Kacker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to W. Eric Wong.

Ethics declarations

Disclaimer

Any mention of commercial products in this paper is for information only; it does not imply recommendation or endorsement by the National Institute of Standards and Technology (NIST).

Additional information

Guest Editor: Hélène Waeselynck

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, L., Wong, W.E., Kuhn, D.R. et al. How does combinatorial testing perform in the real world: an empirical study. Empir Software Eng 25, 2661–2693 (2020). https://doi.org/10.1007/s10664-019-09799-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-019-09799-2

Keywords

Search

Navigation