Skip to main content
SpringerLink
Log in

An approach for experimentally evaluating effectiveness and efficiency of coverage criteria for software testing

  • TACAS 06
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

Experimental work in software testing has generally focused on evaluating the effectiveness and effort requirements of various coverage criteria. The important issue of testing efficiency has not been sufficiently addressed. In this paper, we describe an approach for comparing the effectiveness and efficiency of test coverage criteria using mutation analysis. For each coverage criterion under study, we generate multiple coverage-adequate minimal test suites for a test-program from a test-pool, which are then executed on a set of systematically generated program mutants to obtain the fault data. We demonstrate the applicability of the proposed approach by describing the results of an experiment comparing the three code-based testing criteria, namely, block coverage, branch coverage, and predicate coverage. Our results suggest that there is a trade-off between effectiveness and efficiency of a coverage criterion. Specifically, the predicate coverage criterion was found to be most effective but least efficient whereas the block coverage criterion was most efficient but least effective. We observed high variability in the performance of block test suites whereas branch and predicate test suites were relatively consistent. Overall results suggest that the branch coverage criterion performs consistently with good efficiency and effectiveness, and it appears to be the most viable option for code-based control flow testing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Andrews, J.H., Briand, L.C., Labiche, Y.: Is mutation an appropriate tool for testing experiments? In: ICSE ’05: Proceedings of the 27th International Conference on Software Engineering, pp. 402–411. ACM Press, New York (2005)

  2. Antoniol, G., Briand, L.C., Penta, M.D., Labiche, Y.: A case study using the round-trip strategy for state-based class testing. In: ISSRE ’02: Proceedings of the 13th International Symposium on Software Reliability Engineering, pp. 269–279. IEEE Computer Society, Washington (2002)

  3. Basili, V.R.: The role of experimentation in software engineering: past, current, and future. In: ICSE ’96: Proceedings of the 18th International Conference on Software Engineering, pp. 442–449. IEEE Computer Society, Washington (1996)

  4. Basili, V.R., Caldeira, G., Rombach, H.D.: The goal question metric approach. In: Encyclopedia of Software Engineering (1994)

  5. Basili V.R. and Selby R.W. (1987). Comparing the effectiveness of software testing strategies. IEEE Trans. Softw. Eng. 13(12): 1278–1296

    Article  Google Scholar 

  6. Basili V.R., Selby R.W. and Hutchens D.H. (1986). Experimentation in software engineering. IEEE Trans. Softw. Eng. 12(7): 733–743

    Google Scholar 

  7. BCEL Homepage. http://jakarta.apache.org/bcel/

  8. Briand L.C. and Labiche Y. (2004). Empirical studies of software testing techniques: challenges, practical strategies and future research. SIGSOFT Softw. Eng. Notes 29(5): 1–3

    Article  Google Scholar 

  9. Briand, L.C., Labiche, Y., Wang, Y.: Using simulation to empirically investigate test coverage criteria based on statechart. In: ICSE ’04: Proceedings of the 26th International Conference on Software Engineering, pp. 86–95. IEEE Computer Society, Washington (2004)

  10. Chang, J., Richardson, D.J., Sankar, S.: Structural specification-based testing with adl. In: ISSTA, pp. 62–70 (1996)

  11. CruiseControl. http://www-dse.doc.ic.ac.uk/concurrency/book

  12. DeMillo R.A., Lipton R.J. and Sayward F.G. (1978). Hints on test data selection: Help for the practicing programmer. IEEE Computer 11(4): 34–41

    Google Scholar 

  13. Dick, J., Faivre, A.: Automating the generation and sequencing of test cases from model-based specifications. In: FME, pp. 268–284 (1993)

  14. Do, H., Elbaum, S., Rothermel, G.: Infrastructure support for controlled experimentation with software testing and regression testing techniques. In: ISESE ’04: Proceedings of the 2004 International Symposium on Empirical Software Engineering, pp. 60–70. IEEE Computer Society, Washington (2004)

  15. Fowler M. (1999). Refactoring: Improving the Design of Existing Code. Addison-Wesley, Reading

    Google Scholar 

  16. Frankl, P.G., Deng, Y.: Comparison of delivered reliability of branch, data flow and operational testing: a case study. In: ISSTA ’00: Proceedings of the 2000 ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 124–134. ACM Press, New York (2000)

  17. Frankl P.G. and Weiss S.N. (1993). An experimental comparison of the effectiveness of branch testing and data flow testing. IEEE Trans Softw. Eng. 19(8): 774–787

    Article  Google Scholar 

  18. Frankl P.G. and Weyuker E.J. (1993). A formal analysis of the fault-detecting ability of testing methods. IEEE Trans Softw. Eng. 19(3): 202–213

    Article  Google Scholar 

  19. Frankl P.G. and Weyuker E.J. (1993). Provable improvements on branch testing. IEEE Trans. Softw. Eng. 19(10): 962–975

    Article  MathSciNet  Google Scholar 

  20. Goodenough J.B. and Gerhart S.L. (1975). Toward a theory of test data selection. IEEE Trans. Software Eng. 1(2): 156–173

    MathSciNet  Google Scholar 

  21. Grieskamp, W., Gurevich, Y., Schulte, W., Veanes, M.: Generating finite state machines from abstract state machines. In: ISSTA ’02: Proceedings of the 2002 ACM SIGSOFT International Symposium on software Testing and Analysis, pp. 112–122. ACM Press, New York (2002)

  22. Hamlet, R.: Theoretical comparison of testing methods. In: TAV3: Proceedings of the ACM SIGSOFT ’89 3rd Symposium on Software Testing, Analysis, and Verification, pp. 28–37. ACM Press, New York (1989)

  23. Höst, M., Wohlin, C., Thelin, T.: Experimental context classification: incentives and experience of subjects. In: ICSE ’05: Proceedings of the 27th International Conference on Software Engineering, pp. 470–478 (2005)

  24. Hutchins, M., Foster, H., Goradia, T., Ostrand T.: Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In ICSE ’94: Proceedings of the 16th International Conference on Software Engineering, pp. 191–200. IEEE Computer Society Press, Los Alamitos (1994)

  25. JavaCodeCoverage. http://atulkg.googlepages.com/JavaCodeCoverage/

  26. JUnit Home Page. http://www.junit.org

  27. Juristo N. and Moreno A.M. (2001). Basics of Software Engineering Experimentation. Springer, Heidelberg

    MATH  Google Scholar 

  28. Juristo N., Moreno A.M. and Vegas S. (2004). Reviewing 25 years of testing technique experiments. Empirical Softw. Engg. 9(1–2): 7–44

    Article  Google Scholar 

  29. Juristo N., Moreno A.M. and Vegas S. (2004). Towards building a solid empirical body of knowledge in testing techniques. SIGSOFT Softw. Eng. Notes 29(5): 1–4

    Article  Google Scholar 

  30. Juristo N., Moreno A.M., Vegas S. and Solari M. (2006). In search of what we experimentally know about unit testing. IEEE Softw. 23(6): 72–80

    Article  Google Scholar 

  31. Kamsties, E., Lott, C.M.: An empirical evaluation of three defect-detection techniques. In: ESEC ’95: Proceedings of the 5th European Software Engineering Conference, pp. 362–383. Springer, London (1995)

  32. Kapoor K. and Bowen J.P. (2004). Experimental evaluation of the tolerance for control-flow test criteria: Research articles. Softw. Test. Verif. Reliab. 14(3): 167–187

    Article  Google Scholar 

  33. Kim, S., Clark, J., McDermid, J.: The rigorous generation of java mutation operators using hazop. In: Proceedings of ICSSEA, pp. 9–10 (1999)

  34. Kitchenham B.A., Pfleeger S.L., Pickard L.M., Jones P.W., Hoaglin D.C., Emam K.E. and Rosenberg J. (2002). Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Eng. 28(8): 721–734

    Article  Google Scholar 

  35. Lingampally, R., Gupta, A., Jalote, P.: A multipurpose code coverage tool for java. In: HICSS ’07: Proceedings of 40th Annual Hawaii International Conference on System Sciences; minitrack on Automated Software Testing and Analysis: Techniques, Practices and Tools, pp. 261–270. IEEE Computer Society, Los Alamitos (2007)

  36. Ma, Y.-S., Offutt, J., Kwon, Y.-R.: Mujava: a mutation system for java. In: ICSE ’06: Proceeding of the 28th International Conference on Software Engineering, pp. 827–830. ACM Press, New York (2006)

  37. Miller, J.: Can results from software engineering experiments be safely combined? In: METRICS ’99: Proceedings of the 6th International Symposium on Software Metrics, pp. 152–158. IEEE Computer Society, Washington (1999)

  38. Monopoly - RealEstate. http://open.ncsu.edu/se/monopoly/

  39. Montgomery D.C. (2001). Design and Analysis of Experiments. Wiley, London

    Google Scholar 

  40. MuJava Home Page. http://www.ise.gmu.edu/~ofut/mujava/

  41. Murray, L., Carrington, D.A., MacColl, I., McDonald, J., Strooper, P.A.: Formal derivation of finite state machines for class testing. In: ZUM, pp. 42–59 (1998)

  42. MySQL Home Page. http://www.mysql.com/

  43. Ntafos, S.C.: An evaluation of required element testing strategies. In: ICSE ’84: Proceedings of the 7th International Conference on Software Engineering, pp. 250–256. IEEE Press, Piscataway (1984)

  44. Offutt A.J. (1992). Investigations of the software testing coupling effect. ACM Trans. Softw. Eng. Methodol. 1(1): 5–20

    Article  Google Scholar 

  45. Offutt, A.J., Xiong, Y., Liu, S.: Criteria for generating specification-based tests. In: ICECCS ’99 : 5th International Conference on Engineering of Complex Computer Systems, pp. 119–129. IEEE Computer Society, Washington (1999)

  46. Shull, F., Basili, V., Carver, J., Maldonado, J.C., Travassos, G.H., Mendonça, M., Fabbri, S.: Replicating software engineering experiments: Addressing the tacit knowledge problem. In: ISESE ’02: Proceedings of the 2002 International Symposium on Empirical Software Engineering, pp. 7–16. IEEE Computer Society, Washington (2002)

  47. Sjoberg D.I.K., Hannay J.E., Hansen O., Kampenes V.B., Karahasanovic A., Liborg N.-K. and Rekdal A.C. (2005). A survey of controlled experiments in software engineering. IEEE Trans. Softw. Eng. 31(9): 733–753

    Article  Google Scholar 

  48. Stocks P. and Carrington D.A. (1996). A framework for specification-based testing. IEEE Trans. Softw. Eng. 22(11): 777–793

    Article  Google Scholar 

  49. Weyuker, E.: Can we measure software testing effectiveness? In: Metrics ’93: Proceedings of 1st International Symposium on Software Metrics, pp. 100–107. IEEE Computer Society, Washington (1993)

  50. Weyuker, E.J., Weiss, S.N., Hamlet, D.: Comparison of program testing strategies. In: TAV4: Proceedings of the Symposium on Testing, Analysis, and Verification, pp. 1–10. ACM Press, New York (1991)

  51. Wohlin C., Runeson P., Host M., Ohlsson M.C., Regnell B. and Wesslen A. (2000). Experimentation in Software Engineering—An Introduction. Kluwer, Dordrecht

    MATH  Google Scholar 

  52. Wood, M., Roper, M., Brooks, A., Miller, J.: Comparing and combining software defect detection techniques: a replicated empirical study. In: ESEC ’97/FSE-5: Proceedings of the 6th European Conference/5th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 262–277. Springer, New York (1997)

  53. Zhu H., Hall P.A.V. and May J.H.R. (1997). Software unit test coverage and adequacy. ACM Comput. Surv. 29(4): 366–427

    Article  Google Scholar 

Download references

Author information

Author notes

  1. Pankaj Jalote

    Present address: Indian Institute of Technology Delhi, Delhi, 110016, India

Authors and Affiliations

  1. Indian Institute of Technology, Kanpur, 208016, India

    Atul Gupta & Pankaj Jalote

Authors
  1. Atul Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pankaj Jalote
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Atul Gupta.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gupta, A., Jalote, P. An approach for experimentally evaluating effectiveness and efficiency of coverage criteria for software testing. Int J Softw Tools Technol Transf 10, 145–160 (2008). https://doi.org/10.1007/s10009-007-0059-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-007-0059-5

Keywords

Search

Navigation