Skip to main content
SpringerLink
Log in

Prioritizing test cases with string distances

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

Test case prioritisation aims at finding an ordering which enhances a certain property of an ordered test suite. Traditional techniques rely on the availability of code or a specification of the program under test. We propose to use string distances on the text of test cases for their comparison and elaborate a prioritisation algorithm. Such a prioritisation does not require code or a specification and can be useful for initial testing and in cases when code is difficult to instrument. In this paper, we also report on experiments performed on the “Siemens Test Suite”, where the proposed prioritisation technique was compared with random permutations and four classical string distance metrics were evaluated. The obtained results, confirmed by a statistical analysis, indicate that prioritisation based on string distances is more efficient in finding defects than random ordering of the test suite: the test suites prioritized using string distances are more efficient in detecting the strongest mutants, and, on average, have a better APFD than randomly ordered test suites. The results suggest that string distances can be used for prioritisation purposes, and Manhattan distance could be the best choice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • Braun, H., Tukey, J.: The Collected Works of John W. Tukey: Multiple Comparisons, 1948–1983, vol. 8. Chapman & Hall, London (1994)

    Google Scholar 

  • Chan, F.T., Chen, T.Y., Mak, I.K., Yu, Y.T.: Proportional sampling strategy: guidelines for software testing practitioners. Inf. Softw. Technol. 38(12), 775–782 (1996)

    Article  Google Scholar 

  • Charikar, M., Hajiaghayi, M.T., Karloff, H.J., Rao, S.: \(l^{2}_{2}\) spreading metrics for vertex ordering problems. In: Proceedings of the Seventeenth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA 2006), Miami, Florida, USA, 22–26 January 2006, pp. 1018–1027. ACM Press, New York (2006)

    Chapter  Google Scholar 

  • Chen, T.Y., Leung, H., Mak, I.K.: Adaptive random testing. In: ASIAN 2004. Lecture Notes in Computer Science, vol. 3321, pp. 320–329. Springer, Berlin (2004)

    Google Scholar 

  • Cohen, W.W., Ravikumar, P., Fienberg, S.E.: A comparison of string distance metrics for name-matching tasks. In: Proc. of IJCAI-03 Workshop on Information Integration on the Web (IIWeb-03), Acapulco, Mexico, pp. 73–78 (2003)

    Google Scholar 

  • Do, H., Elbaum, S.G., Rothermel, G.: Supporting controlled experimentation with testing techniques: an infrastructure and its potential impact. Empir. Softw. Eng. 10(4), 405–435 (2005)

    Article  Google Scholar 

  • Elbaum, S.G., Malishevsky, A.G., Rothermel, G.: Test case prioritization: a family of empirical studies. IEEE Trans. Softw. Eng. 28(2), 159–182 (2002)

    Article  Google Scholar 

  • Feijs, L.M.G., Goga, N., Mauw, S., Tretmans, J.: Test selection, trace distance and heuristics. In: Schieferdecker, I., König, H., Wolisz, A. (eds.) Testing of Communicating Systems XIV, Applications to Internet Technologies and Services, Proceedings of the IFIP 14th International Conference on Testing Communicating Systems (TestCom 2002), Berlin, Germany, 19–22 March 2002, vol. 210, pp. 267–282. Kluwer Academic, Norwell (2002)

    Google Scholar 

  • Hamming, R.W.: Error-detecting and error-correcting codes. Bell Syst. Tech. J. 29(2), 147–160 (1950)

    MathSciNet  Google Scholar 

  • Harrold, M.J., Gupta, R., Soffa, M.L.: A methodology for controlling the size of a test suite. ACM Transactions Software Engineering. Methodology 2(3), 270–285 (1993)

    Google Scholar 

  • Heimdahl, M.P.E., Devaraj, G.: On the effect of test-suite reduction on automatically generated model-based tests. Autom. Softw. Eng. 14(1), 37–57 (2007)

    Article  Google Scholar 

  • Hemmati, H., Arcuri, A., Briand, L.: Reducing the cost of model-based testing through test case diversity. In: Petrenko, A., da Silva Simão, A., Maldonado, J.C. (eds.) Proceedings of 22nd IFIP Int. Conf. on Testing Software and Systems (ICTSS 2010). Lecture Notes in Computer Science, vol. 6435, pp. 63–78. Springer, Berlin (2010)

    Google Scholar 

  • Hennessy, M., Power, J.F.: An analysis of rule coverage as a criterion in generating minimal test suites for grammar-based software. In: 20th IEEE/ACM Int. Conf. on Automated Software Engineering (ASE 2005). ACM, New York (2005)

    Google Scholar 

  • Hutchins, M., Foster, H., Goradia, T., Ostrand, T.J.: Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In: 16th Int. Conf. on Software Engineering (ICSE’94), pp. 191–200 (1994)

    Chapter  Google Scholar 

  • Jiang, B., Zhang, Z., Chan, W.K., Tse, T.H.: Adaptive random test case prioritization. In: 24th IEEE/ACM International Conference on Automated Software Engineering (ASE 2009), Auckland, New Zealand, 16–20 November 2009, pp. 233–244. IEEE Comput. Soc., Los Alamitos (2009)

    Chapter  Google Scholar 

  • Jones, J.A., Harrold, M.J.: Test-suite reduction and prioritization for modified condition/decision coverage. IEEE Trans. Softw. Eng. 29(3), 195–209 (2003)

    Article  Google Scholar 

  • Kessis, M., Ledru, Y., Vandome, G.: Experiences in coverage testing of a Java middleware. In: 5th Int. Workshop on Software Engineering and Middleware. ACM, New York (2005)

    Google Scholar 

  • Kovács, G., Németh, G.Á., Subramaniam, M., Pap, Z.: Optimal string edit distance based test suite reduction for SDL specifications. In: Reed, R., Bilgic, A., Gotzhein, R. (eds.) Design for Motes and Mobiles, Proceedings of 14th International SDL Forum (SDL 2009), Bochum, Germany, 22–24 September 2009. Lecture Notes in Computer Science, vol. 5719, pp. 82–97. Springer, Berlin (2009)

    Chapter  Google Scholar 

  • Ledru, Y., Petrenko, A., Boroday, S.: Using string distances for test case prioritisation. In: 24th IEEE/ACM Int. Conf. on Automated Software Engineering (ASE 2009), Short paper, pp. 510–514. IEEE Press, New York (2009)

    Chapter  Google Scholar 

  • Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals (in Russian). Dokl. Akad. Nauk SSSR 163(4), 845–848 (1965)

    MathSciNet  Google Scholar 

  • Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals (English translation). Sov. Phys. Dokl. 10(8), 707–710 (1966)

    MathSciNet  Google Scholar 

  • Li, Z., Harman, M., Hierons, R.M.: Search algorithms for regression test case prioritization. IEEE Trans. Softw. Eng. 33(4), 225–237 (2007)

    Article  Google Scholar 

  • Malaiya, Y.: Antirandom testing: getting the most out of black-box testing. In: 6th Int. Symp. on Software Reliability Engineering. IEEE Press, New York (1995)

    Google Scholar 

  • Mayer, J., Schneckenburger, C.: An empirical analysis and comparison of random testing techniques. In: Int. Symp. on Empirical Software Engineering (ISESE 2006). ACM Press, New York (2006)

    Google Scholar 

  • Qu, X., Cohen, M.B., Rothermel, G.: Configuration-aware regression testing: an empirical study of sampling and prioritization. In: Ryder, B.G., Zeller, A. (eds.) Proceedings of the ACM/SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2008), Seattle, WA, USA, 20–24 July 2008, pp. 75–86. ACM Press, New York (2008)

    Chapter  Google Scholar 

  • Ramanathan, M.K., Koyutürk, M., Grama, A., Jagannathan, S.: Phalanx: a graph-theoretic framework for test case prioritization. In: Wainwright, R.L., Haddad, H. (eds.) Proceedings of the 2008 ACM Symposium on Applied Computing (SAC), Fortaleza, Ceara, Brazil, 16–20 March 2008, pp. 667–673. ACM Press, New York (2008)

    Chapter  Google Scholar 

  • Rothermel, G., Harrold, M.J., Ostrin, J., Hong, C.: An empirical study of the effects of minimization on the fault detection capabilities of test suites. In: Int. Conf. on Software Maintenance, pp. 34–43. IEEE Press, New York (1998)

    Google Scholar 

  • Rothermel, G., Untch, R.H., Chu, C., Harrold, M.J.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 27(10), 929–948 (2001)

    Article  Google Scholar 

  • Smith, A.M., Geiger, J., Kapfhammer, G.M., Soffa, M.L.: Test suite reduction and prioritization with call trees. In: 22nd IEEE/ACM Int. Conf. on Automated Software Engineering (ASE 2007), pp. 539–540. ACM Press, New York (2007)

    Google Scholar 

  • Snedecor, G., Cochran, W.: Statistical Methods, 6th edn. Iowa State University Press, Ames (1957)

    Google Scholar 

  • Srikanth, H., Williams, L., Osborne, J.: System test case prioritization of new and regression test cases. In: International Symposium on Empirical Software Engineering. IEEE Comput. Soc., Los Alamitos (2005)

    Google Scholar 

  • Srivastava, A., Thiagarajan, J.: Effectively prioritizing tests in development environment. In: ACM/SIGSOFT Int. Symp. on Software Testing and Analysis (ISSTA 2002). ACM, New York (2002)

    Google Scholar 

  • Vuong, S.T., Alilovic-Curgus, J.: On test coverage metrics for communication protocols. In: Kroon, J., Heijink, R.J., Brinksma, E. (eds.) Protocol Test Systems, IV, Proceedings of the IFIP TC6/WG6.1 Fourth International Workshop on Protocol Test Systems, Leidschendam, The Netherlands, 15–17 October 1991. IFIP Transactions, vol. C-3, pp. 31–45. North-Holland, Amsterdam (1992)

    Google Scholar 

  • Walcott, K.R., Soffa, M.L., Kapfhammer, G.M., Roos, R.S.: Timeaware test suite prioritization. In: ACM/SIGSOFT Int. Symp. on Software Testing and Analysis. ACM Press, New York (2006)

    Google Scholar 

  • Wong, W.E., Horgan, J.R., London, S., Agrawal, H.: A study of effective regression testing in practice. In: 8th Int. Symp. on Software Reliability Engineering. IEEE Press, New York (1997)

    Google Scholar 

  • Wong, W.E., Horgan, J.R., Mathur, A.P., Pasquini, A.: Test set size minimization and fault detection effectiveness: a case study in a space application. J. Syst. Softw. 48, 79–89 (1999)

    Article  Google Scholar 

  • Yin, H., Lebne-Dengel, Z., Malaiya, Y.K.: Automatic test generation using checkpoint encoding and antirandom testing. In: 8th Int. Symp. on Software Reliability Engineering, pp. 84–95. IEEE Press, New York (1997)

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Laboratoire d’Informatique de Grenoble (LIG), UMR 5217, UJF-Grenoble 1, Grenoble-INP, UPMF-Grenoble 2, CNRS, 38041, Grenoble, France

    Yves Ledru & Nadine Mandran

  2. Centre de Recherche Informatique de Montréal, 405 Ogilvy Avenue, Suite 101, Montréal, Québec, H3N 1M3, Canada

    Alexandre Petrenko & Sergiy Boroday

Authors
  1. Yves Ledru
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexandre Petrenko
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sergiy Boroday
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nadine Mandran
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yves Ledru.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ledru, Y., Petrenko, A., Boroday, S. et al. Prioritizing test cases with string distances. Autom Softw Eng 19, 65–95 (2012). https://doi.org/10.1007/s10515-011-0093-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10515-011-0093-0

Keywords

Search

Navigation