Abstract
Test case prioritisation aims at finding an ordering which enhances a certain property of an ordered test suite. Traditional techniques rely on the availability of code or a specification of the program under test. We propose to use string distances on the text of test cases for their comparison and elaborate a prioritisation algorithm. Such a prioritisation does not require code or a specification and can be useful for initial testing and in cases when code is difficult to instrument. In this paper, we also report on experiments performed on the “Siemens Test Suite”, where the proposed prioritisation technique was compared with random permutations and four classical string distance metrics were evaluated. The obtained results, confirmed by a statistical analysis, indicate that prioritisation based on string distances is more efficient in finding defects than random ordering of the test suite: the test suites prioritized using string distances are more efficient in detecting the strongest mutants, and, on average, have a better APFD than randomly ordered test suites. The results suggest that string distances can be used for prioritisation purposes, and Manhattan distance could be the best choice.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Braun, H., Tukey, J.: The Collected Works of John W. Tukey: Multiple Comparisons, 1948–1983, vol. 8. Chapman & Hall, London (1994)
Chan, F.T., Chen, T.Y., Mak, I.K., Yu, Y.T.: Proportional sampling strategy: guidelines for software testing practitioners. Inf. Softw. Technol. 38(12), 775–782 (1996)
Charikar, M., Hajiaghayi, M.T., Karloff, H.J., Rao, S.: \(l^{2}_{2}\) spreading metrics for vertex ordering problems. In: Proceedings of the Seventeenth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA 2006), Miami, Florida, USA, 22–26 January 2006, pp. 1018–1027. ACM Press, New York (2006)
Chen, T.Y., Leung, H., Mak, I.K.: Adaptive random testing. In: ASIAN 2004. Lecture Notes in Computer Science, vol. 3321, pp. 320–329. Springer, Berlin (2004)
Cohen, W.W., Ravikumar, P., Fienberg, S.E.: A comparison of string distance metrics for name-matching tasks. In: Proc. of IJCAI-03 Workshop on Information Integration on the Web (IIWeb-03), Acapulco, Mexico, pp. 73–78 (2003)
Do, H., Elbaum, S.G., Rothermel, G.: Supporting controlled experimentation with testing techniques: an infrastructure and its potential impact. Empir. Softw. Eng. 10(4), 405–435 (2005)
Elbaum, S.G., Malishevsky, A.G., Rothermel, G.: Test case prioritization: a family of empirical studies. IEEE Trans. Softw. Eng. 28(2), 159–182 (2002)
Feijs, L.M.G., Goga, N., Mauw, S., Tretmans, J.: Test selection, trace distance and heuristics. In: Schieferdecker, I., König, H., Wolisz, A. (eds.) Testing of Communicating Systems XIV, Applications to Internet Technologies and Services, Proceedings of the IFIP 14th International Conference on Testing Communicating Systems (TestCom 2002), Berlin, Germany, 19–22 March 2002, vol. 210, pp. 267–282. Kluwer Academic, Norwell (2002)
Hamming, R.W.: Error-detecting and error-correcting codes. Bell Syst. Tech. J. 29(2), 147–160 (1950)
Harrold, M.J., Gupta, R., Soffa, M.L.: A methodology for controlling the size of a test suite. ACM Transactions Software Engineering. Methodology 2(3), 270–285 (1993)
Heimdahl, M.P.E., Devaraj, G.: On the effect of test-suite reduction on automatically generated model-based tests. Autom. Softw. Eng. 14(1), 37–57 (2007)
Hemmati, H., Arcuri, A., Briand, L.: Reducing the cost of model-based testing through test case diversity. In: Petrenko, A., da Silva Simão, A., Maldonado, J.C. (eds.) Proceedings of 22nd IFIP Int. Conf. on Testing Software and Systems (ICTSS 2010). Lecture Notes in Computer Science, vol. 6435, pp. 63–78. Springer, Berlin (2010)
Hennessy, M., Power, J.F.: An analysis of rule coverage as a criterion in generating minimal test suites for grammar-based software. In: 20th IEEE/ACM Int. Conf. on Automated Software Engineering (ASE 2005). ACM, New York (2005)
Hutchins, M., Foster, H., Goradia, T., Ostrand, T.J.: Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In: 16th Int. Conf. on Software Engineering (ICSE’94), pp. 191–200 (1994)
Jiang, B., Zhang, Z., Chan, W.K., Tse, T.H.: Adaptive random test case prioritization. In: 24th IEEE/ACM International Conference on Automated Software Engineering (ASE 2009), Auckland, New Zealand, 16–20 November 2009, pp. 233–244. IEEE Comput. Soc., Los Alamitos (2009)
Jones, J.A., Harrold, M.J.: Test-suite reduction and prioritization for modified condition/decision coverage. IEEE Trans. Softw. Eng. 29(3), 195–209 (2003)
Kessis, M., Ledru, Y., Vandome, G.: Experiences in coverage testing of a Java middleware. In: 5th Int. Workshop on Software Engineering and Middleware. ACM, New York (2005)
Kovács, G., Németh, G.Á., Subramaniam, M., Pap, Z.: Optimal string edit distance based test suite reduction for SDL specifications. In: Reed, R., Bilgic, A., Gotzhein, R. (eds.) Design for Motes and Mobiles, Proceedings of 14th International SDL Forum (SDL 2009), Bochum, Germany, 22–24 September 2009. Lecture Notes in Computer Science, vol. 5719, pp. 82–97. Springer, Berlin (2009)
Ledru, Y., Petrenko, A., Boroday, S.: Using string distances for test case prioritisation. In: 24th IEEE/ACM Int. Conf. on Automated Software Engineering (ASE 2009), Short paper, pp. 510–514. IEEE Press, New York (2009)
Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals (in Russian). Dokl. Akad. Nauk SSSR 163(4), 845–848 (1965)
Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals (English translation). Sov. Phys. Dokl. 10(8), 707–710 (1966)
Li, Z., Harman, M., Hierons, R.M.: Search algorithms for regression test case prioritization. IEEE Trans. Softw. Eng. 33(4), 225–237 (2007)
Malaiya, Y.: Antirandom testing: getting the most out of black-box testing. In: 6th Int. Symp. on Software Reliability Engineering. IEEE Press, New York (1995)
Mayer, J., Schneckenburger, C.: An empirical analysis and comparison of random testing techniques. In: Int. Symp. on Empirical Software Engineering (ISESE 2006). ACM Press, New York (2006)
Qu, X., Cohen, M.B., Rothermel, G.: Configuration-aware regression testing: an empirical study of sampling and prioritization. In: Ryder, B.G., Zeller, A. (eds.) Proceedings of the ACM/SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2008), Seattle, WA, USA, 20–24 July 2008, pp. 75–86. ACM Press, New York (2008)
Ramanathan, M.K., Koyutürk, M., Grama, A., Jagannathan, S.: Phalanx: a graph-theoretic framework for test case prioritization. In: Wainwright, R.L., Haddad, H. (eds.) Proceedings of the 2008 ACM Symposium on Applied Computing (SAC), Fortaleza, Ceara, Brazil, 16–20 March 2008, pp. 667–673. ACM Press, New York (2008)
Rothermel, G., Harrold, M.J., Ostrin, J., Hong, C.: An empirical study of the effects of minimization on the fault detection capabilities of test suites. In: Int. Conf. on Software Maintenance, pp. 34–43. IEEE Press, New York (1998)
Rothermel, G., Untch, R.H., Chu, C., Harrold, M.J.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 27(10), 929–948 (2001)
Smith, A.M., Geiger, J., Kapfhammer, G.M., Soffa, M.L.: Test suite reduction and prioritization with call trees. In: 22nd IEEE/ACM Int. Conf. on Automated Software Engineering (ASE 2007), pp. 539–540. ACM Press, New York (2007)
Snedecor, G., Cochran, W.: Statistical Methods, 6th edn. Iowa State University Press, Ames (1957)
Srikanth, H., Williams, L., Osborne, J.: System test case prioritization of new and regression test cases. In: International Symposium on Empirical Software Engineering. IEEE Comput. Soc., Los Alamitos (2005)
Srivastava, A., Thiagarajan, J.: Effectively prioritizing tests in development environment. In: ACM/SIGSOFT Int. Symp. on Software Testing and Analysis (ISSTA 2002). ACM, New York (2002)
Vuong, S.T., Alilovic-Curgus, J.: On test coverage metrics for communication protocols. In: Kroon, J., Heijink, R.J., Brinksma, E. (eds.) Protocol Test Systems, IV, Proceedings of the IFIP TC6/WG6.1 Fourth International Workshop on Protocol Test Systems, Leidschendam, The Netherlands, 15–17 October 1991. IFIP Transactions, vol. C-3, pp. 31–45. North-Holland, Amsterdam (1992)
Walcott, K.R., Soffa, M.L., Kapfhammer, G.M., Roos, R.S.: Timeaware test suite prioritization. In: ACM/SIGSOFT Int. Symp. on Software Testing and Analysis. ACM Press, New York (2006)
Wong, W.E., Horgan, J.R., London, S., Agrawal, H.: A study of effective regression testing in practice. In: 8th Int. Symp. on Software Reliability Engineering. IEEE Press, New York (1997)
Wong, W.E., Horgan, J.R., Mathur, A.P., Pasquini, A.: Test set size minimization and fault detection effectiveness: a case study in a space application. J. Syst. Softw. 48, 79–89 (1999)
Yin, H., Lebne-Dengel, Z., Malaiya, Y.K.: Automatic test generation using checkpoint encoding and antirandom testing. In: 8th Int. Symp. on Software Reliability Engineering, pp. 84–95. IEEE Press, New York (1997)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ledru, Y., Petrenko, A., Boroday, S. et al. Prioritizing test cases with string distances. Autom Softw Eng 19, 65–95 (2012). https://doi.org/10.1007/s10515-011-0093-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10515-011-0093-0