Skip to main content
SpringerLink
Log in

Black-box tree test case generation through diversity

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

To identify defects and security risks in many real-world applications structured test cases, including test cases structured as trees are required. A simple approach is to generate random trees as test cases [random testing (RT)]; however, the RT approach is not very effective. In this work, we investigate and extend the black-box tree test case generation approaches. We introduce a novel model to produce superior test case generation based around the idea of measuring the diversity of a tree test set. This initial approach is further extended by adding a second model which describes the distribution of tree sizes. Both models are realized via a multi-objective optimization algorithm. An empirical study is performed with four real-world programs indicating that the generated tree test cases outperform test cases generated by other methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Ali, S., Briand, L.C., Hemmati, H., Panesar-Walawege, R.K.: A systematic review of the application and empirical investigation of search-based test case generation. IEEE Trans. Softw. Eng. 36(6), 742–762 (2010). https://doi.org/10.1109/TSE.2009.52

    Article  Google Scholar 

  • Ammann, P.E., Knight, J.C.: Data diversity: an approach to software fault tolerance. IEEE Trans. Comput. 37(4), 418–425 (1988). https://doi.org/10.1109/12.2185

    Article  Google Scholar 

  • Anand, S., et al.: An orchestrated survey of methodologies for automated software test case generation. J. Syst. Softw. 86(8), 1978–2001 (2013). https://doi.org/10.1016/j.jss.2013.02.061

    Article  Google Scholar 

  • Andrews, J.H., Briand, L.C., Labiche, Y., Namin, A.S.: Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Trans. Softw. Eng. 32(8), 608–624 (2006). https://doi.org/10.1109/TSE.2006.83

    Article  Google Scholar 

  • Arcuri, A., Briand, L.: Adaptive random testing: An illusion of effectiveness? In: Proceedings of the 2011 International Symposium on Software Testing and Analysis. ISSTA’11. ACM, New York, pp. 265–275 (2011). https://doi.org/10.1145/2001420.2001452

  • Bai, X., Dong, W., Tsai, W.T., Chen, Y.: WSDL-based automatic test case generation for Web services testing. In: IEEE International Workshop Service-Oriented System Engineering, 2005. SOSE 2005, pp. 207–212 (2005). https://doi.org/10.1109/SOSE.2005.43

  • Barbosa, D., Mendelzon, A.O., Keenleyside, J., Lyons, K.: ToXgene: an extensible template-based data generator for XML. In: In WebDB, pp. 49–54 (2002)

  • Baroudi, J.J., Orlikowski, W.J.: The problem of statistical power in MIS research. MIS Q. 1989, 87–106 (1989)

    Article  Google Scholar 

  • Bartolini, C., Bertolino, A., Marchetti, E., Polini, A.: WS-TAXI: a WSDL-based testing tool for web services. In: International Conference on Software Testing Verification and Validation, 2009. ICST’09, pp. 326–335 (2009). https://doi.org/10.1109/ICST.2009.28

  • Benford, F.: The law of anomalous numbers. Proc. Am. Philos. Soc. 78(4), 551–572 (1938)

    MATH  Google Scholar 

  • Berger, A., Hill, T.P.: A basic theory of Benford’s Law. Probab. Surv. 8(2011), 1–126 (2011a)

    Article  MathSciNet  MATH  Google Scholar 

  • Berger, A., Hill, T.P.: Benford’s law strikes back: no simple explanation in sight for mathematical gem. Math. Intell. 33(1), 85–91 (2011b). https://doi.org/10.1007/s00283-010-9182-3

    Article  MathSciNet  MATH  Google Scholar 

  • Bertolino, A., Gao, J., Marchetti, E., Polini, A.: Automatic test data generation for XML schema-based partition testing. In: Proceedings of the 2nd International Workshop on Automation of Software Test. AST’07, pp. 4 . IEEE Computer Society, Washington, DC (2007a). https://doi.org/10.1109/AST.2007.6

  • Bertolino, A., Gao, J., Marchetti, E., Polini, A.: Systematic generation of XML instances to test complex software applications. In: Guelfi, N., Buchs, D. (eds.) Rapid Integration of Software Engineering Techniques. Lecture Notes in Computer Science, pp. 114–129. Springer, Berlin (2007). https://doi.org/10.1007/978-3-540-71876-5_8

    Chapter  Google Scholar 

  • Bertolino, A., Gao, J., Marchetti, E., Polini, A.: TAXI—a tool for XML-based testing. In: Companion to the Proceedings of the 29th International Conference on Software Engineering. ICSE COMPANION’07, pp. 53–54. IEEE Computer Society, Washington, DC (2007c). https://doi.org/10.1109/ICSECOMPANION.2007.72

  • Bishop, P.G.: The variation of software survival time for different operational input profiles (or why you can wait a long time for a big bug to fail). In: The 23rd International Symposium on Fault-Tolerant Computing, 1993. FTCS-23. Digest of Papers, pp. 98–107 (1993). https://doi.org/10.1109/FTCS.1993.627312

  • Budd, T.A., Lipton, R.J., DeMillo, R.A., Sayward, F.G.: Mutation Analysis. Department of Computer Science, Yale University, New Haven (1979)

    Book  Google Scholar 

  • Buttler, D.: A short survey of document structure similarity algorithms. In: The 5th International Conference on Internet Computing (2004)

  • Chan, F.T., Chen, T.Y., Mak, I.K., Yu, Y.T.: Proportional sampling strategy: guidelines for software testing practitioners. Inf. Softw. Technol. 38(12), 775–782 (1996). https://doi.org/10.1016/0950-5849(96)01103-2

    Article  Google Scholar 

  • Chan, K., Chen, T., Towey, D.: Restricted Random Testing. In: Kontio, J., Conradi, R. (eds.) Software Quality—ECSQ 2002. Lecture Notes in Computer Science, pp. 321–330. Springer, Berlin (2002)

    Google Scholar 

  • Chen, T.Y., Tse, T.H., Yu, Y.T.: Proportional sampling strategy: a compendium and some insights. J. Syst. Softw. 58(1), 65–81 (2001). https://doi.org/10.1016/S0164-1212(01)00028-0

    Article  Google Scholar 

  • Chen, T.Y., Leung, H., Mak, I.K.: Adaptive Random Testing. In: Maher, M. (ed.) Advances in Computer Science—ASIAN 2004. Lecture Notes in Computer Science, pp. 3156–3157. Springer, Berlin (2005)

    Google Scholar 

  • Chen, T.Y., Kuo, F.-C., Merkel, R.: On the statistical properties of testing effectiveness measures. J. Syst. Softw. 79(5), 591–601 (2006). https://doi.org/10.1016/j.jss.2005.05.029

    Article  Google Scholar 

  • Chen, T.Y., Kuo, F.-C., Liu, H., Wong, W.E.: Code coverage of adaptive random testing. IEEE Trans. Reliab. 62(1), 226–237 (2013). https://doi.org/10.1109/TR.2013.2240898

    Article  Google Scholar 

  • Ciupa, I., Leitner, A., Oriol, M., Meyer, B.: ARTOO: adaptive random testing for object-oriented software. In: ACM/IEEE 30th International Conference on Software Engineering, 2008. ICSE’08, pp. 71–80 (2008). https://doi.org/10.1145/1368088.1368099

  • Cohen, J.: Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum, Mahwah (1988)

    MATH  Google Scholar 

  • Cohen, J.: A power primer. Psychol. Bull. 112(1), 155–159 (1992)

    Article  Google Scholar 

  • Connor, R., Simeoni, F., Iakovos, M., Moss, R.: A bounded distance metric for comparing tree structure. Inf. Syst. 36(4), 748–764 (2011). https://doi.org/10.1016/j.is.2010.12.003

    Article  Google Scholar 

  • De Jong, K.A., Spears, W.M.: An analysis of the interacting roles of population size and crossover in genetic algorithms. In: Parallel Problem Solving from Nature, pp. 38–47. Springer (1991)

  • Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002). https://doi.org/10.1109/4235.996017

    Article  Google Scholar 

  • Downey, A.B.: The structural cause of file size distributions. In: Proceedings of 9th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, 2001, pp 361–370 (2001). https://doi.org/10.1109/MASCOT.2001.948888

  • Duran, J.W., Ntafos, S.C.: An evaluation of random testing. IEEE Trans. Softw. Eng. SE-10(4), 438–444 (1984). https://doi.org/10.1109/TSE.1984.5010257

    Article  Google Scholar 

  • Durtschi, C., Hillison, W., Pacini, C.: The effective use of Benford’s law to assist in detecting fraud in accounting data. J. Forensic Acc. 5(1), 17–34 (2004)

    Google Scholar 

  • Feldt, R., Poulding, S.: Finding test data with specific properties via metaheuristic search. In: 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE), pp. 350–359 (2013). https://doi.org/10.1109/ISSRE.2013.6698888

  • Finelli, G.B.: NASA software failure characterization experiments. Reliab. Eng. Syst. Saf. 32(1–2), 155–169 (1991). https://doi.org/10.1016/0951-8320(91)90052-9

    Article  Google Scholar 

  • Forrester, J.E., Miller, B.P.: An empirical study of the robustness of windows NT applications using random testing. In: Proceedings of the 4th conference on USENIX Windows Systems Symposium, pp. 59–68. USENIX Association, Berkeley, CA (2000)

  • Frankl, P.G., Weiss, S.N., Hu, C.: All-uses versus mutation testing: an experimental comparison of effectiveness. Syst. Softw. 38(3), 235–253 (1997). https://doi.org/10.1016/S0164-1212(96)00154-9

    Article  Google Scholar 

  • Fraser, G., Arcuri, A.: EvoSuite: automatic test suite generation for object-oriented software. In: Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering. ESEC/FSE’11, pp. 416–419. ACM, New York (2011). https://doi.org/10.1145/2025113.2025179

  • Ganesh, V., Kiezun, A., Artzi, S., Guo, P.J., Hooimeijer, P., Ernst, M.: HAMPI: a string solver for testing, analysis and vulnerability detection. In: Computer Aided Verification, pp. 1–19 (2011)

  • Geyer, C.L., Williamson, P.P.: Detecting fraud in data sets using benford’s law. Commun. Stat. Simul. Comput. 33(1), 229–246 (2004). https://doi.org/10.1081/SAC-120028442

    Article  MathSciNet  MATH  Google Scholar 

  • Godefroid, P. Random testing for security: blackbox vs. whitebox fuzzing. In: Proceedings of the 2nd international workshop on Random testing: co-located with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2007). RT’07, pp. 1. ACM, New York (2007)

  • Godefroid, P., Klarlund, N., Sen, K.: DART: directed automated random testing. SIGPLAN Not. 40(6), 213–223 (2005). https://doi.org/10.1145/1064978.1065036

    Article  Google Scholar 

  • Harman, M., Jones, B.F.: Search-based software engineering. Inf. Softw. Technol. 43(14), 833–839 (2001). https://doi.org/10.1016/S0950-5849(01)00189-6

    Article  Google Scholar 

  • Harman, M.: The current state and future of search based software engineering. In: 2007 Future of Software Engineering. FOSE’07, pp. 342–357. IEEE Computer Society, Washington, DC (2007). https://doi.org/10.1109/FOSE.2007.29

  • Havrikov, N., Höschele, M., Galeotti, J.P., Zeller, A.: XMLMate: evolutionary XML test generation. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 719–722 (2014)

  • Hill, T.P.: The significant-digit phenomenon. Am. Math. Mon. 102(4), 322–327 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  • Jiang, T., Wang, L., Zhang, K.: Alignment of trees—an alternative to tree edit. Theor. Comput. Sci. 143(1), 137–148 (1995). https://doi.org/10.1016/0304-3975(95)80029-9

    Article  MathSciNet  MATH  Google Scholar 

  • Just, R., Jalali, D., Inozemtseva, L., Ernst, M.D., Holmes, R., Fraser, G.: Are mutants a valid substitute for real faults in software testing. In: International Symposium on the Foundations of Software Engineering (FSE) (2014) (to appear)

  • Lee, S.C., Offutt, J.: Generating test cases for XML-based Web component interactions using mutation analysis. In: Proceedings of 12th International Symposium on Software Reliability Engineering, 2001. ISSRE 2001, pp. 200–209 (2001). https://doi.org/10.1109/ISSRE.2001.989474

  • Lipner, S., Howard, M.: The Trustworthy Computing Security Development Lifecycle document (SDL) (2005). https://msdn.microsoft.com/en-us/library/ms995349.aspx

  • Lv, J., Hu, H., Cai, K.Y., Chen, T.Y.: Adaptive and random partition software testing. IEEE Trans. Syst. Man. Cybern. Syst. PP(99), 1 (2014). https://doi.org/10.1109/TSMC.2014.2318019

    Google Scholar 

  • Ma, Y.-S., Offutt, J., Kwon, Y.R.: MuJava: an automated class mutation system. Softw. Test. Verif. Reliab. 15(2), 97–133 (2005). https://doi.org/10.1002/stvr.308

    Article  Google Scholar 

  • Mayer, J., Schneckenburger, C.: An empirical analysis and comparison of random testing techniques. In: Proceedings of the 2006 ACM/IEEE International Symposium on Empirical Software Engineering. ISESE’06, pp. 105–114. ACM, New York (2006). https://doi.org/10.1145/1159733.1159751

  • Muller-Molina, A., Hirata, K., Shinohara, T.: A tree distance function based on multi-sets. In: Chawla, S., et al. (eds.) New Frontiers in Applied Data Mining. Lecture Notes in Computer Science, pp. 87–98. Springer, Berlin (2009)

    Chapter  Google Scholar 

  • Newcomb, S.: Note on the frequency of use of the different digits in natural numbers. Am. J. Math. 4(1), 39–40 (1881)

    Article  MathSciNet  MATH  Google Scholar 

  • Nierman, A., Jagadish, H.V.: Evaluating structural similarity in XML documents. In: Proceedings of 5th International Workshop on the Web and Databases (WebDB 2002), pp. 61–66, Madison, Wisconsin (2002)

  • Nigrini, M.J., Mittermaier, L.J.: The use of Benford’s law as an aid in analytical procedures. Auditing 16(1997), 52–67 (1997)

    Google Scholar 

  • Offutt, J., Xu, W.: Generating test cases for web services using data perturbation. SIGSOFT Softw. Eng. Notes 29(5), 1–10 (2004). https://doi.org/10.1145/1022494.1022529

    Google Scholar 

  • Pacheco, C., Lahiri, S.K., Ball, T.: Finding errors in .NET with feedback-directed random testing. In: Proceedings of the 2008 International Symposium on Software Testing and Analysis. ISSTA’08, pp. 87–96. ACM, New York (2008)

  • Pacheco, C., Lahiri, S.K., Ernst, M.D., Ball, T.: Feedback-directed random test generation. In: 29th International Conference on Software Engineering, 2007. ICSE 2007, pp. 75–84 (2007). https://doi.org/10.1109/ICSE.2007.37

  • Raimi, R.A.: The first digit problem. Am. Math. Mon. 83(7), 521–538 (1976)

    Article  MathSciNet  MATH  Google Scholar 

  • Santelices, R., Harrold, M.J., Orso, A.: Precisely detecting runtime change interactions for evolving software. In: 2010 3rd International Conference on Software Testing, Verification and Validation (ICST), pp. 429–438 (2010). https://doi.org/10.1109/ICST.2010.29

  • Santelices, R., Harrold, M.J.: Exploiting program dependencies for scalable multiple-path symbolic execution. In: Proceedings of the 19th International Symposium on Software Testing and Analysis. ISSTA, pp. 195–206. ACM, New York (2010). https://doi.org/10.1145/1831708.1831733

  • Scheemaecke, M.D.: NanoXML (2001). http://nanoxml.sourceforge.net/orig

  • Schneckenburger, C., Mayer, J.: Towards the determination of typical failure patterns. In: 4th International Workshop on Software Quality Assurance: In Conjunction with the 6th ESEC/FSE Joint Meeting. SOQUA’07, pp. 90–93. ACM, New York (2007). https://doi.org/10.1145/1295074.1295091

  • Selkow, S.M.: The tree-to-tree editing problem. Inf. Process. Lett. 6(6), 184–186 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  • Shahbazi, A.: Miller J (2015) Black-box string test case generation through a multi-objective optimization. IEEE Trans. Under Revis. Softw. Eng. 42, 361–378 (2015)

    Article  Google Scholar 

  • Shahbazi, A., Miller, J.: Extended subtree: a new similarity function for tree structured data. IEEE Trans. Knowl. Data Eng. 26(4), 864–877 (2014). https://doi.org/10.1109/TKDE.2013.53

    Article  Google Scholar 

  • Shahbazi, A., Tappenden, A.F., Miller, J.: Centroidal Voronoi tessellations—a new approach to random testing. IEEE Trans. Softw. Eng. 39(2), 163–183 (2013). https://doi.org/10.1109/TSE.2012.18

    Article  Google Scholar 

  • Sneed, H.M., Huang, S.: WSDLTest—a tool for testing web services. In: 8th IEEE International Symposium on Web Site Evolution, 2006. WSE’06, pp. 14–21 (2006) https://doi.org/10.1109/WSE.2006.24

  • Spears, W.M.: Adapting crossover in evolutionary algorithms. In: Proceedings of the 4th Annual Conference on Evolutionary Programming, pp. 367–384. MIT Press (1995)

  • StAX.: Home of the original Stax (2006). http://stax.codehaus.org

  • Stephens, M.A.: Use of the Kolmogorov–Smirnov, Cramer–Von Mises and related statistics without extensive tables. J. R. Stat. Soc. Ser. B 1970, 115–122 (1970)

    MATH  Google Scholar 

  • Tai, K.-C.: The tree-to-tree correction problem. J. ACM 26(3), 422–433 (1979). https://doi.org/10.1145/322139.322143

    Article  MathSciNet  MATH  Google Scholar 

  • Tanaka, E., Tanaka, K.: The tree-to-tree editing problem. Int. J. Pattern Recog. Artif. Intell. 2(2), 221–240 (1988)

    Article  Google Scholar 

  • Tappenden, A.F., Miller, J.: A novel evolutionary approach for adaptive random testing. IEEE Trans. Reliab. 58(4), 619–633 (2009). https://doi.org/10.1109/TR.2009.2034288

    Article  Google Scholar 

  • Tappenden, A., Beatty, P., Miller, J., Geras, A., Smith, M. (2005) Agile security testing of web-based systems via HTTPUnit. In: Agile Conference, 2005. Proceedings. pp. 29–38. http://dx.doi.org/10.1109/ADC.2005.11

  • Valiente, G.: An efficient bottom-up distance between trees. In: Proceedings 8th International Symposium on String Processing and Information Retrieval, 2001. SPIRE 2001, pp. 212–219 (2001). https://doi.org/10.1109/SPIRE.2001.989761

  • Vanderveen, P., Janzen, M., Tappenden, A.F.: A web service test generator. In: 2014 IEEE International Conference on Software Maintenance and Evolution (ICSME), pp. 516–520 (2014). https://doi.org/10.1109/ICSME.2014.85

  • Wang, J.T.L., Zhang, K.: Finding similar consensus between trees: an algorithm and a distance hierarchy. Pattern Recognit. 34(1), 127–137 (2001). https://doi.org/10.1016/S0031-3203(99)00199-5

    Article  MATH  Google Scholar 

  • White, L.J., Cohen, E.I.: A domain strategy for computer program testing. IEEE Trans. Softw. Eng. SE-6(3), 247–257 (1980). https://doi.org/10.1109/TSE.1980.234486

    Article  MATH  Google Scholar 

  • Whitley, D.: A genetic algorithm tutorial. Stat. Comput. 4(2), 65–85 (1994)

    Article  Google Scholar 

  • Xu, W., Offutt, J., Luo, J.: Testing Web services by XML perturbation. In: 16th IEEE International Symposium on Software Reliability Engineering, 2005. ISSRE 2005, vol. 10, pp.–266 (2005). https://doi.org/10.1109/ISSRE.2005.44

  • Yang, R., Kalnis, P., Tung, A.K.H.: Similarity evaluation on tree-structured data. In: Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data. SIGMOD’05, pp. 754–765. ACM, New York (2005). https://doi.org/10.1145/1066157.1066243

  • Yoshikawa, T., Shimura, K., Ozawa, T.: Random program generator for Java JIT compiler test system. In: Proceedings of 3rd International Conference on Quality Software, 2003, pp. 20–23 (2003). https://doi.org/10.1109/QSIC.2003.1319081

  • Zhang, K.: Shasha D (1989) Simple fast algorithms for the editing distance between trees and related problems. SIAM J. Comput. 18(6), 1245–1262 (1989)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work is supported by the NSERC (National Science and Engineering Research Council of Canada).

Author information

Authors and Affiliations

  1. Department Electrical and Computer Engineering, University of Alberta, Edmonton, Canada

    Ali Shahbazi, Mahsa Panahandeh & James Miller

Authors
  1. Ali Shahbazi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mahsa Panahandeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. James Miller
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to James Miller.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shahbazi, A., Panahandeh, M. & Miller, J. Black-box tree test case generation through diversity. Autom Softw Eng 25, 531–568 (2018). https://doi.org/10.1007/s10515-018-0232-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10515-018-0232-y

Keywords

Search

Navigation