Abstract
A common application of search-based software testing is to generate test cases for all goals defined by a coverage criterion (e.g., statements, branches, mutants). Rather than generating one test case at a time for each of these goals individually, whole test suite generation optimizes entire test suites towards satisfying all goals at the same time. There is evidence that the overall coverage achieved with this approach is superior to that of targeting individual coverage goals. Nevertheless, there remains some uncertainty on whether the whole test suite approach might be inferior to a more focused search in the case of particularly difficult coverage goals. In this paper, we perform an in-depth analysis to study if this is the case. An empirical study on 100 Java classes reveals that indeed there are some testing goals that are easier to cover with the traditional approach. However, their number is not only very small in comparison with those which are only covered by the whole test suite approach, but also those coverage goals appear in small classes for which both approaches already obtain high coverage.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Ali, S., Briand, L., Hemmati, H., Panesar-Walawege, R.: A systematic review of the application and empirical investigation of search-based test-case generation. IEEE Transactions on Software Engineering (TSE) 36(6), 742–762 (2010)
Arcuri, A.: A theoretical and empirical analysis of the role of test sequence length in software testing for structural coverage. IEEE Transactions on Software Engineering (TSE) 38(3), 497–519 (2012)
Arcuri, A.: It really does matter how you normalize the branch distance in search-based software testing. Software Testing, Verification and Reliability (STVR) 23(2), 119–147 (2013)
Arcuri, A., Briand, L.: A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability, STVR (2012), doi:10.1002/stvr.1486
Arcuri, A., Iqbal, M.Z., Briand, L.: Random testing: Theoretical results and practical implications. IEEE Transactions on Software Engineering (TSE) 38(2), 258–277 (2012)
Arcuri, A., Yao, X.: Search based software testing of object-oriented containers. Inform. Sciences 178(15), 3075–3095 (2008)
Baresi, L., Lanzi, P.L., Miraz, M.: Testful: An evolutionary test approach for java. In: IEEE International Conference on Software Testing, Verification and Validation (ICST), pp. 185–194 (2010)
Fraser, G., Arcuri, A.: EvoSuite: Automatic test suite generation for object-oriented software. In: ACM Symposium on the Foundations of Software Engineering (FSE), pp. 416–419 (2011)
Fraser, G., Arcuri, A.: Sound empirical evidence in software testing. In: ACM/IEEE International Conference on Software Engineering (ICSE), pp. 178–188 (2012)
Fraser, G., Arcuri, A.: Handling test length bloat. Software Testing, Verification and Reliability, STVR (2013), doi:10.1002/stvr.1495
Fraser, G., Arcuri, A.: Whole test suite generation. IEEE Transactions on Software Engineering 39(2), 276–291 (2013)
Fraser, G., Arcuri, A.: Achieving scalable mutation-based generation of whole test suites. Empirical Software Engineering (EMSE) (to appear, 2014)
Fraser, G., Zeller, A.: Mutation-driven generation of unit tests and oracles. IEEE Transactions on Software Engineering (TSE) 28(2), 278–292 (2012)
Harman, M., Kim, S.G., Lakhotia, K., McMinn, P., Yoo, S.: Optimizing for the number of tests generated in search based test data generation with an application to the oracle cost problem. In: International Workshop on Search-Based Software Testing, SBST (2010)
Harman, M., Mansouri, S.A., Zhang, Y.: Search-based software engineering: Trends, techniques and applications. ACM Computing Surveys (CSUR)Â 45(1), 11 (2012)
Jia, Y., Harman, M.: An analysis and survey of the development of mutation testing. Technical Report TR-09-06, CREST Centre, King’s College London, London, UK (September 2009)
Korel, B.: Automated software test data generation. IEEE Transactions on Software Engineering, 870–879 (1990)
Lakhotia, K., McMinn, P., Harman, M.: An empirical investigation into branch coverage for c programs using cute and austin. J. Syst. Softw. 83(12) (December 2010)
McMinn, P.: Search-based software test data generation: A survey. Software Testing, Verification and Reliability 14(2), 105–156 (2004)
Miller, W., Spooner, D.L.: Automatic generation of floating-point test data. IEEE Transactions on Software Engineering 2(3), 223–226 (1976)
Ribeiro, J.C.B.: Search-based test case generation for object-oriented Java software using strongly-typed genetic programming. In: Genetic and Evolutionary Computation Conference (GECCO), pp. 1819–1822. ACM (2008)
Tonella, P.: Evolutionary testing of classes. In: ACM Int. Symposium on Software Testing and Analysis (ISSTA), pp. 119–128 (2004)
Wappler, S., Lammermann, F.: Using evolutionary algorithms for the unit testing of object-oriented software. In: Genetic and Evolutionary Computation Conference (GECCO), pp. 1053–1060. ACM (2005)
Wegener, J., Baresel, A., Sthamer, H.: Evolutionary test environment for automatic structural testing. Information and Software Technology 43(14), 841–854 (2001)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Arcuri, A., Fraser, G. (2014). On the Effectiveness of Whole Test Suite Generation. In: Le Goues, C., Yoo, S. (eds) Search-Based Software Engineering. SSBSE 2014. Lecture Notes in Computer Science, vol 8636. Springer, Cham. https://doi.org/10.1007/978-3-319-09940-8_1
Download citation
DOI: https://doi.org/10.1007/978-3-319-09940-8_1
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-09939-2
Online ISBN: 978-3-319-09940-8
eBook Packages: Computer ScienceComputer Science (R0)