Abstract
This paper reports about the two rounds of the Java Unit Testing Tool Competition that ran in the context of the Search Based Software Testing (SBST) workshop at ICST 2013 and the first Future Internet Testing (FITTEST) workshop at ICTSS 2013. It describes the main objectives of the benchmark, the Java classes that were selected in both competitions, the data that was collected, the tools that were used for data collection, the protocol that was carried out to execute the benchmark and how the final benchmark scores for each participating tool were calculated. Eventually, we discuss the challenges encountered during the events, what we learned and how we plan to improve our framework for future competitions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Apache Commons Lang v3.1. http://commons.apache.org/lang. Accessed 22 Feb 2013
Apache Lucene v4.1.0. http://lucene.apache.org. Accessed 22 Feb 2013
Async Http Client v1.7.20. https://github.com/AsyncHttpClient/async-http-client. Accessed 03 Sept 2013
Barbecue v1.5 beta. http://barbecue.sourceforge.net. Accessed 22 Feb 2013
Cobertura v1.9.4.1. http://cobertura.sourceforge.net. Accessed 22 Feb 2013
Eclipse checkstyle plugin v5.6.1. http://eclipse-cs.sourceforge.net/. Accessed 2 Sept 2013
Gdata Java Client v1.4.7.1. https://code.google.com/p/gdata-java-client/. Accessed 8 July 2013
Guava v15. https://code.google.com/p/guava-libraries/. Accessed 8 Sept 2013
Hibernate v4.2.7. http://www.hibernate.org. Accessed 2 Sept 2013
JaCoCo v0.6.3. http://www.eclemma.org/jacoco/. Accessed 24 Oct 2013
Java Machine Learning Libraryr v0.1.7. http://java-ml.sourceforge.net/. Accessed 9 Sept 2013
Java Wikipedia Library v0.9.2. https://code.google.com/p/jwpl/. Accessed 17 July 2013
Joda Time v2.0. http://joda-time.sourceforge.net. Accessed 22 Feb 2013
Pitest v0.3.1. http://pitest.org. Accessed 24 Oct 2013
Randoop v1.3.3. http://code.google.com/p/randoop/. Accessed 22 Feb 2013
Scribe v1.3.5. https://github.com/fernandezpablo85/scribe-java. Accessed 3 Sept 2013
sqlsheet v6.4. https://code.google.com/p/sqlsheet. Accessed 22 Feb 2013
Twitter4j v3.0.4. http://twitter4j.org/en/. Accessed 3 Sept 2013
Andrews, J., Menzies, T., Li, F.: Genetic algorithms for randomized unit testing. IEEE Trans. Softw. Eng. 37(1), 80–94 (2011)
Basili, V.R., Shull, F., Lanubile, F.: Building knowledge through families of experiments. IEEE Trans. Softw. Eng. 25(4), 456–473 (1999)
Daniel, B., Boshernitsan, M.: Predicting effectiveness of automatic testing tools. In: 23rd IEEE/ACM International Conference on Automated Software Engineering 2008, ASE 2008, pp. 363–366 (2008)
Fraser, G., Zeller, A.: Mutation-driven generation of unit tests and oracles. IEEE Trans. Softw. Eng 38(2), 278–292 (2012)
Fraser, G., Arcuri, A.: Sound empirical evidence in software testing. In: Proceedings of the 2012 International Conference on Software Engineering, ICSE 2012, pp. 178–188. IEEE Press, Piscataway (2012). http://dl.acm.org/citation.cfm?id=2337223.2337245
Fraser, G., Arcuri, A.: Evosuite at the SBST 2013 tool competition. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 406–409, March 2013
Fraser, G., Arcuri, A.: Evosuite at the second unit testing tool competition. In: Future Internet Testing (FITTEST) Workshop (2014)
Harman, M., Jia, Y., Langdon, W.B.: Strong higher order mutation-based test data generation. In: Gyimóthy, T., Zeller, A. (eds.) SIGSOFT FSE. pp. 212–222. ACM (2011). http://doi.acm.org/10.1145/2025113.2025144
Kitchenham, B., Dyba, T., Jorgensen, M.: Evidence-based software engineering. In: Proceedings of ICSE, pp. 273–281. IEEE (2004)
Kitchenham, B.A., Pfleeger, S.L., Pickard, L.M., Jones, P.W., Hoaglin, D.C., Emam, K.E., Rosenberg, J.: Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Eng. 28(8), 721–734 (2002)
Nistor, A., Luo, Q., Pradel, M., Gross, T., Marinov, D.: Ballerina: Automatic generation and clustering of efficient random unit tests for multithreaded code. In: 2012 34th International Conference on Software Engineering (ICSE), pp. 727–737, June 2012
Pacheco, C., Ernst, M.D.: Randoop: feedback-directed random testing for java. In: Companion to the 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications Companion, OOPSLA ’07, pp. 815–816. ACM, New York (2007). http://doi.acm.org/10.1145/1297846.1297902
Pacheco, C., Lahiri, S.K., Ball, T.: Finding errors in.net with feedback-directed random testing. In: Proceedings of the 2008 International Symposium on Software Testing and Analysis, ISSTA ’08, pp. 87–96. ACM, New York (2008). http://doi.acm.org/10.1145/1390630.1390643
Prasetya, I.: Measuring T2 against SBST 2013 benchmark suite. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 410–413, March 2013
Prasetya, I.: T3, a combinator-based random testing tool for java: benchmarking. In: Future Internet Testing (FITTEST) Workshop (2014)
Schuler, D., Zeller, A.: Javalanche: efficient mutation testing for java. In: ESEC/FSE ’09: Proceedings of the 7th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 297–298, Aug 2009
Schuler, D., Zeller, A.: (un-)covering equivalent mutants. In: ICST ’10: Proceedings of the 3rd International Conference on Software Testing, Verification and Validation, pp. 45–54. IEEE Computer Society, Apr 2010
Sim, S., Easterbrook, S., Holt, R.: Using benchmarking to advance research: a challenge to software engineering. In: Proceedings of the 25th International Conference on Software Engineering, 2003, pp. 74–83, May 2003
Tonella, P., Torchiano, M., Du Bois, B., Systä, T.: Empirical studies in reverse engineering: state of the art and future trends. Empirical Softw. Engg. 12(5), 551–571 (2007)
Vegas, S., Basili, V.: A characterisation schema for software testing techniques. Empirical Softw. Engg. 10(4), 437–466 (2005)
Acknowledgements
This work is funded by the European Project with the acronym FITTEST (Future Internet Testing) and contract number (ICT-257574). We would also like to thank Arthur Baars for his initial efforts in setting up the benchmark architecture.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Bauersfeld, S., Vos, T.E.J., Lakhotia, K. (2014). Unit Testing Tool Competitions – Lessons Learned. In: Vos, T., Lakhotia, K., Bauersfeld, S. (eds) Future Internet Testing. FITTEST 2013. Lecture Notes in Computer Science(), vol 8432. Springer, Cham. https://doi.org/10.1007/978-3-319-07785-7_5
Download citation
DOI: https://doi.org/10.1007/978-3-319-07785-7_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-07784-0
Online ISBN: 978-3-319-07785-7
eBook Packages: Computer ScienceComputer Science (R0)