demonstration

Unit testing tool competition: round four

Authors:

Urko Rueda,

René Just,

Juan P. Galeotti,

Tanja E. J. VosAuthors Info & Claims

SBST '16: Proceedings of the 9th International Workshop on Search-Based Software Testing

Pages 19 - 28

https://doi.org/10.1145/2897010.2897018

Published: 14 May 2016 Publication History

Get Access

Abstract

This paper describes the methodology and results of the 4th edition of the Java Unit Testing Tool Competition. This year's competition features a number of infrastructure improvements, new test effectiveness metrics, and the evaluation of the test generation tools for multiple time budgets. Overall, the competition evaluated four automated test generation tools. This paper details the methodology and contains the full results of the competition.

References

[1]

S. Bauersfeld, T. Vos, and K. Lakhotia. Unit testing tool competition -- lessons learned. Workshop on Future Internet Testing (FITTEST), 2013.

Google Scholar

[2]

G. Fraser and A. Arcuri. Evosuite: automatic test suite generation for object-oriented software. In SIGSOFT FSE, pages 416--419, 2011.

Digital Library

Google Scholar

[3]

R. Just. The Major mutation framework: Efficient and scalable mutation analysis for Java. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), pages 433--436, San Jose, CA, USA, July 23--25 2014.

Digital Library

Google Scholar

[4]

R. Just, D. Jalali, and M. D. Ernst. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), pages 437--440, San Jose, CA, USA, July 23--25 2014.

Digital Library

Google Scholar

[5]

R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser. Are mutants a valid substitute for real faults in software testing? In Proceedings of the Symposium on the Foundations of Software Engineering (FSE), pages 654--665, Hong Kong, November 18--20 2014.

Digital Library

Google Scholar

[6]

L. Ma, C. Artho, C. Zhang, H. Sato, M. Hagiya, Y. Tanabe, and M. Yamamoto. GRT at the SBST 2015 tool competition. In Proceedings of the Eighth International Workshop on Search-Based Software Testing, pages 48--51, 2015.

Digital Library

Google Scholar

[7]

C. Pacheco and M. D. Ernst. Randoop: feedback-directed random testing for java. In Companion to the 22nd ACM SIGPLAN conference on Object-oriented programming systems and applications companion, OOPSLA '07, pages 815--816, New York, NY, USA, 2007. ACM.

Digital Library

Google Scholar

[8]

I. Prasetya. T3, a combinator-based random testing tool for Java: Benchmarking. Int. Workshop Future Internet Testing, Lecture Notes in Computer Science, 8432, 2014.

Google Scholar

[9]

I. S. W. B. Prasetya, T. E. J. Vos, and A. Baars. Trace-based reflexive testing of OO programs with T2. In 1st Int. Conf. on Software Testing, Verification, and Validation (ICST), pages 151--160. IEEE, 2008.

Digital Library

Google Scholar

[10]

U. Rueda, T. E. J. Vos, and I. S. W. B. Prasetya. Unit Testing Tool Competition - Round Three. In G. Gay and G. Antoniol, editors, 8th IEEE/ACM International Workshop on Search-Based Software Testing, SBST 2015, Florence, Italy, May 18-19, 2015, pages 19--24, 2015.

Digital Library

Google Scholar

[11]

A. Sakti, G. Pesant, and Y. Gueheneuc. Instance generator and problem representation to improve object oriented code coverage. To appear in IEEE Transactions on Software Engineering, 2015.

Google Scholar

Cited By

View all

Neelofar NSmith-Miles KMuñoz MAleti A(2023)Instance Space Analysis of Search-Based Software TestingIEEE Transactions on Software Engineering10.1109/TSE.2022.322833449:4(2642-2660)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2022.3228334
Derakhshanfar PDevroey XPanichella AZaidman Avan Deursen A(2023)Generating Class-Level Integration Tests Using Call Site InformationIEEE Transactions on Software Engineering10.1109/TSE.2022.320962549:4(2069-2087)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2022.3209625
Devroey XGambi AGaleotti JJust RKifetew FPanichella APanichella S(2022) JUGE : An infrastructure for benchmarking Java unit test generators Software Testing, Verification and Reliability10.1002/stvr.183833:3Online publication date: 20-Dec-2022
https://doi.org/10.1002/stvr.1838
Show More Cited By

Unit testing tool competition: round four
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis

Recommendations

Java Unit Testing Tool Competition: Eighth Round
ICSEW'20: Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops

We report on the results of the eighth edition of the Java unit testing tool competition. This year, two tools, EvoSuite and Randoop, were executed on a benchmark with (i) new classes under test, selected from open-source software projects, and (ii) the ...
Java unit testing tool competition: sixth round
SBST '18: Proceedings of the 11th International Workshop on Search-Based Software Testing

We report on the advances in this sixth edition of the JUnit tool competitions. This year the contest introduces new benchmarks to assess the performance of JUnit testing tools on different types of real-world software projects. Following on the ...
Unit testing tool competition: round three
SBST '15: Proceedings of the Eighth International Workshop on Search-Based Software Testing

This paper describes the third round of the Java Unit Testing Tool Competition. This edition of the contest evaluates no less than seven automated testing tools! And, like during the second round, test suites written by human testers are also used for ...

Comments

Information & Contributors

Information

Published In

SBST '16: Proceedings of the 9th International Workshop on Search-Based Software Testing

May 2016

68 pages

ISBN:9781450341660

DOI:10.1145/2897010

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 May 2016

Check for updates

Author Tags

Qualifiers

Demonstration

Funding Sources

PERTEST project

Conference

ICSE '16

Sponsor:

ACM
SIGSOFT
IEEE-CS\DATC
TCSE

ICSE '16: 38th International Conference on Software Engineering

May 14 - 22, 2016

Texas, Austin

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
234
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Neelofar NSmith-Miles KMuñoz MAleti A(2023)Instance Space Analysis of Search-Based Software TestingIEEE Transactions on Software Engineering10.1109/TSE.2022.322833449:4(2642-2660)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2022.3228334
Derakhshanfar PDevroey XPanichella AZaidman Avan Deursen A(2023)Generating Class-Level Integration Tests Using Call Site InformationIEEE Transactions on Software Engineering10.1109/TSE.2022.320962549:4(2069-2087)Online publication date: 1-Apr-2023
https://doi.org/10.1109/TSE.2022.3209625
Devroey XGambi AGaleotti JJust RKifetew FPanichella APanichella S(2022) JUGE : An infrastructure for benchmarking Java unit test generators Software Testing, Verification and Reliability10.1002/stvr.183833:3Online publication date: 20-Dec-2022
https://doi.org/10.1002/stvr.1838
Chen YGopinath RTadakamalla AErnst MHolmes RFraser GAmmann PJust RGrundy JLe Goues CLo D(2020)Revisiting the relationship between fault detection, test adequacy criteria, and test set sizeProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering10.1145/3324884.3416667(237-249)Online publication date: 21-Dec-2020
https://dl.acm.org/doi/10.1145/3324884.3416667
Gay GJust R(2020)Defects4J as a Challenge Case for the Search-Based Software Engineering CommunitySearch-Based Software Engineering10.1007/978-3-030-59762-7_19(255-261)Online publication date: 30-Sep-2020
https://doi.org/10.1007/978-3-030-59762-7_19
Serra DGrano GPalomba FFerrucci FGall HBacchelli AStorey MAdams BHaiduc S(2019)On the effectiveness of manual and automatic unit test generationProceedings of the 16th International Conference on Mining Software Repositories10.1109/MSR.2019.00028(121-125)Online publication date: 26-May-2019
https://dl.acm.org/doi/10.1109/MSR.2019.00028
Silva IAlves EMachado PKulesza UPrikladnicki RGerosa MWerner CAndrade R(2018)Can automated test case generation cope with extract method validation?Proceedings of the XXXII Brazilian Symposium on Software Engineering10.1145/3266237.3266274(152-161)Online publication date: 17-Sep-2018
https://dl.acm.org/doi/10.1145/3266237.3266274
Panichella AKifetew FTonella P(2018)Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the TargetsIEEE Transactions on Software Engineering10.1109/TSE.2017.266343544:2(122-158)Online publication date: 1-Feb-2018
https://doi.org/10.1109/TSE.2017.2663435
Dimjašević MHowar FLuckow KRakamarić Z(2018)Study of Integrating Random and Symbolic Testing for Object-Oriented SoftwareIntegrated Formal Methods10.1007/978-3-319-98938-9_6(89-109)Online publication date: 9-Aug-2018
https://doi.org/10.1007/978-3-319-98938-9_6
Krüger JAl‐Hajjaji MLeich TSaake G(2018)Mutation operators for feature‐oriented software product linesSoftware Testing, Verification and Reliability10.1002/stvr.167629:1-2Online publication date: 26-Jul-2018
https://doi.org/10.1002/stvr.1676
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Recommendations

Java Unit Testing Tool Competition: Eighth Round

Java unit testing tool competition: sixth round

Unit testing tool competition: round three

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations