Skip to main content
SpringerLink
Log in
Book cover

International Symposium on Search Based Software Engineering

SSBSE 2020: Search-Based Software Engineering pp 255–261Cite as

Defects4J as a Challenge Case for the Search-Based Software Engineering Community

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNPSE,volume 12420))

Abstract

Defects4J is a collection of reproducible bugs, extracted from real-world Java software systems, together with a supporting infrastructure for using these bugs. Defects4J has been widely used to evaluate software engineering research, including research on automated test generation, program repair, and fault localization. Defects4J has recently grown substantially, both in number of software systems and number of bugs. This report proposes that Defects4J can serve as a benchmark for Search-Based Software Engineering (SBSE) research as well as a catalyst for new innovations. Specifically, it outlines the current Defects4J dataset and infrastructure, and details how it can serve as a challenge case to support SBSE research and to expand Defects4J itself.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    The entire process is documented at https://github.com/rjust/defects4j/tree/master/framework/bug-mining/README.md.

References

  1. Cobertura. https://cobertura.github.io/cobertura/

  2. Almulla, H., Salahirad, A., Gay, G.: Using search-based test generation to discover real faults in Guava. In: Menzies, T., Petke, J. (eds.) SSBSE 2017. LNCS, vol. 10452, pp. 153–160. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66299-2_13

    Chapter  Google Scholar 

  3. Dantas, A., de Souza, E.F., Souza, J., Camilo-Junior, C.G.: Code naturalness to assist search space exploration in search-based program repair methods. In: Nejati, S., Gay, G. (eds.) SSBSE 2019. LNCS, vol. 11664, pp. 164–170. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-27455-9_12

    Chapter  Google Scholar 

  4. Fraser, G., Staats, M., McMinn, P., Arcuri, A., Padberg, F.: Does automated unit test generation really help software testers? A controlled empirical study. ACM Trans. Softw. Eng. Methodol. 24(4), 23:1–23:49 (2015). https://doi.org/10.1145/2699688

  5. Gay, G.: Challenges in using search-based test generation to identify real faults in mockito. In: Sarro, F., Deb, K. (eds.) SSBSE 2016. LNCS, vol. 9962, pp. 231–237. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-47106-8_17

    Chapter  Google Scholar 

  6. Colanzi, T.E., McMinn, P. (eds.): SSBSE 2018. LNCS, vol. 11036. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99241-9

    Book  Google Scholar 

  7. Harman, M., Jia, Y., Langdon, W.B.: Babel pidgin: SBSE can grow and graft entirely new functionality into a real world system. In: Le Goues, C., Yoo, S. (eds.) SSBSE 2014. LNCS, vol. 8636, pp. 247–252. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-09940-8_20

    Chapter  Google Scholar 

  8. Just, R.: The major mutation framework: efficient and scalable mutation analysis for java. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, pp. 433–436. ACM, New York (2014). https://doi.org/10.1145/2610384.2628053

  9. Just, R., Jalali, D., Ernst, M.D.: Defects4J: a database of existing faults to enable controlled testing studies for Java programs. In: Proceedings of the 2014 International Symposium on Software Testing and Analysis, ISSTA 2014, pp. 437–440. ACM, New York (2014). https://doi.org/10.1145/2610384.2628055

  10. Just, R., Parnin, C., Drosos, I., Ernst, M.D.: Comparing developer-provided to user-provided tests for fault localization and automated program repair. In: Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), pp. 287–297 (2018)

    Google Scholar 

  11. Martinez, M., Durieux, T., Sommerard, R., Xuan, J., Monperrus, M.: Automatic repair of real bugs in java: a large-scale experiment on the defects4j dataset. Empirical Softw. Eng. 22(4), 1936–1964 (2016). https://doi.org/10.1007/s10664-016-9470-4

    Article  Google Scholar 

  12. Motwani, M., Soto, M., Brun, Y., Just, R., Le Goues, C.: Quality of automated program repair on real-world defects. IEEE Trans. Softw. Eng. PP, 1 (2020)

    Article  Google Scholar 

  13. de Oliveira Barros, M., de Almeida Farzat, F.: What can a big program teach us about optimization? In: Ruhe, G., Zhang, Y. (eds.) SSBSE 2013. LNCS, vol. 8084, pp. 275–281. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-39742-4_24

    Chapter  Google Scholar 

  14. Pacheco, C., Ernst, M.D.: Randoop: feedback-directed random testing for java. In: Companion to the 22nd ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications Companion, OOPSLA 2007, pp. 815–816. ACM, New York (2007). https://doi.org/10.1145/1297846.1297902

  15. Panichella, A.: A systematic comparison of search algorithms for topic modelling—a study on duplicate bug report identification. In: Nejati, S., Gay, G. (eds.) SSBSE 2019. LNCS, vol. 11664, pp. 11–26. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-27455-9_2

    Chapter  Google Scholar 

  16. Pearson, S., et al.: Evaluating and improving fault localization. In: 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), pp. 609–620 (2017)

    Google Scholar 

  17. Petke, J., Brownlee, A.E.I.: Software improvement with Gin: a case study. In: Nejati, S., Gay, G. (eds.) SSBSE 2019. LNCS, vol. 11664, pp. 183–189. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-27455-9_14

    Chapter  Google Scholar 

  18. Rueda, U., Just, R., Galeotti, J.P., Vos, T.E.: Unit testing tool competition: round four. In: Proceedings of the International Workshop on Search-Based Software Testing (SBST), pp. 19–28, May 2016

    Google Scholar 

  19. Sarro, F.: Search-based predictive modelling for software engineering: how far have we gone? In: Nejati, S., Gay, G. (eds.) SSBSE 2019. LNCS, vol. 11664, pp. 3–7. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-27455-9_1

    Chapter  Google Scholar 

  20. Shamshiri, S., Just, R., Rojas, J.M., Fraser, G., McMinn, P., Arcuri, A.: Do automatically generated unit tests find real faults? An empirical study of effectiveness and challenges. In: Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), ASE 2015. ACM, New York (2015)

    Google Scholar 

  21. Vogel, T., Tran, C., Grunske, L.: Does diversity improve the test suite generation for mobile applications? In: Nejati, S., Gay, G. (eds.) SSBSE 2019. LNCS, vol. 11664, pp. 58–74. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-27455-9_5

    Chapter  Google Scholar 

  22. Zamani, S., Hemmati, H.: Revisiting hyper-parameter tuning for search-based test data generation. In: Nejati, S., Gay, G. (eds.) SSBSE 2019. LNCS, vol. 11664, pp. 137–152. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-27455-9_10

    Chapter  Google Scholar 

  23. Zhang, Y., Harman, M., Jia, Y., Sarro, F.: Inferring test models from Kate’s bug reports using multi-objective search. In: Barros, M., Labiche, Y. (eds.) SSBSE 2015. LNCS, vol. 9275, pp. 301–307. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-22183-0_27

    Chapter  Google Scholar 

Download references

Acknowledgements

This material is based upon work supported by the National Science Foundation under grants CNS-1823172 and CCF-1942055.

Author information

Authors and Affiliations

  1. Chalmers and the University of Gothenburg, Gothenburg, Sweden

    Gregory Gay

  2. University of Washington, Seattle, WA, USA

    René Just

Authors
  1. Gregory Gay
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. René Just
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gregory Gay .

Editor information

Editors and Affiliations

  1. Monash University, Melbourne, VIC, Australia

    Aldeida Aleti

  2. Delft University of Technology, Delft, The Netherlands

    Annibale Panichella

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gay, G., Just, R. (2020). Defects4J as a Challenge Case for the Search-Based Software Engineering Community. In: Aleti, A., Panichella, A. (eds) Search-Based Software Engineering. SSBSE 2020. Lecture Notes in Computer Science(), vol 12420. Springer, Cham. https://doi.org/10.1007/978-3-030-59762-7_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-59762-7_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-59761-0

  • Online ISBN: 978-3-030-59762-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation