Skip to main content
SpringerLink
Log in

Where were the repair ingredients for Defects4j bugs?

Exploring the impact of repair ingredient retrieval on the performance of 24 program repair systems

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

A significant body of automated program repair research has built approaches under the redundancy assumption. Patches are then heuristically generated by leveraging repair ingredients (change actions and donor code) that are found in code bases (either the buggy program itself or big code). For example, common change actions (i.e., fix patterns) are frequently mined offline and serve as an important ingredient for many patch generation engines. Although the repetitiveness of code changes has been studied in general, the literature provides little insight into the relationship between the performance of the repair system and the source code base where the change actions were mined. Similarly, donor code is another important repair ingredient to concretize patches guided by abstract patterns. Yet, little attention has been paid to where such ingredients can actually be found. Through a large scale empirical study on the execution results of 24 repair systems evaluated on real-world bugs from Defects4J, we provide a comprehensive view on the distribution of repair ingredients that are relevant for these bugs. In particular, we show that (1) a half of bugs cannot be fixed simply because the relevant repair ingredient is not available in the search space of donor code; (2) bugs that are correctly fixed by literature tools are mostly addressed with shallow change actions; (3) programs with little history of changes can benefit from mining change actions in other programs; (4) parts of donor code to repair a given bug can be found separately at different search locations; (5) bug-triggering test cases are a rich source for donor code search.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

Data Availability

All relevant artifacts of our study are publicly available at:

https://github.com/DehengYang/repair-ingredients.

Notes

  1. ReturnStatement, IfStatement, etc.

  2. The revision history may not be able to explore for some projects but most APR studies (Martinez et al. 2014; Pan et al. 2009; Barr et al. 2014; Nguyen et al. 2013; Liu et al. 2018; Zhong and Su 2015) assume that the revision history is available. The revision history in our study is also available as Defects4J bugs are all collected from real-world and large-scale projects.

  3. https://github.com/AutoProRepair/PatchParser.git

  4. https://github.com/program-repair/defects4j-dissection/blob/master/defects4j-patch.md

  5. In contrast with the mining of code change actions, most APR systems scan only the production code of a program for donor code.

  6. This number is different from 169 shown in Fig. 6 since the revisions related to Chart-11 and 26 are not available to explore as stated in the above paragraphs.

References

  • Barr ET, Brun Y, Devanbu PT, Harman M, Sarro F (2014) The plastic surgery hypothesis. In: Proceedings of the 22nd ACM SIGSOFT International symposium on foundations of software engineering. ACM, pp 306–317

  • Britton T, Jeng L, Carver G, Cheak P, Katzenellenbogen T (2013) Reversible debugging software. Judge Bus. School, Univ. Cambridge, Cambridge, UK, Tech. Rep

  • Chen L, Pei Y, Furia CA (2017) Contract-based program repair without the contracts. In: Proceedings of the 32nd IEEE/ACM international conference on automated software engineering. IEEE, pp 637–647

  • Durieux T, Cornu B, Seinturier L, Monperrus M (2017) Dynamic patch generation for null pointer exceptions using metaprogramming. In: Proceedings of the 24th international conference on software analysis, evolution and reengineering. IEEE, pp 349–358

  • Durieux T, Madeiral F, Martinez M, Abreu R (2019) Empirical review of java program repair tools: A large-scale experiment on 2,141 bugs and 23,551 repair attempts. In: Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, pp 302–313

  • Falleri J-R, Morandat F, Blanc X, Martinez M, Monperrus M (2014) Fine-grained and accurate source code differencing. In: Proceedings of the 29th ACM/IEEE International conference on automated software engineering. ACM, pp 313–324

  • Gazzola L, Micucci D, Mariani L (2017) Automatic software repair: A survey. IEEE Trans Softw Eng 45(1):34–67

    Article  Google Scholar 

  • Ghanbari A, Benton S, Zhang L (2019) Practical program repair via bytecode mutation. In: Proceedings of the 28th ACM SIGSOFT International symposium on software testing and analysis. ACM, pp 19–30

  • GitHub (2021) Defects4j. https://github.com/rjust/defects4j/releases/tag/v1.4.0

  • Goues CL, Dewey-Vogt M, Forrest S, Weimer W (2012) A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each. In: Proceedings of the 34th International conference on software engineering. IEEE, pp 3–13

  • Goues CL, Nguyen TV, Forrest S, Weimer W (2012) GenProg: A generic method for automatic software repair. IEEE Trans Softw Eng 38(1):54–72

    Article  Google Scholar 

  • Goues CL, Pradel M, Roychoudhury A (2019) Automated program repair. Commun ACM 62(12):56–65

    Article  Google Scholar 

  • Gupta R, Pal S, Kanade A, Shevade S (2017) Deepfix: Fixing common c language errors by deep learning. In: Proceedings of the 31st AAAI conference on artificial intelligence. AAAI, pp 1345–1351

  • Hovemeyer D, Pugh W (2004) Finding bugs is easy. ACM sigplan notices 39(12):92–106

    Article  Google Scholar 

  • Hua J, Zhang M, Wang K, Khurshid S (2018) Towards practical program repair with on-demand candidate generation. In: Proceedings of the 40th international conference on software engineering. ACM, pp 12–23

  • Jiang L, Misherghi G, Su Z, Glondum S (2007) Deckard: Scalable and accurate tree-based detection of code clones. In: 29th International conference on software engineering (ICSE’07). IEEE, pp 96–105

  • Jiang J, Ren L, Xiong Y, Zhang L (2019) Inferring program transformations from singular examples via big code. In: Proceedings of the 34th IEEE/ACM International conference on automated software engineering. pp 255–266

  • Jiang J, Xiong Y, Xia X (2019) A manual inspection of defects4j bugs and its implications for automatic program repair. Sci Chin Inf Sci 62(10):200102

    Article  Google Scholar 

  • Jiang J, Xiong Y, Zhang H, Gao Q, Chen X (2018) Shaping program repair space with existing patches and similar code. In: Proceedings of the 27th ACM SIGSOFT International symposium on software testing and analysis. ACM, pp 298–309

  • Just R, Jalali D, Ernst MD (2014) Defects4J: A database of existing faults to enable controlled testing studies for java programs. In: Proceedings of the 2014 International symposium on software testing and analysis. ACM, pp 437–440

  • Just R, Parnin C, Drosos I, Ernst MD (2018) Comparing developer-provided to user-provided tests for fault localization and automated program repair. In: Proceedings of the 27th ACM SIGSOFT International symposium on software testing and analysis. ACM, pp 287–297

  • Kim J, Kim S (2019) Automatic patch generation with context-based change application. Empir Softw Eng 24(6):4071–4106

    Article  Google Scholar 

  • Kim D, Nam J, Song J, Kim S (2013) Automatic patch generation learned from human-written patches. In: Proceedings of the 35th International conference on software engineering. IEEE, pp 802–811

  • Koyuncu A, Liu K, Bissyandé TF, Kim D, Klein J, Monperrus M, Traon YL (2019) Fixminer: Mining relevant fix patterns for automated program repair. Empir Softw Eng 25(3):1980–2024

    Article  Google Scholar 

  • Koyuncu A, Liu K, Bissyandé TF, Kim D, Monperrus M, Klein J, Traon YL (2019) iFixR: Bug report driven program repair. In: Proceedings of the 27the ACM joint european software engineering conference and symposium on the foundations of software engineering. ACM, pp 314–325

  • Le X-BD, Bao L, Lo D, Xia X, Li S, Pasareanu C (2019) On reliability of patch correctness assessment. In: Proceedings of the 41st International conference on software engineering. IEEE, pp 524–535

  • Le XBD, Lo D, Goues CL (2016) History driven program repair. In: Proceedings of the 23rd IEEE International conference on software analysis, evolution, and reengineering. IEEE, pp 213–224

  • Liu K, Kim D, Bissyandé TF, Yoo S, Traon YL (2018) Mining fix patterns for findbugs violations. IEEE Trans Softw Eng 47(1):165–188

    Article  Google Scholar 

  • Liu K, Kim D, Koyuncu A, Li L, Bissyandé TF, Traon YL (2018) A closer look at real-world patches. In: 2018 IEEE International conference on software maintenance and evolution (ICSME). IEEE, pp 275–286

  • Liu K, Koyuncu A, Bissyandé TF, Kim D, Klein J, Traon YL (2019) You cannot fix what you cannot find! an investigation of fault localization bias in benchmarking automated program repair systems. In: Proceedings of the 12th IEEE International conference on software testing, verification and validation. IEEE, pp 102–113

  • Liu K, Koyuncu A, Kim D, Bissyandé TF (2019) TBar: Revisiting template-based automated program repair. In: Proceedings of the 28th ACM SIGSOFT international symposium on software testing and analysis. ACM, pp 31–42

  • Liu K, Koyuncu A, Kim D, Bissyandé TF (2019) Avatar: Fixing semantic bugs with fix patterns of static analysis violations. In: Proceedings of the 26th IEEE international conference on software analysis, evolution and reengineering. IEEE, pp 456–467

  • Liu K, Koyuncu A, Kim K, Kim D, Bissyandé TF (2018) LSRepair: Live search of fix ingredients for automated program repair. In: Proceedings of the 25th Asia-Pacific Software Engineering Conference. IEEE, pp 658–662

  • Liu K, Li L i, Koyuncu A, Kim D, Liu Z, Klein J, Bissyandé TF (2021) A critical review on the evaluation of automated program repair systems. J Syst Softw 171:110817

    Article  Google Scholar 

  • Liu K, Wang S, Koyuncu A, Kim K, Bissyandé TF, Kim D, Wu P, Klein J, Mao X, Traon YL (2020) On the efficiency of test suite based program repair: A systematic assessment of 16 automated repair systems for java programs. In: Rothermel G, Bae D-H (eds) Proceedings of the 42nd International conference on software engineering. ACM, pp 615–627

  • Liu X, Zhong H (2018) Mining stackoverflow for program repair. In: Proceedings of the 25th IEEE international conference on software analysis, evolution and reengineering. IEEE, pp 118–129

  • Long F, Amidon P, Rinard M (2017) Automatic inference of code transforms for patch generation. In: Proceedings of the 11th joint meeting on foundations of software engineering. ACM, pp 727–739

  • Long F, Rinard M (2016) An analysis of the search spaces for generate and validate patch generation systems. In: Proceedings of the 38th International conference on software engineering. IEEE, pp 702–713

  • Lou Y, Chen J, Zhang L, Hao D, Zhang L (2019) History-driven build failure fixing: how far are we?. In: Proceedings of the 28th ACM SIGSOFT International symposium on software testing and analysis. ACM, pp 43–54

  • Martinez M, Durieux T, Sommerard R, Xuan J, Monperrus M (2017) Automatic repair of real bugs in java A large-scale experiment on the defects4j dataset. Empir Softw Eng 22(4):1936–1964

    Article  Google Scholar 

  • Martinez M, Martin Monperrus. (2016) Astor: A program repair library for java. In: Proceedings of the 25th International symposium on software testing and analysis. ACM, pp 441–444

  • Martinez M, Monperrus M (2015) Mining software repair models for reasoning on the search space of automated program fixing. Empir Softw Eng 20 (1):176–205

    Article  Google Scholar 

  • Martinez M, Weimer W, Monperrus M (2014) Do the fix ingredients already exist? an empirical inquiry into the redundancy assumptions of program repair approaches. In: Companion proceedings of the 36th international conference on software engineering. ACM, pp 492–495

  • Mockus A, Votta LG (2000) Identifying reasons for software changes using historic databases. In: Proceedings of the international conference on software maintenance. pp 120–130

  • Monperrus M (2018) Automatic software repair: A bibliography. ACM Comput Surv 51(1):17:1–17:24

    Article  Google Scholar 

  • Motwani M, Sankaranarayanan S, Just R, Brun Y (2018) Do automated program repair techniques repair hard and important bugs? Empir Softw Eng 23(5):2901–2947

    Article  Google Scholar 

  • Nguyen HA, Nguyen AT, Nguyen TT, Nguyen TN, Rajan H (2013) A study of repetitiveness of code changes in software evolution. In: 2013 28th IEEE/ACM International conference on automated software engineering (ASE), pp 180–190

  • Pan K, Kim S, James Whitehead E (2009) Toward an understanding of bug fix patterns. Empir Softw Eng 14(3):286–315

    Article  Google Scholar 

  • Qi Z, Long F, Achour S, Rinard M (2015) An analysis of patch plausibility and correctness for generate-and-validate patch generation systems. In: Proceedings of the international symposium on software testing and analysis. ACM, pp 24–36

  • Qi Y, Mao X, Lei Y, Dai Z, Wang C (2014) The strength of random search on automated program repair. In: Proceedings of the 36th International Conference on Software Engineering. ACM, pp 254–265

  • Qi Y, Mao X, Lei Y, Wang C (2013) Using automated program repair for evaluating the effectiveness of fault localization techniques. In: Proceedings of the 22nd International symposium on software testing and analysis. ACM, pp 191–201

  • Qi X, Reiss SP (2017) Leveraging syntax-related code for automated program repair. In: Proceedings of the 32nd IEEE/ACM international conference on automated software engineering. IEEE, pp 660–670

  • Qin Y, Wang S, Liu K, Mao X, Bissyandé TF (2021) On the impact of flaky tests in automated program repair. In: Proceedings of the 28th IEEE International conference on software analysis, evolution and reengineering. IEEE, pp 295–306

  • Rolim R, Soares G, Gheyi R, D’Antoni L (2018) Learning quick fixes from code repositories. arXiv preprint arXiv:1803.03806

  • Roy CK, Cordy JR, Koschke R (2009) Comparison and evaluation of code clone detection techniques and tools: A qualitative approach. Sci Comput Programm 74(7):470–495

    Article  MathSciNet  Google Scholar 

  • Saha RK, Lyu Y, Yoshida H, Prasad MR (2017) Elixir: Effective object-oriented program repair. In: Proceedings of the 32nd IEEE/ACM International conference on automated software engineering. IEEE, pp 648–659

  • Saha S, Saha RK, Prasad MR (2019) Harnessing evolution for multi-hunk program repair. In: Proceedings of the 41st international conference on software engineering. IEEE, pp 13–24

  • Sobreira V, Durieux T, Madeiral F, Monperrus M, de Almeida Maia M (2018) Dissection of a bug dataset: Anatomy of 395 patches from defects4j. In: Proceedings of the IEEE 25th International conference on software analysis, evolution and reengineering. IEEE, pp 130–140

  • Tian H, Liu K, Kaboré AK, Koyuncu A, Li L, Klein J, Bissyandé TF (2020) Evaluating representation learning of code changes for predicting patch correctness in program repair. In: Proceedings of the 35th IEEE/ACM International conference on automated software engineering. IEEE, pp 981–992

  • Weimer W, Nguyen TV, Goues CL, Forrest S (2009) Automatically finding patches using genetic programming. In: Proceedings of the 31st international conference on software engineering. IEEE, pp 364–374

  • Wen M, Chen J, Wu R, Hao D, Cheung S-C (2018) Context-aware patch generation for better automated program repair. In: Proceedings of the 40th International conference on software engineering. ACM, pp 1–11

  • Xiong Y, Liu X, Zeng M, Zhang L, Huang G (2018) Identifying patch correctness in test-based program repair. In: Proceedings of the 40th International conference on software engineering. ACM, pp 789–799

  • Xiong Y, Wang J, Yan R, Zhang J, Han S, Huang G, Zhang L (2017) Precise condition synthesis for program repair. In: Proceedings of the 39th IEEE/ACM international conference on software engineering. IEEE, pp 416–426

  • Xu X, Sui Y, Yan H, Xue J (2019) Vfix: value-flow-guided precise program repair for null pointer dereferences. In: Proceedings of the 41st International Conference on Software Engineering. IEEE, pp 512–523

  • Xuan J, Martinez M, Demarco F, Clement M, Marcote SL, Durieux T, Le Berre D, Monperrus M (2017) Nopol: Automatic repair of conditional statement bugs in java programs. IEEE Trans Softw Eng 43(1):34–55

    Article  Google Scholar 

  • Xuan J, Monperrus M (2014) Test case purification for improving fault localization. In: Proceedings of the 22nd ACM SIGSOFT International symposium on foundations of software engineering. ACM, pp 52–63

  • Yang D, Qi Y, Mao X (2017) An empirical study on the usage of fault localization in automated program repair. In: Proceedings of the 33rd IEEE International conference on software maintenance and evolution. IEEE, pp 504–508

  • Yang D, Qi Y, Mao X, Lei Y (2021) Evaluating the usage of fault localization in automated program repair: an empirical study. Front Comput Sci 15 (1):1–15

    Google Scholar 

  • Yuan Y, Banzhaf W (2018) Arja: Automated repair of java programs via multi-objective genetic programming. IEEE Trans Softw Eng

  • Yue R, Na M, Wang Q (2017) A characterization study of repeated bug fixes. In: 2017 IEEE International conference on software maintenance and evolution (ICSME). IEEE, pp 422–432

  • Zhong H, Su Z (2015) An empirical study on real bug fixes. In: Proceedings of the 37th international conference on software engineering. IEEE, pp 913–923

Download references

Acknowledgements

This research was supported by the Open Project Program of the State Key Laboratory of Mathematical Engineering and Advanced Computing (No. 2020A06) and the Open Project Program of the Key Laboratory of Safety-Critical Software (Nanjing University of Aeronautics and Astronautics), Ministry of Industry and Information Technology (No. XCA20026), the National Key R&D Program of China (No. 2020AAA0107704), the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2021R1A5A1021944), the National Defense Basic Scientific Research Program (No. WDZC20205500308), the Fundamental Research Funds for the Central Universities (Nos. 2021CDJKYJH032, 2020CDCGRJ037, 2020CDCGRJ072) and the National Natural Science Foundation of China (Nos. 61872445, 61672529), as well as the funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 949014).

Author information

Authors and Affiliations

  1. National University of Defense Technology, Changsha, China

    Deheng Yang & Xiaoguang Mao

  2. Nanjing University of Aeronautics and Astronautics, Nanjing, China

    Kui Liu

  3. State Key Laboratory of Mathematical Engineering and Advanced Computing, Wuxi, China

    Kui Liu

  4. Key Laboratory of Safety-Critical Software (Nanjing University of Aeronautics and Astronautics), Ministry of Industry and Information Technology, Nanjing, China

    Kui Liu

  5. Kyungpook National University, Daegu, South Korea

    Dongsun Kim

  6. Sabanci University, Istanbul, Turkey

    Anil Koyuncu

  7. SnT, University of Luxembourg, Luxembourg City, Luxembourg

    Kisub Kim, Haoye Tian, Jacques Klein & Tegawendé F. Bissyandé

  8. Chongqing University, Chongqing, China

    Yan Lei

Authors
  1. Deheng Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kui Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dongsun Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anil Koyuncu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kisub Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Haoye Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yan Lei
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xiaoguang Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jacques Klein
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Tegawendé F. Bissyandé
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kui Liu.

Additional information

Communicated by: Saurabh Sinha

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, D., Liu, K., Kim, D. et al. Where were the repair ingredients for Defects4j bugs?. Empir Software Eng 26, 122 (2021). https://doi.org/10.1007/s10664-021-10003-7

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-021-10003-7

Keywords

Search

Navigation