Skip to main content

Advertisement

Springer Nature Link
Log in

DALO-APR: LLM-based automatic program repair with data augmentation and loss function optimization

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Automatic program repair (APR) has made significant strides with the advent of large language models (LLMs) such as T5 and CodeT5. However, LLM-based APR models may rely on repetitive repair patterns due to limited training data diversity, resulting in suboptimal performance. Additionally, common loss functions, such as cross-entropy, may not fully prioritize repair locations or optimize the model’s output probability distribution to favor more accurate repair candidates. To address these challenges, this paper proposes a method for LLM-Based APR with Data Augmentation and Loss Function Optimization (DALO-APR). The data augmentation strategy expands the variety of repair patterns by randomly deleting, inserting, swapping tokens, and injecting errors. The optimized loss function helps the model rank more accurate repair candidates higher. Experimental results on Java, JavaScript, Python, and C datasets demonstrate that DALO-APR improves both error localization and bug fixing. Compared to baseline models, DALO-APR shows improvements across multiple metrics, especially with a 105.65% increase in 100% accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Algorithm 2
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Liu X, Zheng Y, Du Z, Ding M, Qian Y, Yang Z, Tang J (2023) Gpt understands, too. AI Open

  2. Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu PJ (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res 21(140):1–67

    MathSciNet  Google Scholar 

  3. Fan Z, Gao X, Mirchev M, Roychoudhury A, Tan SH (2023) Automated repair of programs from large language models. In: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 1469–1481. IEEE

  4. Mao A, Mohri M, Zhong Y (2023) Cross-entropy loss functions: Theoretical analysis and applications. In: International Conference on Machine Learning, pp. 23803–23828. PMLR

  5. Horváth D, Csuvik V, Gyimóthy T, Vidács L (2023) An extensive study on model architecture and program representation in the domain of learning-based automated program repair. In: 2023 IEEE/ACM International Workshop on Automated Program Repair (APR), pp. 31–38. IEEE

  6. Zhu Q, Sun Z, Xiao Y-a, Zhang W, Yuan K, Xiong Y, Zhang L (2021) A syntax-guided edit decoder for neural program repair. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 341–353

  7. Xia CS, Wei Y, Zhang L (2023) Automated program repair in the era of large pre-trained language models. In: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 1482–1494. IEEE

  8. Ye H, Martinez M, Durieux T, Monperrus M (2021) A comprehensive study of automatic program repair on the quixbugs benchmark. J Syst Softw 171:110825

    Google Scholar 

  9. Jiang N, Lutellier T, Tan L (2021) Cure: Code-aware neural machine translation for automatic program repair. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp. 1161–1173. IEEE

  10. Ye H, Monperrus M (2024) Iter: Iterative neural repair for multi-location patches. In: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp. 1–13

  11. Liu Z, Zhu S, Qin B, Chen H, Song L (2021) Automatically detecting and fixing concurrency bugs in go software systems. In: Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, pp. 616–629

  12. Lemieux C, Inala JP, Lahiri SK, Sen S (2023) Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 919–931. IEEE

  13. Chen Z, Kommrusch S, Monperrus M (2022) Neural transfer learning for repairing security vulnerabilities in c code. IEEE Trans Softw Eng 49(1):147–165

    Google Scholar 

  14. Gissurarson MP, Applis L, Panichella A, Deursen A, Sands D (2022) Propr: property-based automatic program repair. In: Proceedings of the 44th International Conference on Software Engineering, pp. 1768–1780

  15. Winter E, Nowack V, Bowes D, Counsell S, Hall T, Haraldsson S, Woodward J (2022) Let’s talk with developers, not about developers: A review of automatic program repair research. IEEE Trans Softw Eng 49(1):419–436

    Google Scholar 

  16. Jiang N, Liu K, Lutellier T, Tan L (2023) Impact of code language models on automated program repair. In: 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 1430–1442. IEEE

  17. Kim Y, Han S, Khamit AY, Yi J (2023) Automated program repair from fuzzing perspective. In: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 854–866

  18. Lajkó M, Csuvik V, Gyimothy T, Vidács L (2024) Automated program repair with the gpt family, including gpt-2, gpt-3 and codex. In: Proceedings of the 5th ACM/IEEE International Workshop on Automated Program Repair, pp. 34–41

  19. Zhang J, Wang X, Zhang H, Sun H, Wang K, Liu X (2019) A novel neural source code representation based on abstract syntax tree. In: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 783–794. IEEE

  20. Dinella E, Dai H, Li Z, Naik M, Song L, Wang K (2020) Hoppity: Learning graph transformations to detect and fix bugs in programs. In: International Conference on Learning Representations (ICLR)

  21. Jensen SH, Jonsson PA, Møller A (2012) Remedying the eval that men do. In: Proceedings of the 2012 International Symposium on Software Testing and Analysis, pp. 34–44

  22. Csuvik V, Horváth D, Lajkó M, Vidács L (2021) Exploring plausible patches using source code embeddings in javascript. In: 2021 IEEE/ACM International Workshop on Automated Program Repair (APR), pp. 11–18. IEEE

  23. Hu Y, Shi X, Zhou Q, Pike L (2022) Fix bugs with transformer through a neural-symbolic edit grammar. arXiv preprint arXiv:2204.06643

  24. Chen J, Tam D, Raffel C, Bansal M, Yang D (2023) An empirical survey of data augmentation for limited data learning in nlp. Trans Assoc Comput Linguistics 11:191–211

    Google Scholar 

  25. Li B, Hou Y, Che W (2022) Data augmentation approaches in natural language processing: A survey. Ai Open 3:71–90

    MATH  Google Scholar 

  26. Orvalho P, Janota M, Manquinho V (2022) Multipas: applying program transformations to introductory programming assignments for data augmentation. In: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1657–1661

  27. Chowdhary K, Chowdhary K (2020) Natural language processing. Fundamentals of artificial intelligence, pp. 603–649

  28. Xie H, Lei Y, Yan M, Yu Y, Xia X, Mao X (2022) A universal data augmentation approach for fault localization. In: Proceedings of the 44th International Conference on Software Engineering, pp. 48–60

  29. Wang Y, Wang W, Joty S, Hoi SC (2021) Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859

  30. Liu Y, Tantithamthavorn C, Liu Y, Li L (2024) On the reliability and explainability of language models for program generation. ACM Trans Softw Eng Methodol 33(5):1–26

    MATH  Google Scholar 

  31. Xu C, Zhu Z, Wang J, Wang J, Zhang W (2024) Fairly evaluating large language model-based recommendation needs revisit the cross-entropy loss. arXiv preprint arXiv:2402.06216

  32. Zhuang H, Qin Z, Jagerman R, Hui K, Ma J, Lu J, Ni J, Wang X, Bendersky M (2023) Rankt5: Fine-tuning t5 for text ranking with ranking losses. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2308–2313

  33. Tufano M, Watson C, Bavota G, Penta MD, White M, Poshyvanyk D (2019) An empirical study on learning bug-fixing patches in the wild via neural machine translation. ACM Trans Softw Eng Methodol (TOSEM) 28(4):1–29

    Google Scholar 

  34. Csuvik V, Vidács L (2022) Fixjs: A dataset of bug-fixing javascript commits. In: Proceedings of the 19th International Conference on Mining Software Repositories, pp. 712–716

  35. Lin D, Koppel J, Chen A, Solar-Lezama A (2017) Quixbugs: A multi-lingual program repair benchmark set based on the quixey challenge. In: Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity, pp. 55–56

  36. Tan SH, Yi J, Mechtaev S, Roychoudhury A, et al (2017) Codeflaws: a programming competition benchmark for evaluating automated program repair tools. In: 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSE-C), pp. 180–182. IEEE

  37. Le Goues C, Nguyen T, Forrest S, Weimer W (2011) Genprog: A generic method for automatic software repair. Ieee Trans Softw Eng 38(1):54–72

    MATH  Google Scholar 

  38. Liu K, Koyuncu A, Kim D, Bissyandé TF (2019) Tbar: Revisiting template-based automated program repair. In: Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 31–42

  39. Jiang N, Lutellier T, Tan L (2021) Cure: Code-aware neural machine translation for automatic program repair. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pp. 1161–1173. https://doi.org/10.1109/ICSE43902.2021.00107

  40. Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M, et al (2020) Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45

  41. Ansel J, Yang E, al H (2024) PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In: 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’24). ACM, ???. https://doi.org/10.1145/3620665.3640366 . https://pytorch.org/assets/pytorch2-2.pdf

  42. Falcon W. The PyTorch Lightning team: PyTorch Lightning. https://doi.org/10.5281/zenodo.3828935 . https://github.com/Lightning-AI/lightning

  43. Wu T, He S, Liu J, Sun S, Liu K, Han Q-L, Tang Y (2023) A brief overview of chatgpt: The history, status quo and potential future development. IEEE/CAA J Autom Sinica 10(5):1122–1136

    MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by the Key Field Research and Development Plan of Guangdong Province (2022B0101070001), Pazhou Labs Second Batch of Cultivation Projects in 2022 (PZL2022KF0008), and Guangdong Natural Science Fund Project (2022A1515110564).

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510006, Guangdong, China

    Shaosheng Wang & Lu Lu

  2. Guangdong Artificial Intelligence and Digital Economy Laboratory (Guangzhou), Guangzhou, 510320, Guangdong, China

    Shaosheng Wang

  3. Pengcheng Laboratory, Shenzhen, 518055, Guangdong, China

    Lu Lu

  4. College of Mathematics and Informatics, South China Agricultural University, Guangzhou, 510642, Guangdong, China

    Shaojian Qiu

  5. Guangdong Provincial Key Laboratory of Tunnel Engineering Safety and Emergency Support Technology and Equipment Enterprises, Guangzhou, 510440, Guangdong, China

    Qingyan Tian & Haishan Lin

Authors
  1. Shaosheng Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Lu Lu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Shaojian Qiu
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Qingyan Tian
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Haishan Lin
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Lu Lu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, S., Lu, L., Qiu, S. et al. DALO-APR: LLM-based automatic program repair with data augmentation and loss function optimization. J Supercomput 81, 640 (2025). https://doi.org/10.1007/s11227-025-07102-3

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11227-025-07102-3

Keywords