Skip to main content

Advertisement

Leveraging meta-data of code for adapting prompt tuning for code summarization

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Prompt tuning alleviates the gap between pre-training and fine-tuning and achieves promising results in various natural language processing (NLP) tasks. However, it is nontrivial for adapting prompt tuning in intelligent code tasks since code-specific knowledge such as abstract syntax tree is usually hierarchy-structured and therefore is hard to be converted into plain text. Recent works (e.g., PT4Code) introduce simple task prompts along with a programming language indicator into prompt template, achieving improvement over non-prompting state-of-the-art code models (e.g., CodeT5). Inspired by this, we propose a novel code-specific prompt paradigm, meta-data prompt, which introduces semi-structured code’s meta-data (attribute-value pairs) into prompt template and facilitates the adaption of prompt tuning techniques into code tasks. Specifically, we find the usage of diverse meta-data attributes and their combinations and employ the OpenPrompt to implement a meta-data prompt based code model, PRIME (PRompt tunIng with MEta-data), via utilizing CodeT5 as the backbone model. We experiment PRIME with the source code summarization task on the publicly available CodeSearchNet benchmark. Results show that 1) using good meta-data can lead to an improvement on the model performance; 2) the proposed meta-data prompt can be combined with traditional task prompt for further improvement; 3) our best-performing model can consistently outperform CodeT5 by an absolute score of 0.73 and PT4Code by an absolute score of 0.48 regarding the averaged BLEU metric across six programming languages.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. Note that method is released after the submission of our manuscript. At the time of submission, our model PRIME had the best performance.

  2. https://github.com/salesforce/CodeT5

  3. https://github.com/adf1178/PT4Code

  4. https://huggingface.co/models

References

  1. Liu P, Yuan W, Fu J, Jiang Z, Hayashi H, Neubig G (2023) Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comput Surv 55(9):195–119535

    Article  Google Scholar 

  2. Gu Y, Han X, Liu Z, Huang M (2022) PPT: pre-trained prompt tuning for few-shot learning. In: Muresan S, Nakov P, Villavicencio A (eds) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp 8410–8423. Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-long.576

  3. Schick T, Schütze H (2021) Exploiting cloze-questions for few-shot text classification and natural language inference. In: Merlo P, Tiedemann J, Tsarfaty R (eds) Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, pp 255–269. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.eacl-main.20

  4. Li XL, Liang P (2021) Prefix-tuning: Optimizing continuous prompts for generation. In: Zong C, Xia F, Li W, Navigli R (eds) Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp 4582–4597. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-long.353

  5. Guo D, Ren S, Lu S, Feng Z, Tang D, Liu S, Zhou L, Duan N, Svyatkovskiy A, Fu S, Tufano M, Deng SK, Clement CB, Drain D, Sundaresan N, Yin J, Jiang D, Zhou M (2021) Graphcodebert: Pre-training code representations with data flow. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, https://openreview.net/forum?id=jLoC4ez43PZ

  6. Niu C, Li C, Ng V, Ge J, Huang L, Luo B (2024) Passsum: leveraging paths of abstract syntax trees and self-supervision for code summarization. J Softw Evol Process 36(6)

  7. Wang C, Yang Y, Gao C, Peng Y, Zhang H, Lyu MR (2022) No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence. In: Roychoudhury A, Cadar C, Kim M (eds) Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, pp 382–394. ACM. https://doi.org/10.1145/3540250.3549113

  8. Peng H, Li G, Wang W, Zhao Y, Jin Z (2021) Integrating tree path in transformer for code representation. In: Ranzato M, Beygelzimer A, Dauphin YN, Liang P, Vaughan JW (eds) Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, Virtual, pp 9343–9354, https://proceedings.neurips.cc/paper/2021/hash/4e0223a87610176ef0d24ef6d2dcde3a-Abstract.html

  9. Husain H, Wu H, Gazit T, Allamanis M, Brockschmidt M (2019) Codesearchnet challenge: evaluating the state of semantic code search. arXiv:1909.09436

  10. Ding N, Hu S, Zhao W, Chen Y, Liu Z, Zheng H, Sun M (2022) Openprompt: An open-source framework for prompt-learning. In: Basile V, Kozareva Z, Stajner S (eds) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 - System Demonstrations, Dublin, Ireland, May 22-27, 2022, pp 105–113. Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-demo.10

  11. Wang Y, Wang W, Joty SR, Hoi SCH (2021) Codet 5: identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In: Moens M, Huang X, Specia L, Yih SW (eds) Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp 8696–8708. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.emnlp-main.685

  12. Feng Z, Guo D, Tang D, Duan N, Feng X, Gong M, Shou L, Qin B, Liu T, Jiang D, Zhou M (2020) Codebert: A pre-trained model for programming and natural languages. In: Cohn T, He Y, Liu Y (eds) Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020. Findings of ACL, vol. EMNLP 2020, pp 1536–1547. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings-emnlp.139

  13. Ahmad WU, Chakraborty S, Ray B, Chang K (2021) Unified pre-training for program understanding and generation. In: Toutanova K, Rumshisky A, Zettlemoyer L, Hakkani-Tür D, Beltagy I, Bethard S, Cotterell R, Chakraborty T, Zhou Y (eds) Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp 2655–2668. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.naacl-main.211

  14. Le H, Wang Y, Gotmare AD, Savarese S, Hoi SC (2022) Coderl: mastering code generation through pretrained models and deep reinforcement learning. In: NeurIPS. http://papers.nips.cc/paper_files/paper/2022/hash/8636419dea1aa9fbd25fc4248e702da4-Abstract-Conference.html

  15. Li M, Yu H, Fan G, Zhou Z, Huang Z (2024) Enhancing code summarization with action word prediction. Neurocomputing 563:126777

    Article  MATH  Google Scholar 

  16. Guo Y, Chai Y, Zhang L, Li H, Luo M, Guo S (2024) Context-based transfer learning for low resource code summarization. Softw. Pract. Exp. 54(3):465–482

    Article  MATH  Google Scholar 

  17. Tsimpoukelli M, Menick J, Cabi S, Eslami SMA, Vinyals O, Hill F (2021) Multimodal few-shot learning with frozen language models. In: Ranzato M, Beygelzimer A, Dauphin YN, Liang P, Vaughan JW (eds) Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, Virtual, pp 200–212. https://proceedings.neurips.cc/paper/2021/hash/01b7575c38dac42f3cfb7d500438b875-Abstract.html

  18. Kumar J, Chimalakonda S (2024) Code summarization without direct access to code - towards exploring federated llms for software engineering. In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, EASE 2024, Salerno, Italy, June 18-21, 2024, pp 100–109. ACM. https://doi.org/10.1145/3661167.3661210

  19. Ahmed T, Pai KS, Devanbu PT, Barr ET (2024) Automatic semantic augmentation of language model prompts (for code summarization). In: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pp 220–122013. ACM. https://doi.org/10.1145/3597503.3639183

  20. Su C, McMillan C (2024) Distilled GPT for source code summarization. Autom Softw Eng 31(1):22

    Article  MATH  Google Scholar 

  21. Hsieh C, Li C, Yeh C, Nakhost H, Fujii Y, Ratner A, Krishna R, Lee C, Pfister T (2023) Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In: Rogers A, Boyd-Graber JL, Okazaki N (eds) Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp 8003–8017. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.findings-acl.507

  22. Lewis PSH, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, Küttler H, Lewis M, Yih W, Rocktäschel T, Riedel S, Kiela D (2020) Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual. https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html

  23. Siddharth L, Luo J (2024) Retrieval augmented generation using engineering design knowledge. Knowl Based Syst 303:112410. https://doi.org/10.1016/J.KNOSYS.2024.112410

    Article  MATH  Google Scholar 

  24. Chen J, Lin H, Han X, Sun L (2024) Benchmarking large language models in retrieval-augmented generation. In: Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp 17754–17762. https://doi.org/10.1609/AAAI.V38I16.29728

  25. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Guyon I, Luxburg U, Bengio S, Wallach HM, Fergus R, Vishwanathan SVN, Garnett R (eds) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp 5998–6008. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

  26. Yang G, Zhou Y, Yu C, Chen X (2021) Deepscc: Source code classification based on fine-tuned roberta (S). In: Chang S (ed) The 33rd International Conference on Software Engineering and Knowledge Engineering, SEKE 2021, KSIR Virtual Conference Center, USA, July 1 - July 10, 2021, pp 499–502. KSI Research Inc., https://doi.org/10.18293/SEKE2021-005

  27. Chen M, Tworek J, Jun H, Yuan Q, Oliveira Pinto HP, Kaplan J, Edwards H, Burda Y, Joseph N, Brockman G, Ray A, Puri R, Krueger G, Petrov M, Khlaaf H, Sastry G, Mishkin P, Chan B, Gray S, Ryder N, Pavlov M, Power A, Kaiser L, Bavarian M, Winter C, Tillet P, Such FP, Cummings D, Plappert M, Chantzis F, Barnes E, Herbert-Voss A, Guss WH, Nichol A, Paino A, Tezak N, Tang J, Babuschkin I,Balaji S, Jain S, Saunders W, Hesse C, Carr AN, Leike J, Achiam J, Misra V, Morikawa E, Radford A, Knight M, Brundage M, Murati M, Mayer K, Welinder P, McGrew B, Amodei D, McCandlish S, Sutskever I, Zaremba W (2021) Evaluating large language models trained on code. arXiv:2107.03374

  28. Mu F, Chen X, Shi L, Wang S, Wang Q (2023) Developer-intent driven code comment generation. In: 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp 768–780. IEEE. https://doi.org/10.1109/ICSE48619.2023.00073

  29. Nie P, Zhang J, Li JJ, Mooney RJ, Gligoric M (2022) Impact of evaluation methodologies on code summarization. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp 4936–4960. https://doi.org/10.18653/v1/2022.acl-long.339

  30. Mastropaolo A, Ciniselli M, Penta MD, Bavota G (2024) Evaluating code summarization techniques: a new metric and an empirical characterization. In: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pp 218–121813. ACM. https://doi.org/10.1145/3597503.3639174

Download references

Acknowledgements

This paper is supported by Guangdong Basic and Applied Basic Research Foundation, China (Grant No. 2021A1515012556).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Dongning Rao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix A: Descriptions of prompt templates

Table 8 PRIME templates experimented in this paper

Table 8 gives a full description of each tested template annotated with abbreviated letters in our experiments. Specifically, L stands for “|Language:[LANG].|”, P stands for “|Path:[PATH].|”, N stands for “|Name:[NAME].|”, C stands for “|Code:[C].|”, S stands for “|Summarization:||[S].|”, and CS stands for “|[C] is summarized as||[S].|”. Task prompts (T1, T2, T3, and T4) are defined in Table 1.

Appendix B: Comparing summaries generated by different templates

Fig. 4
figure 4

Comparing summaries generated by different PRIME templates

We conduct case studies for comparing summaries generated by a large amount of PRIME templates in Fig. 4. see Table 8 for detailed descriptions of templates.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, Z., Wang, D. & Rao, D. Leveraging meta-data of code for adapting prompt tuning for code summarization. Appl Intell 55, 211 (2025). https://doi.org/10.1007/s10489-024-06197-0

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-06197-0

Keywords