Abstract
Prompt tuning alleviates the gap between pre-training and fine-tuning and achieves promising results in various natural language processing (NLP) tasks. However, it is nontrivial for adapting prompt tuning in intelligent code tasks since code-specific knowledge such as abstract syntax tree is usually hierarchy-structured and therefore is hard to be converted into plain text. Recent works (e.g., PT4Code) introduce simple task prompts along with a programming language indicator into prompt template, achieving improvement over non-prompting state-of-the-art code models (e.g., CodeT5). Inspired by this, we propose a novel code-specific prompt paradigm, meta-data prompt, which introduces semi-structured code’s meta-data (attribute-value pairs) into prompt template and facilitates the adaption of prompt tuning techniques into code tasks. Specifically, we find the usage of diverse meta-data attributes and their combinations and employ the OpenPrompt to implement a meta-data prompt based code model, PRIME (PRompt tunIng with MEta-data), via utilizing CodeT5 as the backbone model. We experiment PRIME with the source code summarization task on the publicly available CodeSearchNet benchmark. Results show that 1) using good meta-data can lead to an improvement on the model performance; 2) the proposed meta-data prompt can be combined with traditional task prompt for further improvement; 3) our best-performing model can consistently outperform CodeT5 by an absolute score of 0.73 and PT4Code by an absolute score of 0.48 regarding the averaged BLEU metric across six programming languages.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
Note that method is released after the submission of our manuscript. At the time of submission, our model PRIME had the best performance.
References
Liu P, Yuan W, Fu J, Jiang Z, Hayashi H, Neubig G (2023) Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comput Surv 55(9):195–119535
Gu Y, Han X, Liu Z, Huang M (2022) PPT: pre-trained prompt tuning for few-shot learning. In: Muresan S, Nakov P, Villavicencio A (eds) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp 8410–8423. Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-long.576
Schick T, Schütze H (2021) Exploiting cloze-questions for few-shot text classification and natural language inference. In: Merlo P, Tiedemann J, Tsarfaty R (eds) Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, pp 255–269. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.eacl-main.20
Li XL, Liang P (2021) Prefix-tuning: Optimizing continuous prompts for generation. In: Zong C, Xia F, Li W, Navigli R (eds) Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp 4582–4597. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-long.353
Guo D, Ren S, Lu S, Feng Z, Tang D, Liu S, Zhou L, Duan N, Svyatkovskiy A, Fu S, Tufano M, Deng SK, Clement CB, Drain D, Sundaresan N, Yin J, Jiang D, Zhou M (2021) Graphcodebert: Pre-training code representations with data flow. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, https://openreview.net/forum?id=jLoC4ez43PZ
Niu C, Li C, Ng V, Ge J, Huang L, Luo B (2024) Passsum: leveraging paths of abstract syntax trees and self-supervision for code summarization. J Softw Evol Process 36(6)
Wang C, Yang Y, Gao C, Peng Y, Zhang H, Lyu MR (2022) No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence. In: Roychoudhury A, Cadar C, Kim M (eds) Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, Singapore, Singapore, November 14-18, 2022, pp 382–394. ACM. https://doi.org/10.1145/3540250.3549113
Peng H, Li G, Wang W, Zhao Y, Jin Z (2021) Integrating tree path in transformer for code representation. In: Ranzato M, Beygelzimer A, Dauphin YN, Liang P, Vaughan JW (eds) Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, Virtual, pp 9343–9354, https://proceedings.neurips.cc/paper/2021/hash/4e0223a87610176ef0d24ef6d2dcde3a-Abstract.html
Husain H, Wu H, Gazit T, Allamanis M, Brockschmidt M (2019) Codesearchnet challenge: evaluating the state of semantic code search. arXiv:1909.09436
Ding N, Hu S, Zhao W, Chen Y, Liu Z, Zheng H, Sun M (2022) Openprompt: An open-source framework for prompt-learning. In: Basile V, Kozareva Z, Stajner S (eds) Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 - System Demonstrations, Dublin, Ireland, May 22-27, 2022, pp 105–113. Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-demo.10
Wang Y, Wang W, Joty SR, Hoi SCH (2021) Codet 5: identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In: Moens M, Huang X, Specia L, Yih SW (eds) Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp 8696–8708. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.emnlp-main.685
Feng Z, Guo D, Tang D, Duan N, Feng X, Gong M, Shou L, Qin B, Liu T, Jiang D, Zhou M (2020) Codebert: A pre-trained model for programming and natural languages. In: Cohn T, He Y, Liu Y (eds) Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020. Findings of ACL, vol. EMNLP 2020, pp 1536–1547. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings-emnlp.139
Ahmad WU, Chakraborty S, Ray B, Chang K (2021) Unified pre-training for program understanding and generation. In: Toutanova K, Rumshisky A, Zettlemoyer L, Hakkani-Tür D, Beltagy I, Bethard S, Cotterell R, Chakraborty T, Zhou Y (eds) Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp 2655–2668. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.naacl-main.211
Le H, Wang Y, Gotmare AD, Savarese S, Hoi SC (2022) Coderl: mastering code generation through pretrained models and deep reinforcement learning. In: NeurIPS. http://papers.nips.cc/paper_files/paper/2022/hash/8636419dea1aa9fbd25fc4248e702da4-Abstract-Conference.html
Li M, Yu H, Fan G, Zhou Z, Huang Z (2024) Enhancing code summarization with action word prediction. Neurocomputing 563:126777
Guo Y, Chai Y, Zhang L, Li H, Luo M, Guo S (2024) Context-based transfer learning for low resource code summarization. Softw. Pract. Exp. 54(3):465–482
Tsimpoukelli M, Menick J, Cabi S, Eslami SMA, Vinyals O, Hill F (2021) Multimodal few-shot learning with frozen language models. In: Ranzato M, Beygelzimer A, Dauphin YN, Liang P, Vaughan JW (eds) Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, Virtual, pp 200–212. https://proceedings.neurips.cc/paper/2021/hash/01b7575c38dac42f3cfb7d500438b875-Abstract.html
Kumar J, Chimalakonda S (2024) Code summarization without direct access to code - towards exploring federated llms for software engineering. In: Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, EASE 2024, Salerno, Italy, June 18-21, 2024, pp 100–109. ACM. https://doi.org/10.1145/3661167.3661210
Ahmed T, Pai KS, Devanbu PT, Barr ET (2024) Automatic semantic augmentation of language model prompts (for code summarization). In: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pp 220–122013. ACM. https://doi.org/10.1145/3597503.3639183
Su C, McMillan C (2024) Distilled GPT for source code summarization. Autom Softw Eng 31(1):22
Hsieh C, Li C, Yeh C, Nakhost H, Fujii Y, Ratner A, Krishna R, Lee C, Pfister T (2023) Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In: Rogers A, Boyd-Graber JL, Okazaki N (eds) Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp 8003–8017. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.findings-acl.507
Lewis PSH, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, Küttler H, Lewis M, Yih W, Rocktäschel T, Riedel S, Kiela D (2020) Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Virtual. https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html
Siddharth L, Luo J (2024) Retrieval augmented generation using engineering design knowledge. Knowl Based Syst 303:112410. https://doi.org/10.1016/J.KNOSYS.2024.112410
Chen J, Lin H, Han X, Sun L (2024) Benchmarking large language models in retrieval-augmented generation. In: Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pp 17754–17762. https://doi.org/10.1609/AAAI.V38I16.29728
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Guyon I, Luxburg U, Bengio S, Wallach HM, Fergus R, Vishwanathan SVN, Garnett R (eds) Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp 5998–6008. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html
Yang G, Zhou Y, Yu C, Chen X (2021) Deepscc: Source code classification based on fine-tuned roberta (S). In: Chang S (ed) The 33rd International Conference on Software Engineering and Knowledge Engineering, SEKE 2021, KSIR Virtual Conference Center, USA, July 1 - July 10, 2021, pp 499–502. KSI Research Inc., https://doi.org/10.18293/SEKE2021-005
Chen M, Tworek J, Jun H, Yuan Q, Oliveira Pinto HP, Kaplan J, Edwards H, Burda Y, Joseph N, Brockman G, Ray A, Puri R, Krueger G, Petrov M, Khlaaf H, Sastry G, Mishkin P, Chan B, Gray S, Ryder N, Pavlov M, Power A, Kaiser L, Bavarian M, Winter C, Tillet P, Such FP, Cummings D, Plappert M, Chantzis F, Barnes E, Herbert-Voss A, Guss WH, Nichol A, Paino A, Tezak N, Tang J, Babuschkin I,Balaji S, Jain S, Saunders W, Hesse C, Carr AN, Leike J, Achiam J, Misra V, Morikawa E, Radford A, Knight M, Brundage M, Murati M, Mayer K, Welinder P, McGrew B, Amodei D, McCandlish S, Sutskever I, Zaremba W (2021) Evaluating large language models trained on code. arXiv:2107.03374
Mu F, Chen X, Shi L, Wang S, Wang Q (2023) Developer-intent driven code comment generation. In: 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023, pp 768–780. IEEE. https://doi.org/10.1109/ICSE48619.2023.00073
Nie P, Zhang J, Li JJ, Mooney RJ, Gligoric M (2022) Impact of evaluation methodologies on code summarization. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp 4936–4960. https://doi.org/10.18653/v1/2022.acl-long.339
Mastropaolo A, Ciniselli M, Penta MD, Bavota G (2024) Evaluating code summarization techniques: a new metric and an empirical characterization. In: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pp 218–121813. ACM. https://doi.org/10.1145/3597503.3639174
Acknowledgements
This paper is supported by Guangdong Basic and Applied Basic Research Foundation, China (Grant No. 2021A1515012556).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix A: Descriptions of prompt templates
Table 8 gives a full description of each tested template annotated with abbreviated letters in our experiments. Specifically, L stands for “|Language:[LANG].|”, P stands for “|Path:[PATH].|”, N stands for “|Name:[NAME].|”, C stands for “|Code:[C].|”, S stands for “|Summarization:||[S].|”, and CS stands for “|[C] is summarized as||[S].|”. Task prompts (T1, T2, T3, and T4) are defined in Table 1.
Appendix B: Comparing summaries generated by different templates
We conduct case studies for comparing summaries generated by a large amount of PRIME templates in Fig. 4. see Table 8 for detailed descriptions of templates.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Jiang, Z., Wang, D. & Rao, D. Leveraging meta-data of code for adapting prompt tuning for code summarization. Appl Intell 55, 211 (2025). https://doi.org/10.1007/s10489-024-06197-0
Accepted:
Published:
DOI: https://doi.org/10.1007/s10489-024-06197-0