Skip to main content
SpringerLink
Log in

Adversarial training and ensemble learning for automatic code summarization

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Natural language summaries of codes are important during software development and maintenance. Recently, deep learning-based models have achieved good performance on automatic code summarization, which encode token sequence or abstract syntax tree (AST) of code with neural networks. However, almost all of these models are trained using maximum likelihood estimation, which do not guarantee the quality of generated summaries. Moreover, existing models that benefit from multiple encoders lack a fined-grained selection between different encoders, and the encoders may be insufficiently optimized. To address these issues and generate better code summaries, we propose a novel code summarization framework based on adversarial training and ensemble learning. It includes two separately trained encoder-decoder models, one for source code sequence and the other for its AST. Here, an efficient approach to obtain AST node sequence is introduced. We train our models via adversarial training, where each model is guided by a well-designed discriminator that learns to evaluate its outputs. During inference, a module named mixture network is introduced to compute an adaptive combination weight of the models’ outputs. We evaluate our framework on a large Java corpus and compare it to several state-of-the-art models. Experimental results show that our approach outperforms the best baseline by 22.6% on BLEU-4, 5.7% on ROUGE-L and 7.6% on METEOR.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. https://en.wikipedia.org/wiki/Perplexity.

  2. https://github.com/c2nes/javalang.

  3. http://www.nltk.org/.

References

  1. Sridhara G, Hill E, Muppaneni D, Pollock LL, Vijay-Shanker K (2010) Towards automatically generating summary comments for Java methods. In: Proceedings of the IEEE/ACM International Conference on Automated Software Engineering, ACM, pp 43–52

  2. Haiduc S, Aponte J, Moreno L, Marcus A (2010) On the use of automated text summarization techniques for summarizing source code. In: Reverse Engineering (WCRE), 2010 17th Working Conference on, IEEE, pp 35–44

  3. McBurney PW, McMillan C (2016) Automatic source code summarization of context for java methods. IEEE Trans Software Eng 42(2):103–119

    Article  Google Scholar 

  4. Movshovitz-Attias D, Cohen WW (2013) Natural language models for predicting programming comments. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL, pp 35–40

  5. Moreno L, Aponte J, Sridhara G, Marcus A, Pollock LL, Vijay-Shanker K (2013) Automatic generation of natural language summaries for Java classes. In: IEEE 21st International Conference on Program Comprehension, pp 23–32

  6. Haiduc S, Aponte J, Marcus A (2010) Supporting program comprehension with source code summarization. In: Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering. Vol 2, ACM, pp 223–226

  7. Iyer S, Konstas I, Cheung A, Zettlemoyer L (2016) Summarizing source code using a neural attention model. In: Proceedings of the 54th annual meeting of the association for computational linguistics, ACL

  8. Allamanis M, Peng H, Sutton CA (2016) A convolutional attention network for extreme summarization of source code. In: International conference on machine learning, pp 2091–2100

  9. Hu X, Li G, Xia X, Lo D, Jin Z (2018) Deep code comment generation. In: Proceedings of the 26th conference on program comprehension, ACM, pp 200–210

  10. Wan Y, Zhao Z, Yang M, Xu G, Ying H, Wu J, Yu PS (2018) Improving automatic source code summarization via deep reinforcement learning. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, ACM, pp 397–407

  11. LeClair A, Jiang S, McMillan C (2019) A neural model for generating natural language summaries of program subroutines. In: Proceedings of the 41st international conference on software engineering, IEEE/ACM, pp 795–806

  12. Hu X, Li G, Xia X, Lo D, Jin Z (2020) Deep code comment generation with hybrid lexical and syntactical information. Empir Softw Eng 25(3):2179–2217

    Article  Google Scholar 

  13. Liang Y, Zhu KQ (2018) Automatic generation of text descriptive comments for code blocks. In: Proceedings of the Thirty-Second AAAI Conference on Artificial intelligence, AAAI Press, pp 5229–5236

  14. Fernandes P, Allamanis M, Brockschmidt M (2019) Structured neural summarization. In: 7th international conference on learning representations

  15. Alon U, Brody S, Levy O, Yahav E (2019) code2seq: Generating sequences from structured representations of code. In: 7th international Conference On Learning Representations

  16. Hu X, Li G, Xia X, Lo D, Lu S, Jin Z (2018) Summarizing source code with transferred api knowledge. In: Proceedings of the twenty-seventh international joint conference on artificial intelligence, pp 2269–2275

  17. Wei B, Li G, Xia X, Fu Z, Jin Z (2019) Code generation as a dual task of code summarization. In: Advances in neural information processing systems, pp 6559–6569

  18. Ye W, Xie R, Zhang J, Hu T, Wang X, Zhang S (2020) Leveraging code generation to improve code retrieval and summarization via dual learning. In: The Web Conference, pp 2309–2319

  19. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Advances in neural information processing systems. pp 3104–3112

  20. Cho K, Merrienboer Bv, Gülçehre Ç, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using rnn encoder-decoder for statistical machine translation. In: Proceedings of the 2014 conference on empirical methods in natural language processing, pp 1724–1734

  21. Luong T, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. In: Proceedings of the 2015 conference on empirical methods in natural language processing, ACL, pp 1412–1421

  22. Wu Y, Schuster M, Chen Z, Le QV, Norouzi M, Macherey W, Krikun M, Cao Y, Gao Q, Macherey K, Klingner J, Shah A, Johnson M, Liu X, Kaiser L, Gouws S, Kato Y, Kudo T, Kazawa H, Stevens K, Kurian G, Patil N, Wang W, Young C, Smith J, Riesa J, Rudnick A, Vinyals O, Corrado G, Hughes M, Dean J (2016) Google's neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144

  23. Hellendoorn VJ, Devanbu PT (2017) Are deep neural networks the best choice for modeling source code?. In: Proceedings of the 2017 11th joint meeting on foundations of software engineering, ACM, pp 763–773

  24. Ranzato MA, Chopra S, Auli M, Zaremba W (2016) Sequence level training with recurrent neural networks. In: 4th International conference on learning representations

  25. Papineni K, Roukos S, Ward T, Zhu W-J (2002) BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL, pp 311–318

  26. Lin C-Y (2004) ROUGE: A package for automatic evaluation of summaries. In: Proceedings of the workshop on text summarization branches out

  27. Callison-Burch C, Osborne M, Koehn P (2006) Re-evaluation the role of bleu in machine translation research. In: 11th Conference of the European chapter of the association for computational linguistics

  28. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville AC, Bengio Y (2014) Generative adversarial nets. In: NIPS, pp 2672–2680

  29. Chen B, Cherry C (2014) A systematic comparison of smoothing techniques for sentence-level BLEU. In: ACL, pp 362–367

  30. Wong E, Liu T, Tan L (2015) CloCom: mining existing source code for automatic comment generation. In: Software analysis, evolution and reengineering (SANER), 2015 IEEE 22nd international Conference on. IEEE, pp 380–389

  31. Wong E, Yang J, Tan L (2013) AutoComment: mining question and answer sites for automatic comment generation. In: Proceedings of the 28th IEEE/ACM international conference on automated software engineering, IEEE, pp 562–567

  32. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  33. Cho K, Merrienboer Bv, Bahdanau D, Bengio Y (2014) On the properties of neural machine translation: encoder-decoder approaches. In: Proceedings of SSST@EMNLP, ACL, pp 103–111

  34. Mirza M, Osindero S (2014) Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784

  35. Yu L, Zhang W, Wang J, Yu Y (2017) SeqGAN: sequence generative adversarial nets with policy gradient. In: AAAI, pp 2852–2858

  36. Li J, Monroe W, Shi T, Jean S, Ritter A, Jurafsky D (2017) Adversarial learning for neural dialogue generation. In: EMNLP, pp 2157–2169

  37. Dognin PL, Melnyk I, Mroueh Y, Ross J, Sercu T (2019) Adversarial semantic alignment for improved image captions. In: CVPR, pp 10463–10471

  38. Chen C, Mu S, Xiao W, Ye Z, Wu L, Ju Q (2019) Improving image captioning with conditional generative adversarial nets. In: AAAI, pp 8142–8150

  39. Yang Z, Chen W, Wang F, Xu B (2018) Improving neural machine translation with conditional sequence generative adversarial nets. In: NAACL-HLT, pp 1346–1355

  40. Wu L, Xia Y, Tian F, Zhao L, Qin T, Lai J, Liu T-Y (2018) Adversarial neural machine translation. In: ACML, pp 534–549

  41. Garmash E, Monz C (2016) Ensemble learning for multi-source neural machine translation. In: 26th International conference on computational linguistics: technical papers, pp 1409–1418

  42. Kim Y (2014) Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882

  43. Hu B, Lu Z, Li H, Chen Q (2015) Convolutional neural network architectures for matching natural language sentences. arXiv preprint arXiv:1503.03244

  44. Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th international conference on machine learning, pp 807–814

  45. Rennie SJ, Marcheret E, Mroueh Y, Ross J, Goel V (2017) Self-critical sequence training for image captioning. Reinforcement learning. Springer, Berlin, pp 1179–1195

    Google Scholar 

  46. Williams RJ (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach Learn 8:229–256

    MATH  Google Scholar 

  47. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp 249–256

  48. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: 3rd International conference on learning representations

  49. Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  50. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. pp 448–456

  51. Denkowski M, Lavie A (2014) Meteor universal: language specific translation evaluation for any target language. In: Proceedings of the EACL 2014 Workshop on Statistical Machine Translation. ACL

  52. Konda VR, Tsitsiklis JN (1999) Actor-critic algorithms. In: Advances in neural information processing systems, pp 1008–1014

  53. LeClair A, Haque S, Wu L, McMillan C (2020) Improved code summarization via a graph neural network. In: 28th International Conference on Program Comprehension, pp 184–195

  54. Gui J, Sun Z, Wen Y, Tao D, Ye J (2020) A review on generative adversarial networks: algorithms, theory, and applications. arXiv preprint arXiv:2001.06937v1

Download references

Funding

This work is partially supported by the NSF of China under Grants No. 61772200 and 61702334, Shanghai Pujiang Talent Program under Grants No. 17PJ1401900, Shanghai Municipal Natural Science Foundation under Grants No. 17ZR1406900 and 17ZR1429700, Educational Research Fund of ECUST under Grant No. ZH1726108.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, East China University of Science and Technology, Shanghai, China

    Ziyi Zhou, Huiqun Yu & Guisheng Fan

  2. Shanghai Engineering Research Center of Smart Energy, Shanghai, China

    Huiqun Yu

Authors
  1. Ziyi Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huiqun Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guisheng Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Huiqun Yu.

Ethics declarations

Conflict of interest

The authors declare that they have Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, Z., Yu, H. & Fan, G. Adversarial training and ensemble learning for automatic code summarization. Neural Comput & Applic 33, 12571–12589 (2021). https://doi.org/10.1007/s00521-021-05907-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-05907-w

Keywords

Search

Navigation