Skip to main content
SpringerLink
Log in

A unified multi-task learning model for AST-level and token-level code completion

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Code completion, one of the most useful features in the Integrated Development Environments (IDEs), can accelerate software development by suggesting the next probable tokens based on existing code in real-time. Recent studies have shown that recurrent neural networks based statistical language models can improve the performance of code completion tools through learning from large-scale software repositories. However, most of the existing approaches treat code completion as a single generation task in which the model predicts the value of the tokens or AST nodes based on the contextual source code without considering the syntactic constraints such as the static type information. Besides, the semantic relationships in programs can be very long. Existing recurrent neural networks based language models are not sufficient to model the long-term dependency. In this paper, we tackle the aforementioned limitations by building a unified multi-task learning based code completion model for both AST-level and token-level code completion. To model the relationship and constraints between the type and value of the code elements, we adopt a multi-task learning framework to predict the type and value of the tokens (AST nodes) simultaneously. To capture the long-term dependency in the input programs, we employ a self-attentional architecture based network as the base language model. We apply our approach to both AST-level and token-level code completion. Experimental results demonstrate the effectiveness of our model when compared with state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. https://www.aixcoder.com/#/

  2. The nodes in the path are non-leaf nodes, and they do not have the value attribute. Thus, we use the node’s type as the representation for the nodes in the path.

  3. https://github.com/python/cpython/blob/3.9/Lib/ast.py

  4. https://github.com/acornjs/acorn

  5. https://github.com/c2nes/javalang

  6. https://github.com/google/gcm

  7. https://github.com/hotchemi/LruCache

  8. For the next node’s type prediction, the upper bound of the accuracy is 100%. For the next node’s value prediction, since the UNK targets are treated as wrong predictions, the upper bound of the accuracy is less than 100%, which depends on the UNK rate of the dataset.

  9. The datasets and code are publicly available in https://figshare.com/s/7eb8819f2a04e8163224

  10. https://github.com/jack57lee/neuralCodeCompletion

  11. 34% = (87.1%-80.6%) / (100%-80.6%), 17% = (83.4%-79.9%) / (100%-79.9%), 25% = (91.4%-88.6%) / (100%-88.6%)

  12. 20% = (73.8%-70.1%) / (89%-70.1%), 20% = (76.8%-74.3%) / (87%-74.3%), 14% = (82.7%-81.0%) / (93%-81.0%)

  13. https://github.com/jack57lee/neuralCodeCompletion

  14. Since the PMN also makes use of the additional information derived from ASTs, the results of using the token sequence as input might understate the accuracy of the plain PMN.

  15. https://github.com/mast-group/OpenVocabCodeNLM

References

  • Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: A system for large-scale machine learning. In: 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16), pp 265–283

  • Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate

  • Bhoopchand A, Rocktäschel T, Barr ET, Riedel S (2016) Learning python code suggestion with a sparse pointer network. CoRR arXiv:1611.08307

  • Bielik P, Raychev V, Vechev MT (2016) PHOG: probabilistic model for code. In: Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, JMLR Workshop and Conference Proceedings, vol 48. JMLR.org, New York City, pp 2933–2942

  • Bruch M, Monperrus M, Mezini M (2009) Learning from examples to improve code completion systems. In: Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering, pp 213–222

  • Caruana R (1997) Multitask learning. Mach Learn 28(1):41–75

    Article  MathSciNet  Google Scholar 

  • Chelba C, Engle D, Jelinek F, Jimenez V, Khudanpur S, Mangu L, Printz H, Ristad E, Rosenfeld R, Stolcke A, Wu D (1997) Structure and performance of a dependency language model. In: Fifth European Conference on Speech Communication and Technology, EUROSPEECH 1997, Rhodes

  • Chelba C, Jelinek F (1998) Exploiting syntactic structure for language modeling. In: Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics-Volume 1. Association for Computational Linguistics, pp 225–231

  • Cho K, van Merrienboer B, Bahdanau D, Bengio Y (2014) On the properties of neural machine translation: Encoder-decoder approaches, pp 103–111

  • Costa C, Figueiredo J, Murta L, Sarma A (2016) Tipmerge: recommending experts for integrating changes across branches. In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, pp 523–534

  • Dai Z, Yang Z, Yang Y, Carbonell JG, Le QV, Salakhutdinov R (2019) Transformer-xl: Attentive language models beyond a fixed-length context. In: Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp 2978–2988

  • Deng L, Hinton GE, Kingsbury B (2013) New types of deep neural network learning for speech recognition and related applications: an overview. In: IEEE international conference on acoustics, speech and signal processing, ICASSP 2013, vancouver, bc, canada, may 26-31, 2013. IEEE, pp 8599–8603

  • Devlin J, Chang M-W, Lee K, Toutanova K (2018) BERT: pre-training of deep bidirectional transformers for language understanding. CoRR arXiv:1810.04805

  • Dong D, Wu H, He W, Yu D, Wang H (2015) Multi-task learning for multiple language translation. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, Volume 1: Long Papers. The Association for Computer Linguistics, Beijing, pp 1723–1732

  • Feng Z, Guo D, Tang D, Duan N, Feng X, Gong M, Shou L, Qin B, Liu T, Jiang D et al (2020) Codebert: A pre-trained model for programming and natural languages. arXiv:2002.08155

  • Gage P (1994) A new algorithm for data compression. C Users J 12(2):23–38

    Google Scholar 

  • Guo H, Pasunuru R, Bansal M (2018) Soft layer-specific multi-task summarization with entailment and question generation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Volume 1: Long Papers. Association for Computational Linguistics, Melbourne, pp 687–697

  • Han J, Deng S, Xia X, Wang D, Yin J (2019) Characterization and prediction of popular projects on github. In: 2019 IEEE 43rd annual computer software and applications conference (COMPSAC), vol 1. IEEE, pp 21–26

  • Hellendoorn VJ, Bird C, Barr ET, Allamanis M (2018) Deep learning type inference. In: Proceedings of the 2018 26th acm joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 152–162

  • Hellendoorn VJ, Devanbu PT (2017) Are deep neural networks the best choice for modeling source code?. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017. ACM, pp 763–773

  • Hellendoorn VJ, Proksch S, Gall HC, Bacchelli A (2019) When code completion fails: A case study on real-world completions. In: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, pp 960–970

  • Hindle A, Barr ET, Su Z, Gabel M, Devanbu PT (2012) On the naturalness of software. In: 34th International Conference on Software Engineering, ICSE 2012, June 2-9, 2012, Zurich, Switzerland. IEEE Computer Society, pp 837–847

  • Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  • Hou D, Pletcher DM (2010) Towards a better code completion system by api grouping, filtering, and popularity-based ranking. In: Proceedings of the 2nd International Workshop on Recommendation Systems for Software Engineering, pp 26–30

  • Hu X, Li G, Xia X, Lo D, Lu S, Jin Z (2018) Summarizing source code with transferred API knowledge. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018. ijcai.org, Stockholm, pp 2269–2275

  • Isonuma M, Fujino T, Mori J, Matsuo Y, Sakata I (2017) Extractive summarization using multi-task learning with document classification. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017. Association for Computational Linguistics, Copenhagen, pp 2101–2110

  • Karampatsis R-M, Babii H, Robbes R, Sutton C, Janes A (2020) Big code!= big vocabulary: Open-vocabulary models for source code. ICSE

  • Khandelwal U, He H, Qi P, Jurafsky D (2018) Sharp nearby, fuzzy far away: How neural language models use context, pp 284–294

  • Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: Bengio Y, LeCun Y (eds) 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings, San Diego

  • Li J, Wang Y, Lyu MR, King I (2018) Code completion with neural attention and pointer networks. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018. ijcai.org, Stockholm, pp 4159–4165, DOI https://doi.org/10.24963/ijcai.2018/578, (to appear in print)

  • Lin Y, Yang S, Stoyanov V, Ji H (2018) A multi-lingual multi-task architecture for low-resource sequence labeling. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Volume 1: Long Papers. Association for Computational Linguistics, Melbourne, pp 799–809

  • Liu C, Wang X, Shin R, Gonzalez JE, Song D (2016) Neural code completion

  • Liu F, Li G, Wei B, Xia X, Fu Z, Jin Z (2020) A self-attentional neural architecture for code completion with multi-task learning. In: Proceedings of the 28th International Conference on Program Comprehension, pp 37–47

  • Liu F, Li G, Zhao Y, Jin Z (2020) Multi-task learning based pre-trained language model for code completion. In: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pp 473–485

  • Liu X, Gao J, He X, Deng L, Duh K, Wang Y-Y (2015) Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In: NAACL HLT 2015, the 2015 conference of the north american chapter of the association for computational linguistics: Human language technologies. The Association for Computational Linguistics, Denver, pp 912–921

  • Long M, Wang J (2015) Learning multiple tasks with deep relationship networks. CoRR arXiv:1506.02117

  • Lu Y, Kumar A, Zhai S, Cheng Y, Javidi T, Feris RS (2017) Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. IEEE Computer Society, Honolulu, pp 1131–1140

  • Luong M-T, Le QV, Sutskever I, Vinyals O, Kaiser L (2016) Multi-task sequence to sequence learning

  • Macbeth G, Razumiejczyk E, Ledesma RD (2011) Cliff’s delta calculator: A non-parametric effect size program for two groups of observations. Univ Psychol 10(2):545–555

    Article  Google Scholar 

  • Maddison C, Tarlow D (2014) Structured generative models of natural source code. In: International Conference on Machine Learning, pp 649–657

  • Malik RS, Patra J, Pradel M (2019) Nl2type: inferring javascript function types from natural language information. In: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, pp 304–315

  • Nguyen AT, Nguyen TN (2015) Graph-based statistical language model for code. In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol 1. IEEE, pp 858–868

  • Nguyen TT, Nguyen AT, Nguyen HA, Nguyen TN (2013) A statistical semantic language model for source code. In: Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE’13. ACM, Saint Petersburg, pp 532–542

  • Peng N, Dredze M (2017) Multi-task domain adaptation for sequence tagging. In: Proceedings of the 2nd Workshop on Representation Learning for NLP, Rep4NLP@ACL 20107. Association for Computational Linguistics, Vancouver, pp 91–100

  • Raychev V, Bielik P, Vechev MT (2016) Probabilistic model for code with decision trees. In: Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016, part of SPLASH 2016. ACM, Amsterdam, pp 731–747

  • Robbes R, Lanza M (2008) How program history can improve code completion. In: 2008 23rd IEEE/ACM International Conference on Automated Software Engineering. IEEE, pp 317–326

  • Ruder S (2017) An overview of multi-task learning in deep neural networks. CoRR arXiv:1706.05098

  • Schuster M, Paliwal KK (1997) Bidirectional recurrent neural networks. IEEE Trans Signal Process 45(11):2673–2681

    Article  Google Scholar 

  • Svyatkovskiy A, Zhao Y, Fu S, Sundaresan N (2019) Pythia: Ai-assisted code completion system. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp 2727–2735

  • Svyatkovskoy A, Lee S, Hadjitofi A, Riechert M, Franco J, Allamanis M (2020) Fast and memory-efficient neural code completion. arXiv:2004.13651

  • Tu Z, Su Z, Devanbu PT (2014) On the localness of software. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE-22). ACM, Hong Kong, pp 269–280

  • Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008

  • Wei B, Li G, Xia X, Fu Z, Jin Z (2019) Code generation as a dual task of code summarization. In: Advances in Neural Information Processing Systems, pp 6563–6573

  • White M, Vendome C, Vásquez ML, Poshyvanyk D (2015) Toward deep learning software repositories. In: 12th IEEE/ACM Working Conference on Mining Software Repositories, MSR 2015. IEEE Computer Society, Florence, pp 334–345

  • Wilcoxon F (1945) Individual comparisons by ranking methods. Biometr Bullet 1(6):80–83

    Article  Google Scholar 

  • Yao Z, Peddamail JR, Sun H (2019) Coacor: code annotation for code retrieval with reinforcement learning. In: The World Wide Web Conference, pp 2203–2214

  • Zaremoodi P, Buntine WL, Haffari G (2018) Adaptive knowledge sharing in multi-task learning: Improving low-resource neural machine translation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Volume 2: Short Papers. Association for Computational Linguistics, Melbourne, pp 656–661

Download references

Acknowledgments

This research is supported by the National Natural Science Foundation of China under Grant Nos. 62072007, 62192731, 62192733, 61832009, 62192730. Zhi Jin and Ge Li are corresponding authors.

Author information

Authors and Affiliations

  1. Key Laboratory of High Confidence Software Technologies (Peking University), Ministry of Education, Beijing, China

    Fang Liu, Ge Li, Bolin Wei, Zhiyi Fu & Zhi Jin

  2. School of Computer Science, Peking University, Beijing, China

    Fang Liu, Ge Li, Bolin Wei, Zhiyi Fu & Zhi Jin

  3. Huawei, China

    Xin Xia

Authors
  1. Fang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ge Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bolin Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xin Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhiyi Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zhi Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Ge Li or Zhi Jin.

Additional information

Communicated by: Yann-Gaël Guéhéneuc, Shinpei Hayashi and Michel R. V. Chaudron

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: International Conference on Program Comprehension (ICPC)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, F., Li, G., Wei, B. et al. A unified multi-task learning model for AST-level and token-level code completion. Empir Software Eng 27, 91 (2022). https://doi.org/10.1007/s10664-022-10140-7

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-022-10140-7

Keywords

Search

Navigation