Abstract
Code completion, one of the most useful features in the Integrated Development Environments (IDEs), can accelerate software development by suggesting the next probable tokens based on existing code in real-time. Recent studies have shown that recurrent neural networks based statistical language models can improve the performance of code completion tools through learning from large-scale software repositories. However, most of the existing approaches treat code completion as a single generation task in which the model predicts the value of the tokens or AST nodes based on the contextual source code without considering the syntactic constraints such as the static type information. Besides, the semantic relationships in programs can be very long. Existing recurrent neural networks based language models are not sufficient to model the long-term dependency. In this paper, we tackle the aforementioned limitations by building a unified multi-task learning based code completion model for both AST-level and token-level code completion. To model the relationship and constraints between the type and value of the code elements, we adopt a multi-task learning framework to predict the type and value of the tokens (AST nodes) simultaneously. To capture the long-term dependency in the input programs, we employ a self-attentional architecture based network as the base language model. We apply our approach to both AST-level and token-level code completion. Experimental results demonstrate the effectiveness of our model when compared with state-of-the-art methods.






Similar content being viewed by others
Notes
The nodes in the path are non-leaf nodes, and they do not have the value attribute. Thus, we use the node’s type as the representation for the nodes in the path.
For the next node’s type prediction, the upper bound of the accuracy is 100%. For the next node’s value prediction, since the UNK targets are treated as wrong predictions, the upper bound of the accuracy is less than 100%, which depends on the UNK rate of the dataset.
The datasets and code are publicly available in https://figshare.com/s/7eb8819f2a04e8163224
34% = (87.1%-80.6%) / (100%-80.6%), 17% = (83.4%-79.9%) / (100%-79.9%), 25% = (91.4%-88.6%) / (100%-88.6%)
20% = (73.8%-70.1%) / (89%-70.1%), 20% = (76.8%-74.3%) / (87%-74.3%), 14% = (82.7%-81.0%) / (93%-81.0%)
Since the PMN also makes use of the additional information derived from ASTs, the results of using the token sequence as input might understate the accuracy of the plain PMN.
References
Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: A system for large-scale machine learning. In: 12th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 16), pp 265–283
Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate
Bhoopchand A, Rocktäschel T, Barr ET, Riedel S (2016) Learning python code suggestion with a sparse pointer network. CoRR arXiv:1611.08307
Bielik P, Raychev V, Vechev MT (2016) PHOG: probabilistic model for code. In: Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, JMLR Workshop and Conference Proceedings, vol 48. JMLR.org, New York City, pp 2933–2942
Bruch M, Monperrus M, Mezini M (2009) Learning from examples to improve code completion systems. In: Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on the foundations of software engineering, pp 213–222
Caruana R (1997) Multitask learning. Mach Learn 28(1):41–75
Chelba C, Engle D, Jelinek F, Jimenez V, Khudanpur S, Mangu L, Printz H, Ristad E, Rosenfeld R, Stolcke A, Wu D (1997) Structure and performance of a dependency language model. In: Fifth European Conference on Speech Communication and Technology, EUROSPEECH 1997, Rhodes
Chelba C, Jelinek F (1998) Exploiting syntactic structure for language modeling. In: Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics-Volume 1. Association for Computational Linguistics, pp 225–231
Cho K, van Merrienboer B, Bahdanau D, Bengio Y (2014) On the properties of neural machine translation: Encoder-decoder approaches, pp 103–111
Costa C, Figueiredo J, Murta L, Sarma A (2016) Tipmerge: recommending experts for integrating changes across branches. In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, pp 523–534
Dai Z, Yang Z, Yang Y, Carbonell JG, Le QV, Salakhutdinov R (2019) Transformer-xl: Attentive language models beyond a fixed-length context. In: Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp 2978–2988
Deng L, Hinton GE, Kingsbury B (2013) New types of deep neural network learning for speech recognition and related applications: an overview. In: IEEE international conference on acoustics, speech and signal processing, ICASSP 2013, vancouver, bc, canada, may 26-31, 2013. IEEE, pp 8599–8603
Devlin J, Chang M-W, Lee K, Toutanova K (2018) BERT: pre-training of deep bidirectional transformers for language understanding. CoRR arXiv:1810.04805
Dong D, Wu H, He W, Yu D, Wang H (2015) Multi-task learning for multiple language translation. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, Volume 1: Long Papers. The Association for Computer Linguistics, Beijing, pp 1723–1732
Feng Z, Guo D, Tang D, Duan N, Feng X, Gong M, Shou L, Qin B, Liu T, Jiang D et al (2020) Codebert: A pre-trained model for programming and natural languages. arXiv:2002.08155
Gage P (1994) A new algorithm for data compression. C Users J 12(2):23–38
Guo H, Pasunuru R, Bansal M (2018) Soft layer-specific multi-task summarization with entailment and question generation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Volume 1: Long Papers. Association for Computational Linguistics, Melbourne, pp 687–697
Han J, Deng S, Xia X, Wang D, Yin J (2019) Characterization and prediction of popular projects on github. In: 2019 IEEE 43rd annual computer software and applications conference (COMPSAC), vol 1. IEEE, pp 21–26
Hellendoorn VJ, Bird C, Barr ET, Allamanis M (2018) Deep learning type inference. In: Proceedings of the 2018 26th acm joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 152–162
Hellendoorn VJ, Devanbu PT (2017) Are deep neural networks the best choice for modeling source code?. In: Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017. ACM, pp 763–773
Hellendoorn VJ, Proksch S, Gall HC, Bacchelli A (2019) When code completion fails: A case study on real-world completions. In: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, pp 960–970
Hindle A, Barr ET, Su Z, Gabel M, Devanbu PT (2012) On the naturalness of software. In: 34th International Conference on Software Engineering, ICSE 2012, June 2-9, 2012, Zurich, Switzerland. IEEE Computer Society, pp 837–847
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780
Hou D, Pletcher DM (2010) Towards a better code completion system by api grouping, filtering, and popularity-based ranking. In: Proceedings of the 2nd International Workshop on Recommendation Systems for Software Engineering, pp 26–30
Hu X, Li G, Xia X, Lo D, Lu S, Jin Z (2018) Summarizing source code with transferred API knowledge. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018. ijcai.org, Stockholm, pp 2269–2275
Isonuma M, Fujino T, Mori J, Matsuo Y, Sakata I (2017) Extractive summarization using multi-task learning with document classification. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017. Association for Computational Linguistics, Copenhagen, pp 2101–2110
Karampatsis R-M, Babii H, Robbes R, Sutton C, Janes A (2020) Big code!= big vocabulary: Open-vocabulary models for source code. ICSE
Khandelwal U, He H, Qi P, Jurafsky D (2018) Sharp nearby, fuzzy far away: How neural language models use context, pp 284–294
Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: Bengio Y, LeCun Y (eds) 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings, San Diego
Li J, Wang Y, Lyu MR, King I (2018) Code completion with neural attention and pointer networks. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018. ijcai.org, Stockholm, pp 4159–4165, DOI https://doi.org/10.24963/ijcai.2018/578, (to appear in print)
Lin Y, Yang S, Stoyanov V, Ji H (2018) A multi-lingual multi-task architecture for low-resource sequence labeling. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Volume 1: Long Papers. Association for Computational Linguistics, Melbourne, pp 799–809
Liu C, Wang X, Shin R, Gonzalez JE, Song D (2016) Neural code completion
Liu F, Li G, Wei B, Xia X, Fu Z, Jin Z (2020) A self-attentional neural architecture for code completion with multi-task learning. In: Proceedings of the 28th International Conference on Program Comprehension, pp 37–47
Liu F, Li G, Zhao Y, Jin Z (2020) Multi-task learning based pre-trained language model for code completion. In: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pp 473–485
Liu X, Gao J, He X, Deng L, Duh K, Wang Y-Y (2015) Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In: NAACL HLT 2015, the 2015 conference of the north american chapter of the association for computational linguistics: Human language technologies. The Association for Computational Linguistics, Denver, pp 912–921
Long M, Wang J (2015) Learning multiple tasks with deep relationship networks. CoRR arXiv:1506.02117
Lu Y, Kumar A, Zhai S, Cheng Y, Javidi T, Feris RS (2017) Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. IEEE Computer Society, Honolulu, pp 1131–1140
Luong M-T, Le QV, Sutskever I, Vinyals O, Kaiser L (2016) Multi-task sequence to sequence learning
Macbeth G, Razumiejczyk E, Ledesma RD (2011) Cliff’s delta calculator: A non-parametric effect size program for two groups of observations. Univ Psychol 10(2):545–555
Maddison C, Tarlow D (2014) Structured generative models of natural source code. In: International Conference on Machine Learning, pp 649–657
Malik RS, Patra J, Pradel M (2019) Nl2type: inferring javascript function types from natural language information. In: 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, pp 304–315
Nguyen AT, Nguyen TN (2015) Graph-based statistical language model for code. In: 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol 1. IEEE, pp 858–868
Nguyen TT, Nguyen AT, Nguyen HA, Nguyen TN (2013) A statistical semantic language model for source code. In: Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, ESEC/FSE’13. ACM, Saint Petersburg, pp 532–542
Peng N, Dredze M (2017) Multi-task domain adaptation for sequence tagging. In: Proceedings of the 2nd Workshop on Representation Learning for NLP, Rep4NLP@ACL 20107. Association for Computational Linguistics, Vancouver, pp 91–100
Raychev V, Bielik P, Vechev MT (2016) Probabilistic model for code with decision trees. In: Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA 2016, part of SPLASH 2016. ACM, Amsterdam, pp 731–747
Robbes R, Lanza M (2008) How program history can improve code completion. In: 2008 23rd IEEE/ACM International Conference on Automated Software Engineering. IEEE, pp 317–326
Ruder S (2017) An overview of multi-task learning in deep neural networks. CoRR arXiv:1706.05098
Schuster M, Paliwal KK (1997) Bidirectional recurrent neural networks. IEEE Trans Signal Process 45(11):2673–2681
Svyatkovskiy A, Zhao Y, Fu S, Sundaresan N (2019) Pythia: Ai-assisted code completion system. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp 2727–2735
Svyatkovskoy A, Lee S, Hadjitofi A, Riechert M, Franco J, Allamanis M (2020) Fast and memory-efficient neural code completion. arXiv:2004.13651
Tu Z, Su Z, Devanbu PT (2014) On the localness of software. In: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, (FSE-22). ACM, Hong Kong, pp 269–280
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008
Wei B, Li G, Xia X, Fu Z, Jin Z (2019) Code generation as a dual task of code summarization. In: Advances in Neural Information Processing Systems, pp 6563–6573
White M, Vendome C, Vásquez ML, Poshyvanyk D (2015) Toward deep learning software repositories. In: 12th IEEE/ACM Working Conference on Mining Software Repositories, MSR 2015. IEEE Computer Society, Florence, pp 334–345
Wilcoxon F (1945) Individual comparisons by ranking methods. Biometr Bullet 1(6):80–83
Yao Z, Peddamail JR, Sun H (2019) Coacor: code annotation for code retrieval with reinforcement learning. In: The World Wide Web Conference, pp 2203–2214
Zaremoodi P, Buntine WL, Haffari G (2018) Adaptive knowledge sharing in multi-task learning: Improving low-resource neural machine translation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Volume 2: Short Papers. Association for Computational Linguistics, Melbourne, pp 656–661
Acknowledgments
This research is supported by the National Natural Science Foundation of China under Grant Nos. 62072007, 62192731, 62192733, 61832009, 62192730. Zhi Jin and Ge Li are corresponding authors.
Author information
Authors and Affiliations
Corresponding authors
Additional information
Communicated by: Yann-Gaël Guéhéneuc, Shinpei Hayashi and Michel R. V. Chaudron
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article belongs to the Topical Collection: International Conference on Program Comprehension (ICPC)
Rights and permissions
About this article
Cite this article
Liu, F., Li, G., Wei, B. et al. A unified multi-task learning model for AST-level and token-level code completion. Empir Software Eng 27, 91 (2022). https://doi.org/10.1007/s10664-022-10140-7
Accepted:
Published:
DOI: https://doi.org/10.1007/s10664-022-10140-7