Boosting source code suggestion with self-supervised Transformer Gated Highway

Highlights

• Transformer Gated Highway with two different self-supervised strategies.

• Shown training procedures impact significantly on modeling performance.

• Similar learning procedures with different model architectures, marginal difference

• Extensive evaluation with two datasets (Java & C#) with five different baselines.

Abstract

Attention-based transformer language models have shown significant performance gains in various natural language tasks. In this work, we explore the impact of transformer language models on the task of source code suggestion. The core intention of this work is to boost the modeling performance for the source code suggestion task and to explore how the training procedures and model architectures impact modeling performance. Additionally, we propose a transformer-based self-supervised learning technique called Transformer Gated Highway that outperforms recurrent and transformer language models of comparable size. The proposed approach combines the Transformer language model with Gated Highway introducing a notion of recurrence. We compare the performance of the proposed approach with transformer-based BERT (CodeTran), RoBERTa (RoBERTaCode), GPT2 (TravTrans), CodeGen and recurrent neural language-based LSTM (CodeLSTM) models. Moreover, we have experimented with various architectural settings for the transformer models to evaluate their impact on modeling performance. The extensive evaluation of the presented approach exhibits better performance on two programming language datasets; $Java$ and $C\#$. Additionally, we have adopted the presented approach for the syntax error correction task to predict the correct syntax token to render its possible implications for other source code modeling tasks.

Introduction

Source code suggestion that is also known as code completion is the most frequently used feature of Integrated Development Environments (IDEs). This feature offers predictions for the next possible token to the software developers, allowing the developers to speed up the software development process. The effectiveness of automatically providing code suggestions requires the next possible code token to be predicted in the top ranks. Traditional code suggestion tools rely on the code context already written in the IDEs and order the predictions based on the counts or arrange them into alphabetical order which significantly limits their capabilities.

This motivated the use of machine learning for the task of source code suggestion, as machine learning methods not only learn source code context effectively but also serve as a code sighting method. These approaches can be trained on real-world codebases that are capable of providing predictions that have not been observed by the IDE yet. Early approaches adapted n-gram language models on sequences of source code tokens (Hindle et al., 2012, Franks et al., 2015). Later, Recurrent Neural Networks (RNNs) (White et al., 2015) and their variants (Hussain et al., 2020, Hussain et al., 2021) have been adopted to improve their accuracy. Although predictive models cannot be expected to be perfect, the accuracy of the current state-of-the-art methods leaves a substantial margin to be improved.

Recently, attention-based transformer model (Vaswani et al., 2017) and their variants (Devlin et al., 2018, Liu et al., 2019a) have revolutionized NLP domain by removing recurrent layers completely. The transformer-based models allow parallelization and achieve a new state-of-the-art in translation and other NLP tasks. The architecture such as BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2018) that are based on transformer architecture, have shown superior performance as compared to RNNs. These models are trained on a large amount of data and with high-performance computational machines having several GPUs.

To this end, we present an empirical study to explore the capabilities of neural language models for the task of source code suggestion. Here, we consider the source code suggestion task as a classification task. Among the various neural language models proposed in the literature, we focus on traditional recurrent neural language models and revolutionized transformer-based language models. Specifically, we experimented with the transformer-based CodeTran (BERT), RoBERTaCode (RoBERTa), TravTrans (GPT2), CodeGen, and LSTM-based CodeLSTM models. We propose a methodology called Transformer Gated Highway which employs a self-supervised learning approach along with an enhanced encoder–decoder model that outperforms the recurrent and transformer baselines. The Gated Highway introduces a notion of recurrence, resulting in better modeling performance. We experimented with two different self-supervised learning approaches to study their impact on modeling performance. Additionally, we show the generalization of our approach by employing it for a different source code modeling task of syntax error correction. For the syntax error correction task, our target is to predict the correct syntax token (abstract token) rather than locating the position of the syntax error. The presented approach is evaluated extensively with two programming language datasets; $Java$ and $C\#$. The results have suggested that on average the proposed approach achieves around 95% accuracy rate for the correct syntax token prediction within the top three ranks and 90% for the source code suggestion task within the top-10 ranks for $Java$. The core intentions of this work are to explore the impact of learning strategies and model architecture design in the modeling of source code for enhanced performance. Specifically, we are interested in studying how various factors (learning strategies and model architecture) impact modeling performance.

The main contributions of this work are as follows:

• We propose a methodology called Transformer Gated Highway (TGH) with self-supervised learning techniques that outperform four baselines including one recurrent-based and four transformer-based. Using a customized decoder along with a specifically tailored learning procedure, the presented Transformer Gated Highway exhibits enhanced performance compared to other baselines.

• We evaluate the proposed approach with two different programming language datasets, $Java$, and $C\#$, for the task of source code suggestion. Additionally, we employed the proposed methodology for the prediction of correct syntax tokens to reflect its generalization and discuss other possible implications.

• Additionally, we experimented with different model architecture settings to evaluate their impact on the modeling performance. Specifically, we experimented with data size and model architecture designs for performance evaluation. We make the material used in this work publicly available.¹

As mentioned earlier, the intent here is to boost the modeling performance for the source code suggestion task. Additionally, we explore how training procedures (i.e., encoding, self-supervised learning) and model architectures impact modeling performance. Specifically, we aim to explore the following research questions during this study.

• How well does the proposed approach perform compared to the transformer models for the task of source code suggestion? To explore and answer this question, we first trained four transformer-based models as baselines for the task of source code suggestion. Specifically, we experimented with CodeTran which employs BERT, ROBERTACode which employs RoBERTa, TravTrans which employs GPT2, and CodeGen language models to evaluate the modeling performance. Additionally, we experiment with different self-supervised learning techniques and explore their impact on modeling performance. The intent here is to have a baseline and push that baseline for enhanced performance. Finally, we compare the enhanced baselines with the proposed approach for performance evaluation.

• What is the modeling performance of traditional Recurrent Neural Networks (RNNs) compared to the proposed approach for the Source code suggestion task? In this research question, we explore the capabilities of traditional RNN-based models compared to the proposed approach. Here, we aim to study if the recurrent language models are completely dominated by transformer language models or if their performance can still be improved. Specifically, we aim to evaluate how different learning procedures with similar model architecture will impact the modeling performance for the task of source code suggestion.

• Exploring the impact of model design, data size, and the generalization of the proposed approach. Here, we study the impact of model design by altering the encoder and decoder architectures. We also aim to study the impact of data size on modeling performance to learn how data size impacts modeling performance. We also aim to study if similar learning procedures with different models have any significant impact on the modeling performance for the task of source code suggestion. Finally, to validate the generalization we employ the proposed approach for the prediction of correct syntax tokens and discuss its applicability for other related tasks.

The rest of the paper is organized in the following manner. We discussed the related background in Section 2. The proposed methodology and the model architecture are discussed in detail in Sections 3 Proposed methodology, 4 Transformer Gated Highway. We discussed the training and evaluation procedures in Section 5 and the results are discussed in Section 6. We discuss the threats to validity in Section 7 followed by the related work discussed in Section 8. Finally, we conclude the work in Section 9.