Self-feeding training method for semi-supervised grammatical error correction

Highlights

• Self-feeding training method can help training grammatical error correction models.

• Data generation models trained with unlabeled data can generate realistic data.

• Denoising artificial noise generates various wrong data pairs for training.

• This method can be extended to domains and languages without labeled data.

Abstract

Grammatical error correction (GEC) has been successful with deep and complex neural machine translation models, but the annotated data to train the model are scarce. We propose a novel self-feeding training method that generates incorrect sentences from freely available correct sentences. The proposed training method can generate appropriate wrong sentences from unlabeled sentences, using a data generation model trained as an autoencoder. It can also add artificial noise to correct sentences to automatically generate incorrect sentences. We show that the GEC models trained with the self-feeding training method are successful without extra annotated data or deeper neural network-based models, achieving F_0.5 score of 0.5982 on the CoNLL-2014 Shared Task test data with a transformer model. The results also show that fully unlabeled training is possible for data-scarce domains and languages.

Introduction

Grammatical error correction (GEC) is a task to correct possible grammatical errors in a sentence. GEC can be directly used for multiple services such as writing assistants, dialog systems, and language learning. The task has gained interest and has been studied for decades, but the task is still challenging.

Neural machine translation (NMT)-based GEC studies have surpassed the previous statistical machine translation (SMT)-based systems. The NMT approach regards the GEC task as translating the erroneous language into a correct one. The NMT approach has been successful using deeper and more complex neural network structures, from recurrent neural networks to convolutional neural networks and transformers (Yuan and Briscoe, 2016, Chollampatt and Ng, 2018, Zhao et al., 2019, Katsumata and Komachi, 2020, Rothe et al., 2021).

The NMT-based GEC systems have become more complex rapidly, but few annotated datasets to train the deep models have been published. The supervised training data for GEC are pairs of grammatically incorrect sentences written by language learners (e.g., English-learning students) and correct sentences edited and annotated by language experts (e.g., English teachers). GEC annotated datasets, including the NUS Corpus of Learner English (NUCLE) (Dahlmeier et al., 2013), Lang-8 data (Mizumoto et al., 2011), CLC FCE dataset (Yannakoudakis et al., 2011), Cambridge English Write & Improve (W&I) and LOCNESS corpus (Bryant et al., 2019) have been opened to the public. However, the sizes and variety of the datasets are incomparable to other tasks, such as machine translation, and they are too small to train the deep and complex NMT models without overfitting, as such annotations are expensive and time-consuming to collect. Some GEC researches use their additional closed annotated data to improve their models, however such results are not reproducible for other research groups.

Compared to annotated data pairs, correct sentences without any annotations are easy to collect from news articles, books, documents or web pages. Some studies tried to synthesize parallel data from those sentences. Open synthetic datasets, such as C4${}_{200M}$ (Stahlberg and Kumar, 2021), or reproducible methods to build them can help other groups train and research GEC models.

To alleviate the data-scarce problem of GEC, we propose a self-feeding training method that incorporates semi-supervised learning and paired data generation using unannotated sentences. Our method is able to generate paired training data from unlabeled sentences with a data generation model. Furthermore, our novel method can select appropriate samples with simple but effective rules, learn to recover wrong words from noisy sentences, and effectively utilize the generated data to train the GEC model. This method can yield multiple erroneous sentences from one unlabeled sentence without additional annotated data or large-scale language model trained with massive data. We experimented the proposed method with multiple semi-supervised training methods on a transformer-based GEC models. The results show that our proposed method is effective for training previous GEC models without extra labeled data or changes to the previous NMT model structure.