Low-Resource Neural Machine Translation Using Fast Meta-learning Method

Abstract

Data sparsity is fundamental reason that affects the quality of low-resource neural machine translation models (NMT), although transfer learning methods can alleviate data sparsity by introducing external knowledge. However, the pre-trained model parameters are only suitable for the current task set, which does not ensure better performance improvement in downstream tasks. Although meta-learning methods have better potential, while meta-parameters are determined by the second-order gradient term corresponding to a specific task, which directly leads to the consumption of computing resources. In addition, the integration and unified representation of external knowledge is also the main factor to improve performance. Therefore, we proposed a fast meta-learning method using multiple-aligned word embedding representation, which can map all languages to the word embedding space of the target language without seed dictionary. Meanwhile, we update the meta-parameters by calculating the cumulative gradient on different tasks to replace the second-order term in the ordinary meta-learning method, which not only pays attention to the potential but also improves the calculation efficiency. We conducted experiments on three low-resource translation tasks of the CCMT2019 data set and found that our method significantly improves the model quality compared with traditional methods, which fully reflects the effectiveness of the proposed method.