Abstract
Transformer-based neural machine translation (NMT) models have achieved state-of-the-art performance in the machine translation community. These models learn the translation knowledge from the parallel corpus through the attention mechanism automatically. However, the model fails to consider the semantic importance of words, where content words play a more important role than functional words in a sentence. This issue is particularly prominent for low-resource translation tasks, where insufficient parallel data results in poor translation quality. To alleviate this issue, a content word augmentation (CWA) method is proposed to improve the encoder for low-resource translation tasks. The main steps are as follows: Firstly, words in a sentence are classified into content and function words based on the content word selection algorithm; Next, two fusion strategies are employed by incorporating the word embedding of content words into the NMT model to augment the encoder. The results of experiments on several translation tasks show that the CWA method outperforms the strong baseline, significantly improving the BLEU score range from 0.24 to 0.57.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, vol. 27 (2014)
Johnson, M., et al.: Google’s multilingual neural machine translation system: enabling zero-shot translation. Trans. Assoc. Comput. Linguist. 5, 339–351 (2017)
Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin, Y.N.: Convolutional sequence to sequence learning. In: International Conference on Machine Learning, pp. 1243–1252. PMLR (2017)
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Wang, X., Tu, Z., Zhang, M.: Incorporating statistical machine translation word knowledge into neural machine translation. IEEE/ACM Trans. Audio Speech Lang. Process. 26(12), 2255–2266 (2018)
Han, D., Li, J., Li, Y., Zhang, M., Zhou, G.: Explicitly modeling word translations in neural machine translation. ACM Trans. Asian Low-Resourc. Lang. Inf. Process. (TALLIP) 19(1), 1–17 (2019)
Arthur, P., Neubig, G., Nakamura, S.: Incorporating discrete translation lexicons into neural machine translation. arXiv preprint arXiv:1606.02006 (2016)
Wanghao, G., Jiangwei, F., Keliang, Z.: Advance research on neural machine translation integrating linguistic knowledge. J. Front. Comput. Sci. Technol. 15(7), 1183 (2021)
Chen, H., Huang, S., Chiang, D., Dai, X., Chen, J.: Combining character and word information in neural machine translation using a multi-level attention. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1284–1293 (2018)
Wang, F., Chen, W., Yang, Z., Xu, S., Xu, B.: Hybrid attention for Chinese character-level neural machine translation. Neurocomputing 358, 44–52 (2019)
Dahlmann, L., Matusov, E., Petrushkov, P., Khadivi, S.: Neural machine translation leveraging phrase-based models in a hybrid search. arXiv preprint arXiv:1708.03271 (2017)
Aharoni, R., Goldberg, Y.: Towards string-to-tree neural machine translation. arXiv preprint arXiv:1704.04743 (2017)
Eriguchi, A., Hashimoto, K., Tsuruoka, Y.: Tree-to-sequence attentional neural machine translation. arXiv preprint arXiv:1603.06075 (2016)
Jean, S., Cho, K., Memisevic, R., Bengio, Y.: On using very large target vocabulary for neural machine translation. arXiv preprint arXiv:1412.2007 (2014)
Weng, R., Huang, S., Zheng, Z., Dai, X., Chen, J.: Neural machine translation with word predictions. arXiv preprint arXiv:1708.01771 (2017)
Chen, K., Wang, R., Utiyama, M., Sumita, E.: Content word aware neural machine translation. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 358–364 (2020)
Ott, M., et al.: fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038 (2019)
Bugliarello, E., Okazaki, N.: Enhancing machine translation with dependency-aware self-attention. arXiv preprint arXiv:1909.03149 (2019)
Edunov, S., Ott, M., Auli, M., Grangier, D., Ranzato, M.: Classical structured prediction losses for sequence to sequence learning. arXiv preprint arXiv:1711.04956 (2017)
Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909 (2015)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318 (2002)
Acknowledgement
This work was supported by National Natural Science Foundation of Liaoning Province, China (Grant no. 2021-YKLH-12, 2022-YKLH-18), Scientific Research Foundation of Liaoning Province (Grant no. LJKQZ2021184), High-level talents research project of Yingkou Institute of Technology (Grant No. YJRC202026).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Li, F., Zhao, Z., Chi, C., Yan, H., Zhang, Z. (2023). A Content Word Augmentation Method for Low-Resource Neural Machine Translation. In: Huang, DS., Premaratne, P., Jin, B., Qu, B., Jo, KH., Hussain, A. (eds) Advanced Intelligent Computing Technology and Applications. ICIC 2023. Lecture Notes in Computer Science(), vol 14089. Springer, Singapore. https://doi.org/10.1007/978-981-99-4752-2_59
Download citation
DOI: https://doi.org/10.1007/978-981-99-4752-2_59
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-4751-5
Online ISBN: 978-981-99-4752-2
eBook Packages: Computer ScienceComputer Science (R0)