Skip to main content
SpringerLink
Log in

Transformer: A General Framework from Machine Translation to Others

  • Review
  • Published:
Machine Intelligence Research Aims and scope Submit manuscript

Abstract

Machine translation is an important and challenging task that aims at automatically translating natural language sentences from one language into another. Recently, Transformer-based neural machine translation (NMT) has achieved great break-throughs and has become a new mainstream method in both methodology and applications. In this article, we conduct an overview of Transformer-based NMT and its extension to other tasks. Specifically, we first introduce the framework of Transformer, discuss the main challenges in NMT and list the representative methods for each challenge. Then, the public resources and toolkits in NMT are listed. Meanwhile, the extensions of Transformer in other tasks, including the other natural language processing tasks, computer vision tasks, audio tasks and multi-modal tasks, are briefly presented. Finally, possible future research directions are suggested.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. P. F. Brown, V. J. D. Pietra, S. A. D. Pietra, R. L. Mercer. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, vol. 19, no. 2, pp. 263–311, 1993.

    Google Scholar 

  2. P. Koehn, F. J. Och, D. Marcu. Statistical phrase-based translation. In Proceedings of the Human Language Technology Conference of North American Chapter of Association for Computational Linguistics, Edmonton, Canada, pp. 127–133, 2003.

  3. J. J. Zhang, C. Q. Zong. Deep neural networks in machine translation: An overview. IEEE Intelligent Systems, vol. 30, no. 5, pp. 16–25, 2015. DOI: https://doi.org/10.1109/MIS.2015.69.

    Google Scholar 

  4. N. Kalchbrenner, P. Blunsom. Recurrent continuous translation models. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, Seattle, USA, pp. 1700–1709, 2013.

    Google Scholar 

  5. I. Sutskever, O. Vinyals, Q. V. Le. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, Canada, pp. 3104–3112, 2014.

  6. D. Bahdanau, K. Cho, Y. Bengio. Neural machine translation by jointly learning to align and translate. [Online], Available: https://arxiv.org/abs/1409.0473, 2015.

  7. Y. H. Wu, M. Schuster, Z. F. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. B. Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, J. Dean. Google’s neural machine translation system: Bridging the gap between human and machine translation. [Online], Available: https://arxiv.org/abs/1609.08144, 2016.

  8. H. Hassan, A. Aue, C. Chen, V. Chowdhary, J. Clark, C. Federmann, X. D. Huang, M. Junczys-Dowmunt, W. Lewis, M. Li, S. J. Liu, T. Y. Liu, R. Q. Luo, A. Menezes, T. Qin, F. Seide, X. Tan, F. Tian, L. J. Wu, S. Z. Wu, Y. C. Xia, D. D. Zhang, Z. R. Zhang, M. Zhou. Achieving human parity on automatic Chinese to English news translation. [Online], Available: https://arxiv.org/abs/1803.05567, 2018.

  9. J. Gehring, M. Auli, D. Grangier, D. Yarats, Y. N. Dauphin. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, pp. 1243–1252, 2017.

  10. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, pp. 6000–6010, 2017.

  11. T. Luong, H. Pham, C. D. Manning. Effective approaches to attention-based neural machine translation. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, Lisbon, Portugal, pp. 1412–1421, 2015. DOI: https://doi.org/10.18653/v1/D15-1166.

    Google Scholar 

  12. J. Gehring, M. Auli, D. Grangier, Y. Dauphin. A convolutional encoder model for neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL, Vancouver, Canada, pp. 123–135, 2017. DOI: https://doi.org/10.18653/vl/P17-1012.

    Google Scholar 

  13. R. Sennrich, B. Haddow, A. Birch. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL, Berlin, Germany, pp. 86–96, 2016. DOI: https://doi.org/10.18653/vl/P16-1009.

    Google Scholar 

  14. A. Karakanta, J. Dehdari, J. van Genabith. Neural machine translation for low-resource languages without parallel corpora. Machine Translation, vol. 32, no. 1, pp. 167–189, 2018. DOI: https://doi.org/10.1007/s10590-017-9203-5.

    Google Scholar 

  15. S. Edunov, M. Ott, M. Auli, D. Grangier. Understanding back-translation at scale. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, Brussels, Belgium, pp. 489–500, 2018. DOI: https://doi.org/10.18653/vl/D18-1045.

    Google Scholar 

  16. J. J. Zhang, C. Q. Zong. Exploiting source-side monolingual data in neural machine translation. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, Austin, USA, pp. 1535–1545, 2016. DOI: https://doi.org/10.18653/vl/D16-1160.

    Google Scholar 

  17. Y. Cheng, W. Xu, Z. J. He, W. He, H. Wu, M. S. Sun, Y. Liu. Semi-supervised learning for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL, Berlin, Germany, pp. 1965–1974, 2016. DOI: https://doi.org/10.18653/v1/P16-1185.

    Google Scholar 

  18. D. He, Y. C. Xia, T. Qin, L. W. Wang, N. H. Yu, T. Y. Liu, W. Y. Ma. Dual learning for machine translation. In Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain, pp. 820–828, 2016.

  19. M. Artetxe, G. Labaka, E. Agirre, K. Cho. Unsupervised neural machine translation. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada, 2018.

  20. G. Lample, A. Conneau, L. Denoyer, M. Ranzato. Unsupervised machine translation using monolingual corpora only. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada, 2018.

  21. T. Mikolov, Q. V. Le, I. Sutskever. Exploiting similarities among languages for machine translation. [Online], Available: https://arxiv.org/abs/1309.4168, 2013.

  22. M. Zhang, Y. Liu, H. B. Luan, M. S. Sun. Adversarial training for unsupervised bilingual lexicon induction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL, Vancouver, Canada, pp. 1959–1970, 2017. DOI: https://doi.org/10.18653/v1/P17-1179.

    Google Scholar 

  23. M. Artetxe, G. Labaka, E. Agirre. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL, Melbourne, Australia, pp. 789–798, 2018. DOI: https://doi.org/10.18653/v1/P18-1073.

    Google Scholar 

  24. T. Mohiuddin, S. Joty. Unsupervised word translation with adversarial autoencoder. Computational Linguistics, vol. 46, no. 2, pp. 257–288, 2020. DOI: https://doi.org/10.1162/coli_a_00374.

    Google Scholar 

  25. M. Artetxe, G. Labaka, E. Agirre. Unsupervised statistical machine translation. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, Brussels, Belgium, pp. 3632–3642, 2018. DOI: https://doi.org/10.18653/v1/D18-1399.

    Google Scholar 

  26. G. Lample, M. Ott, A. Conneau, L. Denoyer, M. Ranzato. Phrase-based & neural unsupervised machine translation. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, Brussels, Belgium, pp. 5039–5049, 2018. DOI: https://doi.org/10.18653/v1/D18-1549.

    Google Scholar 

  27. S. Ren, Z. R. Zhang, S. J. Liu, M. Zhou, S. Ma. Unsupervised neural machine translation with SMT as posterior regularization. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, the 31st Innovative Applications of Artificial Intelligence Conference and 9th AAAI Symposium on Educational Advances in Artificial Intelligence, Honolulu, USA, Article number 30, 2019. DOI: https://doi.org/10.1609/aaai.v33i01.3301241.

  28. M. Artetxe, G. Labaka, E. Agirre. An effective approach to unsupervised machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL, Florence, Italy, pp. 194–203, 2019. DOI: https://doi.org/10.18653/v1/P19-1019.

    Google Scholar 

  29. X. Garcia, A. Siddhant, O. Firat, A. Parikh. Harnessing multilinguality in unsupervised machine translation for rare languages. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, pp. 1126–1137, 2021. DOI: https://doi.org/10.18653/v1/2021.naacl-main.89.

  30. A. Üstün, A. Berard, L. Besacier, M. Gallé. Multilingual unsupervised neural machine translation with denoising adapters. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, pp. 6650–6662, 2021. DOI: https://doi.org/10.18653/v1/2021.emnlp-main.533.

  31. G. H. Chen, S. M. Ma, Y. Chen, L. Dong, D. D. Zhang, J. Pan, W. P. Wang, F. R. Wei. Zero-shot cross-lingual transfer of neural machine translation with multilingual pretrained encoders. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, pp. 15–26, 2021. DOI: https://doi.org/10.18653/v1/2021.emnlpmain.2.

  32. G. H. Chen, S. M. Ma, Y. Chen, D. D. Zhang, J. Pan, W. P. Wang, F. R. Wei. Towards making the most of cross-lingual transfer for zero-shot neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, Dublin, Ireland, pp. 142–157, 2022. DOI: https://doi.org/10.18653/v1/2022.acl-long.12.

    Google Scholar 

  33. O. Firat, K. Cho, Y. Bengio. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, San Diego, USA, pp. 866–875, 2016. DOI: https://doi.org/10.18653/v1/N16-1101.

    Google Scholar 

  34. M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. H. Wu, Z. F. Chen, N. Thorat, F. Viégas, M. Wattenberg, G. Corrado, M. Hughes, J. Dean. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, vol. 5, pp. 339–351, 2017. DOI: https://doi.org/10.1162/tacl_a_00065.

    Google Scholar 

  35. R. Aharoni, M. Johnson, O. Firat. Massively multilingual neural machine translation. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, Minneapolis, USA, pp. 3874–3884, 2019. DOI: https://doi.org/10.18653/v1/N19-1388.

    Google Scholar 

  36. D. Sachan, G. Neubig. Parameter sharing methods for multilingual self-attentional translation models. In Proceedings of the 3rd Conference on Machine Translation: Research Papers, ACL, Brussels, Belgium, pp. 261–271, 2018. DOI: https://doi.org/10.18653/v1/W18-6327.

    Google Scholar 

  37. G. Blackwood, M. Ballesteros, T. Ward. Multilingual neural machine translation with task-specific attention. In Proceedings of the 27th International Conference on Computational Linguistics, ACL, Santa Fe, USA, pp. 3112–3122, 2018.

    Google Scholar 

  38. A. Bapna, O. Firat. Simple, scalable adaptation for neural machine translation. In Proceedings of Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, ACL, Hong Kong, China, pp. 1538–1548, 2019. DOI: https://doi.org/10.18653/v1/D19-1165.

    Google Scholar 

  39. A. Eriguchi, S. F. Xie, T. Qin, H. Hassan. Building multi-lingual machine translation systems that serve arbitrary XY translations. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, Seattle, USA, pp. 600–606, 2022. DOI: https://doi.org/10.18653/v1/2022.naacl-main.44.

    Google Scholar 

  40. W. Y. Xie, Y. Feng, S. H. Gu, D. Yu. Importance-based neuron allocation for multilingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL, pp. 5725–5737, 2021. DOI: https://doi.org/10.18653/v1/2021.acl-long.445.

  41. Z. H. Lin, L. W. Wu, M. X. Wang, L. Li. Learning language specific sub-network for multilingual machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL, pp. 293–305, 2021. DOI: https://doi.org/10.18653/v1/2021.acl-long.25.

  42. Q. Wang, J. J. Zhang. Parameter differentiation based multilingual neural machine translation. In Proceedings of the 36th AAAI Conference on Artificial Intelligence, the 34th AAAI Conference on Innovative Applications of Artificial Intelligence and the 12th IAAI Symposium on Educational Advances in Artificial Intelligence, pp. 11440–11448, 2022.

  43. X. P. Qiu, T. X. Sun, Y. G. Xu, Y. F. Shao, N. Dai, X. J. Huang. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, vol. 63, no. 10, pp. 1872–1897, 2020. DOI: https://doi.org/10.1007/s11431-020-1647-3.

    Google Scholar 

  44. M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, L. Zettlemoyer. Deep contextualized word representations. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, New Orleans, USA, pp. 2227–2237, 2018. DOI: https://doi.org/10.18653/v1/N18-1202.

    Google Scholar 

  45. J. Devlin, M. W. Chang, K. Lee, K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, Minneapolis, USA, pp. 4171–4186, 2019. DOI: https://doi.org/10.18653/v1/N19-1423.

    Google Scholar 

  46. S. Edunov, A. Baevski, M. Auli. Pre-trained language model representations for language generation. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, Minneapolis, USA, pp. 4052–4059, 2019. DOI: https://doi.org/10.18653/v1/N19-1409.

    Google Scholar 

  47. J. H. Zhu, Y. C. Xia, L. J. Wu, D. He, T. Qin, W. G. Zhou, H. Q. Li, T. Y. Liu. Incorporating BERT into neural machine translation. In Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.

  48. J. C. Yang, M. X. Wang, H. Zhou, C. Q. Zhao, W. N. Zhang, Y. Yu, L. Li. Towards making the most of BERT in neural machine translation. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 5, pp. 9378–9385, 2020. DOI: https://doi.org/10.1609/aaai.v34i05.6479.

    Google Scholar 

  49. K. T. Song, X. Tan, T. Qin, J. F. Lu, T. Y. Liu. MASS: Masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, Long Beach, USA, pp. 5926–5936, 2019.

  50. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Q. Zhou, W. Li, P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, vol. 21, no. 1, Article number 140, 2020.

  51. M. Lewis, Y. H. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL, pp. 7871–7880, 2020. DOI: https://doi.org/10.18653/v1/2020.acl-main.703.

  52. W. X. Wang, W. X. Jiao, Y. C. Hao, X. Wang, S. M. Shi, Z. P. Tu, M. Lyu. Understanding and improving sequence-to-sequence pretraining for neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, Dublin, Ireland, pp. 2591–2600, 2022. DOI: https://doi.org/10.18653/v1/2022.acl-long.185.

    Google Scholar 

  53. T. Pires, E. Schlinger, D. Garrette. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL, Florence, Italy, pp. 4996–5001, 2019. DOI: https://doi.org/10.18653/v1/P19-1493.

    Google Scholar 

  54. A. Conneau, G. Lample. Cross-lingual language model pretraining. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, Article number 634, 2019.

  55. Z. H. Lin, X. Pan, M. X. Wang, X. P. Qiu, J. T. Feng, H. Zhou, L. Li. Pre-training multilingual neural machine translation by leveraging alignment information. In Proceedings of Conference on Empirical Methods in Natural Language Processing ACL, pp. 2649–2663, 2020. DOI: https://doi.org/10.18653/v1/2020.emnlp-main.210.

  56. Z. W. Chi, L. Dong, S. M. Ma, S. H. Huang, S. Singhal, X. L. Mao, H. Y. Huang, X. Song, F. R. Wei. mT6: Multilingual pretrained text-to-text transformer with translation pairs. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, pp. 1671–1683, 2021. DOI: https://doi.org/10.18653/v1/2021.emnlp-main.125.

  57. Y. H. Liu, J. T. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, L. Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, vol. 8, pp. 726–742, 2020. DOI: https://doi.org/10.1162/tacl_a_00343.

    Google Scholar 

  58. P. F. Li, L. Y. Li, M. Zhang, M. H. Wu, Q. Liu. Universal conditional masked language pre-training for neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, Dublin, Ireland, pp. 6379–6391, 2022. DOI: https://doi.org/10.18653/v1/2022.acl-long.442.

    Google Scholar 

  59. R. Bawden, R. Sennrich, A. Birch, B. Haddow. Evaluating discourse phenomena in neural machine translation. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, New Orleans, USA, pp. 1304–1313, 2018. DOI: https://doi.org/10.18653/v1/N18-1118.

    Google Scholar 

  60. E. Voita, P. Serdyukov, R. Sennrich, I. Titov. Context-aware neural machine translation learns anaphora resolution. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL, Melbourne, Australia, pp. 1264–1274, 2018. DOI: https://doi.org/10.18653/V1/P18-1117.

    Google Scholar 

  61. J. C. Zhang, H. B. Luan, M. S. Sun, F. F. Zhai, J. F. Xu, M. Zhang, Y. Liu. Improving the transformer translation model with document-level context. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, Brussels, Belgium, pp. 533–542, 2018. DOI: https://doi.org/10.18653/v1/D18-1049.

    Google Scholar 

  62. B. Zhang, A. Bapna, M. Johnson, A. Dabirmoghaddam, N. Arivazhagan, O. Firat. Multilingual document-level translation enables zero-shot transfer from sentences to documents. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, Dublin, Ireland, pp. 4176–4192, 2022. DOI: https://doi.org/10.18653/v1/2022.acl-long.287.

    Google Scholar 

  63. Z. P. Tu, Y. Liu, S. M. Shi, T. Zhang. Learning to remember translation history with a continuous cache. Transactions of the Association for Computational Linguistics, vol. 6, pp. 407–420, 2018. DOI: https://doi.org/10.1162/tacl_a_00029.

    Google Scholar 

  64. E. Voita, R. Sennrich, I. Titov. When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL, Florence, Italy, pp. 1198–1212, 2019. DOI: https://doi.org/10.18653/v1/P19-1116.

    Google Scholar 

  65. E. Voita, R. Sennrich, I. Titov. Context-aware monolingual repair for neural machine translation. In Proceedings of Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, ACL, Hong Kong, China, pp. 877–886, 2019. DOI: https://doi.org/10.18653/v1/D19-1081.

    Google Scholar 

  66. L. Lupo, M. Dinarelli, L. Besacier. Divide and rule: Effective pre-training for context-aware multi-encoder translation models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, Dublin, Ireland, pp. 4557–4572, 2022. DOI: https://doi.org/10.18653/v1/2022.acl-long.312.

    Google Scholar 

  67. S. H. Kuang, D. Y. Xiong, W. H. Luo, G. D. Zhou. Modeling coherence for neural machine translation with dynamic and topic caches. In Proceedings of the 27th International Conference on Computational Linguistics, ACL, Santa Fe, USA, pp. 596–606, 2018.

    Google Scholar 

  68. X. M. Kang, Y. Zhao, J. J. Zhang, C. Q. Zong. Dynamic context selection for document-level neural machine translation via reinforcement learning. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, pp. 2242–2254, 2020. DOI: https://doi.org/10.18653/vl/2020.emnlp-main.175.

  69. P. Y. Huang, F. Liu, S. R. Shiang, J. Oh, C. Dyer. Attention-based multimodal neural machine translation. In Proceedings of the 1st Conference on Machine Translation: Volume 2, Shared Task Papers, ACL, Berlin, Germany, pp. 639–645, 2016. DOI: https://doi.org/10.18653/v1/W16-2360.

    Google Scholar 

  70. I. Calixto, M. Rios, W. Aziz. Latent variable model for multi-modal translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL, Florence, Italy, pp. 6392–6405, 2019. DOI: https://doi.org/10.18653/v1/P19-1642.

    Google Scholar 

  71. J. Ive, P. Madhyastha, L. Specia. Distilling translations with visual awareness. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL, Florence, Italy, pp. 6525–6538, 2019. DOI: https://doi.org/10.18653/v1/P19-1653.

    Google Scholar 

  72. X. Huang, J. J. Zhang, C. Q. Zong. Entity-level cross-modal learning improves multi-modal machine translation. In Proceedings of the Findings of the Association for Computational Linguistics: EMNLP, ACL, Punta Cana, Dominican Republic, pp. 1067–1080, 2021. DOI: https://doi.org/10.18653/v1/2021.findings-emnlp.92.

    Google Scholar 

  73. D. X. Wang, D. Y. Xiong. Efficient object-level visual context modeling for multimodal machine translation: Masking irrelevant objects helps grounding. In Proceedings of the 35 th AAAI Conference on Artificial Intelligence, Palo Alto, USA, pp. 2720–2728, 2021. DOI: https://doi.org/10.1609/aaai.v35i4.16376.

  74. B. Li, C. H. Lv, Z. F. Zhou, T. Zhou, T. Xiao, A. X. Ma, J. B. Zhu. On vision features in multimodal machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, Dublin, Ireland, pp. 6327–6337, 2022. DOI: https://doi.org/10.18653/v1/2022.acl-long.438.

    Google Scholar 

  75. A. Bérard, O. Pietquin, L. Besacier, C. Servan. Listen and translate: A proof of concept for end-to-end speech-to-text translation. In Proceedings of the NIPS Workshop on End-to-end Learning for Speech and Audio Processing, Barcelona, Spain, 2016. [Online], Available: https://hal.science/hal-01408086.

  76. R. J. Weiss, J. Chorowski, N. Jaitly, Y. H. Wu, Z. F. Chen. Sequence-to-sequence models can directly translate foreign speech. In Proceedings of Interspeech, Stockholm, Sweden, pp. 2625–2629, 2017. DOI: https://doi.org/10.21437/Inter-speech.2017-503.

  77. R. Ye, M. X. Wang, L. Li. Cross-modal contrastive learning for speech translation. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, Seattle, USA, pp. 5099–5113, 2022. DOI: https://doi.org/10.18653/v1/2022.naacl-main.376.

    Google Scholar 

  78. G. Sant, G. I. Gállego, B. Alastruey, M. R. Costa-Jussà. Multiformer: A head-configurable transformer-based model for direct speech translation. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, ACL, Seattle, USA, pp. 277–284, 2022. DOI: https://doi.org/10.18653/v1/2022.naacl-srw.34.

    Google Scholar 

  79. T. K. Lam, S. Schamoni, S. Riezler. Sample, translate, recombine: Leveraging audio alignments for data augmentation in end-to-end speech translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, Dublin, Ireland, pp. 245–254, 2022. DOI: https://doi.org/10.18653/v1/2022.acl-short.27.

    Google Scholar 

  80. S. Bansal, H. Kamper, K. Livescu, A. Lopez, S. Goldwater. Pre-training on high-resource speech recognition improves low-resource speech-to-text translation. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, Minneapolis, USA, pp. 58–68, 2019. DOI: https://doi.org/10.18653/v1/N19-1006.

    Google Scholar 

  81. C. Y. Wang, Y. Wu, S. J. Liu, Z. L. Yang, M. Zhou. Bridging the gap between pre-training and fine-tuning for end-to-end speech translation. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 5, pp. 9161–9168, 2020. DOI: https://doi.org/10.1609/aaai.v34i05.6452.

    Google Scholar 

  82. S. Y. Chen, C. Y. Wang, Z. Y. Chen, Y. Wu, S. J. Liu, Z. Chen, J. Y. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. M. Qian, Y. Qian, J. Wu, M. Zeng, X. Z. Yu, F. R. Wei. WavLM: Large-scale self-supervised pretraining for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022. DOI: https://doi.org/10.1109/JSTSP.2022.3188113.

    Google Scholar 

  83. Y. Tang, H. Y. Gong, N. Dong, C. H. Wang, W. N. Hsu, J. T. Gu, A. Baevski, X. Li, A. Mohamed, M. Auli, J. Pino. Unified speech-text pre-training for speech translation and recognition. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, Dublin, Ireland, pp. 1488–1499, 2022. DOI: https://doi.org/10.18653/v1/2022.acl-long.105.

    Google Scholar 

  84. Y. C. Liu, H. Xiong, J. J. Zhang, Z. J. He, H. Wu, H. F. Wang, C. Q. Zong. End-to-end speech translation with knowledge distillation. Proceedings of the 20th Annual Conference of the International Speech Communication Association, Graz, Austria, pp. 1128–1132, 2019.

  85. H. Inaguma, T. Kawahara, S. Watanabe. Source and target bidirectional knowledge distillation for end-to-end speech translation. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, pp. 1872–1881, 2021. DOI: https://doi.org/10.18653/v1/2021.naacl-main.150.

  86. Y. Tang, J. Pino, X. Li, C. H. Wang, D. Genzel. Improving speech translation by understanding and learning from the auxiliary text translation task. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL, pp. 4252–4261, 2021. DOI: https://doi.org/10.18653/v1/2021.acl-long.328.

  87. Y. Ren, J. L. Liu, X. Tan, C. Zhang, T. Qin, Z. Zhao, T. Y. Liu. SimulSpeech: End-to-end simultaneous speech to text translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL, pp. 3787–3796, 2020. DOI: https://doi.org/10.18653/v1/2020.acl-main.350.

  88. E. Salesky, M. Sperber, A. W. Black. Exploring phoneme-level speech representations for end-to-end speech translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL, Florence, Italy, pp. 1835–1841, 2019. DOI: https://doi.org/10.18653/v1/P19-1179.

    Google Scholar 

  89. E. Salesky, A. W. Black. Phone features improve speech translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL, pp. 2388–2397, 2020. DOI: https://doi.org/10.18653/v1/2020.acl-main.217.

  90. J. T. Gu, J. Bradbury, C. M. Xiong, V. O. K. Li, R. Socher. Non-autoregressive neural machine translation. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada, 2018. [Online], Available: https://openreview.net/pdf?id=B118BtlCb.

  91. X. W. Zhang, J. S. Su, Y. Qin, Y. Liu, R. R. Ji, H. J. Wang. Asynchronous bidirectional decoding for neural machine translation. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and 8th AAAI Symposium on Educational Advances in Artificial Intelligence, New Orleans, USA, Article number 699, 2018.

  92. L. Zhou, J. J. Zhang, C. Q. Zong. Synchronous bidirectional neural machine translation. Transactions of the Association for Computational Linguistics, vol. 7, pp. 91–105, 2019. DOI: https://doi.org/10.1162/tacl_a_00256.

    Google Scholar 

  93. J. J. Zhang, L. Zhou, Y. Zhao, C. Q. Zong. Synchronous bidirectional inference for neural sequence generation. Artificial Intelligence, vol. 281, Article number 103234, 2020. DOI: https://doi.org/10.1016/j.artint.2020.103234.

  94. Y. R. Wang, F. Tian, D. He, T. Qin, C. X. Zhai, T. Y. Liu. Non-autoregressive machine translation with auxiliary regularization. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence, the 31st Innovative Applications of Artificial Intelligence Conference and the 9th AAAI Symposium on Educational Advances in Artificial Intelligence, Honolulu, USA, Article number 659, 2019. DOI: https://doi.org/10.1609/aaai.v33i01.33015377.

  95. L. Zhou, J. J. Zhang, Y. Zhao, C. Q. Zong. Non-autoregressive neural machine translation with distortion model. In Proceedings of the 9th CCF International Conference on Natural Language Processing and Chinese Computing, Springer, Zhengzhou, China, pp. 403–415, 2020. DOI: https://doi.org/10.1007/978-3-030-60450-9_32.

    Google Scholar 

  96. L. Ding, L. Y. Wang, S. M. Shi, D. C. Tao, Z. P. Tu. Redistributing low-frequency words: Making the most of monolingual data in non-autoregressive translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, Dublin, Ireland, pp. 2417–2426, 2022. DOI: https://doi.org/10.18653/v1/2022.acl-long.172.

    Google Scholar 

  97. C. Z. Shao, X. F. Wu, Y. Feng. One reference is not enough: Diverse distillation with reference selection for non-autoregressive translation. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, Seattle, USA, pp. 3779–3791, 2022. DOI: https://doi.org/10.18653/v1/2022.naacl-main.277.

    Google Scholar 

  98. C. Q. Wang, J. Zhang, H. Q. Chen. Semi-autoregressive neural machine translation. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, Brussels, Belgium, pp. 479–488, 2018. DOI: https://doi.org/10.18653/v1/D18-1044.

    Google Scholar 

  99. M. Ghazvininejad, O. Levy, Y. H. Liu, L. Zettlemoyer. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, ACL, Hong Kong, China, pp. 6112–6121, 2019. DOI: https://doi.org/10.18653/v1/D19-1633.

    Google Scholar 

  100. J. T. Gu, C. H. Wang, J. K. Zhao. Levenshtein transformer. In Proceedings of the 33rd Conference on Neural Information Processing Systems, Vancouver, Canada, 2019.

  101. M. H. Zhu, J. L. Wang, C. G. Yan. Non-autoregressive neural machine translation with consistency regularization optimized variational framework. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, Seattle, USA, pp. 607–617, 2022. DOI: https://doi.org/10.18653/v1/2022.naacl-main.45.

    Google Scholar 

  102. J. Lee, E. Mansimov, K. Cho. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, Brussels, Belgium, pp. 1173–1182, 2018. DOI: https://doi.org/10.18653/v1/D18-1149.

    Google Scholar 

  103. C. Zeng, J. J. Chen, T. Y. Zhuang, R. Xu, H. Yang, Q. Ying, S. M. Tao, Y. H. Xiao. Neighbors are not strangers: Improving non-autoregressive translation under low-frequency lexical constraints. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, Seattle, USA, pp. 5777–5790, 2022. DOI: https://doi.org/10.18653/v1/2022.naacl-main.424.

    Google Scholar 

  104. R. Sennrich, B. Haddow, A. Birch. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the 1st Conference on Machine Translation: Volume 2, Shared Task Papers, ACL, Berlin, Germany, pp. 371–376, 2016. DOI: https://doi.org/10.18653/v1/W16-2323.

    Google Scholar 

  105. Y. C. Liu, L. Zhou, Y. N. Wang, Y. Zhao, J. J. Zhang, C. Q. Zong. A comparable study on model averaging, ensembling and reranking in NMT. In Proceedings of the 7th CCF International Conference on Natural Language Processing and Chinese Computing, Springer, Hohhot, China, pp. 299–308, 2018. DOI: https://doi.org/10.1007/978-3-319-99501-4_26.

    Google Scholar 

  106. L. M. Liu, M. Utiyama, A. Finch, E. Sumita. Agreement on target-bidirectional neural machine translation. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, San Diego, USA, pp. 411–416, 2016. DOI: https://doi.org/10.18653/v1/N16-1046.

    Google Scholar 

  107. Z. R. Zhang, S. Z. Wu, S. J. Liu, M. Li, M. Zhou, T. Xu. Regularizing neural machine translation by target-bidirectional agreement. In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 1, pp. 443–450, 2019. DOI: https://doi.org/10.1609/aaai.v33i01.3301443.

    Google Scholar 

  108. J. S. Su, X. W. Zhang, Q. Lin, Y. Qin, J. F. Yao, Y. Liu. Exploiting reverse target-side contexts for neural machine translation via asynchronous bidirectional decoding. Artificial Intelligence, vol. 277, Article number 103168, 2019. DOI: https://doi.org/10.1016/j.artint.2019.103168.

  109. L. Zhou, J. J. Zhang, C. Q. Zong, H. Yu. Sequence generation: From both sides to the middle. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, Macao, China, pp. 5471–5477, 2019.

  110. P. Arthur, G. Neubig, S. Nakamura. Incorporating discrete translation lexicons into neural machine translation. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, Austin, USA, pp. 1557–1567, 2016. DOI: https://doi.org/10.18653/v1/D16-1162.

    Google Scholar 

  111. Y. Feng, S. Y. Zhang, A. D. Zhang, D. Wang, A. Abel. Memory-augmented neural machine translation. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, Copenhagen, Denmark, pp. 1390–1399, 2017. DOI: https://doi.org/10.18653/v1/D17-1146.

    Google Scholar 

  112. J. C. Zhang, Y. Liu, H. B. Luan, J. F. Xu, M. S. Sun. Prior knowledge integration for neural machine translation using posterior regularization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL, Vancouver, Canada, pp. 1514–1523, 2017. DOI: https://doi.org/10.18653/v1/P17-1139.

    Google Scholar 

  113. Y. Zhao, J. J. Zhang, Z. J. He, C. Q. Zong, H. Wu. Addressing troublesome words in neural machine translation. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, Brussels, Belgium, pp. 391–400, 2018. DOI: https://doi.org/10.18653/v1/D18-1036.

    Google Scholar 

  114. M. T. Luong, C. D. Manning. Achieving open vocabulary neural machine translation with hybrid word-character models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL, Berlin, Germany, pp. 1054–1063, 2016. DOI: https://doi.org/10.18653/v1/P16-1100.

    Google Scholar 

  115. R. Sennrich, B. Haddow, A. Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL, Berlin, Germany, pp. 1715–1725, 2016. DOI: https://doi.org/10.18653/v1/P16-1162.

    Google Scholar 

  116. X. Wang, Z. P. Tu, D. Y. Xiong, M. Zhang. Translating phrases in neural machine translation. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, Copenhagen, Denmark, pp. 1421–1431, 2017. DOI: https://doi.org/10.18653/v1/D17-1149.

    Google Scholar 

  117. L. Dahlmann, E. Matusov, P. Petrushkov, S. Khadivi. Neural machine translation leveraging phrase-based models in a hybrid search. In Proceedings of Conference on Empirical Methods in Natural Language Processing, ACL, Copenhagen, Denmark, pp. 1411–1420, 2017. DOI: https://doi.org/10.18653/v1/D17-1148.

    Google Scholar 

  118. Y. Zhao, Y. N. Wang, J. J. Zhang, C. Q. Zong. Phrase table as recommendation memory for neural machine translation. In Proceedings of International Joint Conference on Artificial Intelligence, Stockholm, Sweden, pp.4609–4615, 2018.

  119. H. F. Xu, J. van Genabith, D. Y. Xiong, Q. H. Liu, J. Y. Zhang. Learning source phrase representations for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL, pp. 386–396, 2020. DOI: https://doi.org/10.18653/v1/2020.acl-main.37.

  120. M. Huck, V. Hangya, A. Fraser. Better OOV translation with bilingual terminology mining. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL, Florence, Italy, pp. 5809–5815, 2019. DOI: https://doi.org/10.18653/v1/P19-1581.

    Google Scholar 

  121. G. Dinu, P. Mathur, M. Federico, Y. Al-Onaizan. Training neural machine translation to apply terminology constraints. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL, Florence, Italy, pp. 3063–3068, 2019. DOI: https://doi.org/10.18653/v1/P19-1294.

    Google Scholar 

  122. Y. Zhao, J. J. Zhang, Y. Zhou, C. Q. Zong. Knowledge graphs enhanced neural machine translation. In Proceedings of the 29th International Joint Conference on Artificial Intelligence, pp. 4039–4045, 2021. DOI: https://doi.org/10.24963/ij-cai.2020/559.

  123. Y. Zhao, L. Xiang, J. N. Zhu, J. J. Zhang, Y. Zhou, C. Q. Zong. Knowledge graph enhanced neural machine translation via multi-task learning on sub-entity granularity. In Proceedings of the 28th International Conference on Computational Linguistics, ACL, Barcelona, Spain, pp. 4495–4505, 2020. DOI: https://doi.org/10.18653/v1/2020.coling-main.397.

    Google Scholar 

  124. J. J. Hu, H. Hayashi, K. Cho, G. Neubig. DEEP: DEnoising entity pre-training for neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, Dublin, Ireland, pp. 1753–1766, 2022. DOI: https://doi.org/10.18653/v1/2022.acl-long.123.

    Google Scholar 

  125. J. Tiedemann, S. Thottingal. OPUS-MT-Building open translation services for the World. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, Lisboa, Portugal, pp. 479–480, 2020.

  126. P. Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X, Phuket, Thailand, pp. 79–86, 2005.

  127. D. Elliott, S. Frank, K. Sima’an, L. Specia. Multi30K: Multilingual English-German image descriptions. In Proceedings of the 5th Workshop on Vision and Language, ACL, Berlin, Germany, pp. 70–74, 2016. DOI: https://doi.org/10.18653/v1/W16-3210.

    Google Scholar 

  128. A. C. Kocabiyikoglu, L. Besacier, O. Kraif. Augmenting librispeech with French translations: A multimodal corpus for direct speech translation evaluation. In Proceedings of the 11th International Conference on Language Resources and Evaluation, Miyazaki, Japan, 2018.

  129. M. A. Di Gangi, R. Cattoni, L. Bentivogli, M. Negri, M. Turchi. MuST-C: A multilingual speech translation corpus. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ACL, Minneapolis, USA, pp. 2012–2017, 2019. DOI: https://doi.org/10.18653/v1/N19-1202.

    Google Scholar 

  130. M. Cettolo, C. Girardi, M. Federico. WIT3: Web inventory of transcribed and translated talks. In Proceedings of the 16th Annual conference of the European Association for Machine Translation, Trento, Italy, pp. 261–268, 2012.

  131. M. Post, G. Kumar, A. Lopez, D. Karakos, C. Callison-Burch, S. Khudanpur. Improved speech-to-text translation with the fisher and callhome Spanish-English speech translation corpus. In Proceedings of the 10th International Workshop on Spoken Language Translation, Heidelberg, Germany, 2013. [Online], Available: https://aclanthology.org/2013.iwslt-papers.14/.

  132. J. J. Zhang, C. Q. Zong. Neural machine translation: Challenges, progress and future. Science China Technological Sciences, vol. 63, no. 10, pp. 2028–2050, 2020. DOI: https://doi.org/10.1007/s11431-020-1632-x.

    Google Scholar 

  133. F. Stahlberg. Neural machine translation: A review. Journal of Artificial Intelligence Research, vol. 69, pp. 343–418, 2020. DOI: https://doi.org/10.1613/jair.1.12007.

    MathSciNet  Google Scholar 

  134. Z. X. Tan, S. Wang, Z. H. Yang, G. Chen, X. C. Huang, M. S. Sun, Y. Liu. Neural machine translation: A review of methods, resources, and tools. AI Open, vol. 1, pp. 5–21, 2020. DOI: https://doi.org/10.1016/j.aiopen.2020.11.001.

    Google Scholar 

  135. T. X. Sun, X. Y. Liu, X. P. Qiu, X. J. Huang. Paradigm shift in natural language processing. Machine Intelligence Research, vol. 19, no. 3, pp. 169–183, 2022. DOI: https://doi.org/10.1007/s11633-022-1331-6.

    Google Scholar 

  136. Y. H. Liu, M. Ott, N. Goyal, J. F. Du, M. Joshi, D. Q. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. [Online], Available: https://arxiv.org/abs/1907.11692, 2019.

  137. M. Joshi, D. Q. Chen, Y. H. Liu, D. S. Weld, L. Zettlemoyer, O. Levy. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, vol. 8, pp. 64–77, 2020. DOI: https://doi.org/10.1162/tacl_a_00300.

    Google Scholar 

  138. A. Radford, K. Narasimhan, T. Salimans, I. Sutskever. Improving language understanding by generative pretraining. [Online], Available: https://openai.com/research/language-unsupervised, Nov. 7, 2022.

  139. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, vol. 1, no. 8, Article number 9, 2019.

  140. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amodei. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, Article number 159, 2020.

  141. P. J. Liu, M. Saleh, E. Pot, B. Goodrich, R. Sepassi, L. Kaiser, N. Shazeer. Generating Wikipedia by summarizing long sequences. In Proceedings of the 6th International Conference on Learning Representations, Vancouver, Canada, 2018.

  142. W. Xiao, I. Beltagy, G. Carenini, A. Cohan. PRIMERA: Pyramid-based masked sentence pre-training for multi-document summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, Dublin, Ireland, pp. 5245–5263, 2022. DOI: https://doi.org/10.18653/v1/2022.acl-long.360.

    Google Scholar 

  143. Y. Rui, V. I. S. Carmona, M. Pourvali, Y. Xing, W. W. Yi, H. B. Ruan, Y. Zhang. Knowledge mining: A cross-disciplinary survey. Machine Intelligence Research, vol. 19, no. 2, pp. 89–114, 2022. DOI: https://doi.org/10.1007/s11633-022-1323-6.

    Google Scholar 

  144. A. Saxena, A. Kochsiek, R. Gemulla. Sequence-to-sequence knowledge graph completion and question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, Dublin, Ireland, pp. 2814–2828, 2022. DOI: https://doi.org/10.18653/v1/2022.acl-long.201.

    Google Scholar 

  145. T. X. Sun, Y. F. Shao, X. P. Qiu, Q. P. Guo, Y. R. Hu, X. J. Huang, Z. Zhang. CoLAKE: Contextualized language and knowledge embedding. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, pp. 3660–3670, 2020. DOI: https://doi.org/10.18653/v1/2020.coling-main.327.

  146. M. Henderson, I. Vulic, D. Gerz, I. Casanueva, P. Budzianowski, S. Coope, G. Spithourakis, T. H. Wen, N. Mrkšić, P. H. Su. Training neural response selection for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL, Florence, Italy, pp. 5392–5404, 2019. DOI: https://doi.org/10.18653/v1/P19-1536.

    Google Scholar 

  147. Y. Z. Zhang, S. Q. Sun, M. Galley, Y. C. Chen, C. Brockett, X. Gao, J. F. Gao, J. J. Liu, B. Dolan. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, ACL, pp. 270–278, 2020. DOI: https://doi.org/10.18653/v1/2020.acl-demos.30.

  148. Z. Y. Ma, J. J. Li, G. H. Li, Y. J. Cheng. UniTranSeR: A unified transformer semantic representation framework for multimodal task-oriented dialog system. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, ACL, Dublin, Ireland, pp. 103–114, 2022. DOI: https://doi.org/10.18653/v1/2022.acl-long.9.

    Google Scholar 

  149. H. B. Bao, L. Dong, F. R. Wei, W. H. Wang, N. Yang, L. Cui, S. H. Piao, M. Zhou. Inspecting unification of encoding and matching with transformer: A case study of machine reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, ACL, Hong Kong, China, pp. 14–18, 2019. DOI: https://doi.org/10.18653/v1/D19-5802.

    Google Scholar 

  150. Z. W. Bai, J. P. Liu, M. Q. Wang, C. X. Yuan, X. J. Wang. Exploiting diverse information in pre-trained language model for multi-choice machine reading comprehension. Applied Sciences, vol. 12, no. 6, Article number 3072, 2022. DOI: https://doi.org/10.3390/app12063072.

  151. K. Nishida, I. Saito, K. Nishida, K. Shinoda, A. Otsuka, H. Asano, J. Tomita. Multi-style generative reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, ACL, Florence, Italy, pp. 2273–2284, 2019. DOI: https://doi.org/10.18653/v1/P19-1220.

    Google Scholar 

  152. C. Zhao, C. Y. Xiong, C. Rosset, X. Song, P. N. Bennett, S. Tiwary. Transformer-XH: Multi-evidence reasoning with extra hop attention. In Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.

  153. G. Izacard, E. Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, ACL, pp. 874–880, 2021. DOI: https://doi.org/10.18653/v1/2021.eacl-main.74.

  154. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. H. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby. An image is worth 16×16 words: Transformers for image recognition at scale. In Proceedings of the 9th International Conference on Learning Representations, 2021.

  155. Z. Liu, Y. T. Lin, Y. Cao, H. Hu, Y. X. Wei, Z. Zhang, S. Lin, B. N. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Montreal, Canada, pp. 10012–10022, 2021. DOI: https://doi.org/10.1109/ICCV48922.2021.00986.

    Google Scholar 

  156. X. Q. Zhang, R. H. Jiang, C. X. Fan, T. Y. Tong, T. Wang, P. C. Huang. Advances in deep learning methods for visual tracking: Literature review and fundamentals. International Journal of Automation and Computing, vol. 18, no. 3, pp. 311–333, 2021. DOI: https://doi.org/10.1007/s11633-020-1274-8.

    Google Scholar 

  157. K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, Y. Wang. Transformer in transformer. In Proceedings of the 35th Annual Conference on Neural Information Processing Systems, pp. 15908–15919, 2021.

  158. L. Yuan, Y. P. Chen, T. Wang, W. H. Yu, Y. J. Shi, Z. H. Jiang, F. E. H. Tay, J. S. Feng, S. C. Yan. Tokens-to-token ViT: Training vision transformers from scratch on ImageNet. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Montreal, Canada, pp. 558–567, 2021. DOI: https://doi.org/10.1109/ICCV48922.2021.00060.

    Google Scholar 

  159. X. Y. Dong, J. M. Bao, D. D. Chen, W. M. Zhang, N. H. Yu, L. Yuan, D. Chen, B. N. Guo. CSWin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, New Orleans, USA, pp. 12124–12134, 2022. DOI: https://doi.org/10.1109/CV-PR52688.2022.01181.

    Google Scholar 

  160. Y. F. Jiang, S. Y. Chang, Z. Y. Wang. TransGAN: Two pure transformers can make one strong GAN, and that can scale up. In Proceedings of the 35th Annual Conference on Neural Information Processing Systems, pp. 14745–14758, 2021.

  161. P. Esser, R. Rombach, B. Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Nashville, USA, pp. 12873–12883, 2021. DOI: https://doi.org/10.1109/CVPR46437.2021.01268.

    Google Scholar 

  162. Y. Gong, Y. A. Chung, J. Glass. AST: Audio spectrogram transformer. [Online], Available: https://arxiv.org/abs/2104.01778, 2021.

  163. L. H. Dong, S. Xu, B. Xu. Speech-transformer: A no-recurrence sequence-to-sequence model for speech recognition. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, Calgary, Canada, pp. 5884–5888, 2018. DOI: https://doi.org/10.1109/ICASSP.2018.8462506.

  164. Z. K. Tian, J. Y. Yi, Y. Bai, J. H. Tao, S. Zhang, Z. Q. Wen. Synchronous transformers for end-to-end speech recognition. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, Barcelona, Spain, pp. 7884–7888, 2020. DOI: https://doi.org/10.1109/ICAS-SP40776.2020.9054260.

  165. N. H. Li, S. J. Liu, Y. Q. Liu, S. Zhao, M. Liu. Neural speech synthesis with transformer network. Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 1, pp. 6706–6713, 2019. DOI: https://doi.org/10.1609/aaai.v33i01.33016706.

    Google Scholar 

  166. Y. Jia, H. G. Zen, J. Shen, Y. Zhang, Y. H. Wu. PnG BERT: Augmented BERT on phonemes and graphemes for neural TTS. In Proceedings of the 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, pp. 151–155, 2021.

  167. G. H. Xu, W. Song, Z. C. Zhang, C. Zhang, X. D. He, B. W. Zhou. Improving prosody modelling with cross-utterance Bert embeddings for end-to-end speech synthesis. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, Toronto, Canada, pp. 6079–6083, 2021. DOI: https://doi.org/10.1109/ICASSP39728.2021.9414102.

  168. R. H. Hu, A. Singh, T. Darrell, M. Rohrbach. Iterative answer prediction with pointer-augmented multimodal transformers for textVQA. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Seattle, USA, pp. 9989–9999, 2020. DOI: https://doi.org/10.1109/CVPR42600.2020.01001.

    Google Scholar 

  169. A. F. Biten, R. Litman, Y. S. Xie, S. Appalaraju, R. Manmatha. LaTr: Layout-aware transformer for scene-text VQA. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, New Orleans, USA, pp. 16527–16537, 2022. DOI: https://doi.org/10.1109/CV-PR52688.2022.01605.

    Google Scholar 

  170. Y. S. Chuang, C. L. Liu, H. Y. Lee, L. S. Lee. Speech-BERT: An audio-and-text jointly learned language model for end-to-end spoken question answering. In Proceedings of the 21st Annual Conference of the International Speech Communication Association, Shanghai, China, pp. 4168–4172, 2020.

  171. L. H. Li, M. Yatskar, D. Yin, C. J. Hsieh, K. W. Chang. VisualBERT: A simple and performant baseline for vision and language. [Online], Available: https://arxiv.org/abs/1908.03557, 2019.

  172. W. J. Su, X. Z. Zhu, Y. Cao, B. Li, L. W. Lu, F. R. Wei, J. F. Dai. VL-BERT: Pre-training of generic visual-linguistic representations. In Proceedings of the 8th International Conference on Learning Representations, Addis Ababa, Ethiopia, 2020.

  173. C. Sun, A. Myers, C. Vondrick, K. Murphy, C. Schmid. VideoBERT: A joint model for video and language representation learning. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Seoul, Republic of Korea, pp. 7463–7472, 2019. DOI: https://doi.org/10.1109/ICCV.2019.00756.

    Google Scholar 

  174. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, pp. 8748–8763, 2021.

  175. R. H. Hu, A. Singh. UniT: Multimodal multitask learning with a unified transformer. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Montreal, Canada, pp. 1419–1429, 2021. DOI: https://doi.org/10.1109/ICCV48922.2021.00147.

    Google Scholar 

  176. L. Yeganova, D. Wiemann, M. Neves, F. Vezzani, A. Siu, I. J. Unanue, M. Oronoz, N. Mah, A. Névéol, D. Martinez, R. Bawden, G. M. Di Nunzio, R. Roller, P. Thomas, C. Grozea, O. Perez-de-Viñaspre, M. V. Navarro, A. J. Yepes. Findings of the WMT 2021 biomedical translation shared task: Summaries of animal experiments as new test set. In Proceedings of the 6th Conference on Machine Translation, ACL, pp. 664–683, 2021.

Download references

Acknowledgements

This work was supported by Natural Science Foundation of China (Nos. 62006224 and 62122088).

Author information

Authors and Affiliations

  1. National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Yang Zhao, Jiajun Zhang & Chengqing Zong

  2. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100190, China

    Yang Zhao, Jiajun Zhang & Chengqing Zong

Authors
  1. Yang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiajun Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chengqing Zong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yang Zhao.

Additional information

Yang Zhao received the Ph. D. degree in pattern recognition and intelligent system from Institute of Automation, Chinese Academy of Sciences, China in 2019. He is currently an associate professor with National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, China.

His research interests include machine translation and natural language processing.

Jiajun Zhang received the Ph.D. degree in computer science from Institute of Automation, Chinese Academy of Sciences, China in 2011. He is currently a professor with National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, China.

His research interests include machine translation, multilingual natural language processing and deep learning.

Chengqing Zong received the Ph. D. degree in computer science from Institute of Computing Technology, Chinese Academy of Sciences, China in 1998. He is currently a professor with National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, China. He is a member of the International Committee on Computational Linguistics and the President of Asian Federation of Natural Language Processing. He is an Associate Editor for the ACM Transactions on Asian and Low-resource Language Information Processing and an Editorial Board Member of the IEEE Intelligent Systems, the journal Machine Translation, and the Journal of Computer Science and Technology. He served ACL-IJCNLP 2015 as the PC Co-Chair, IJCAI 2017, IJCAI-ECAI 2018, and AAAI 2019 as the Area Chair, and IJCNLP 2017 as the General Chair.

His research interests include natural language processing, machine translation and sentiment analysis.

Declarations of Conflict of interest

The authors declared that they have no conflicts of interest to this work.

Colored figures are available in the online version at https://link.springer.com/journal/11633

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, Y., Zhang, J. & Zong, C. Transformer: A General Framework from Machine Translation to Others. Mach. Intell. Res. 20, 514–538 (2023). https://doi.org/10.1007/s11633-022-1393-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-022-1393-5

Keywords

Search

Navigation