Skip to main content
SpringerLink
Log in

Machine translation and its evaluation: a study

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

Machine translation (namely MT) has been one of the most popular fields in computational linguistics and Artificial Intelligence (AI). As one of the most promising approaches, MT can potentially break the language barrier of people from all over the world. Despite a number of studies in MT, there are few studies in summarizing and comparing MT methods. To this end, in this paper, we principally focus on presenting the two mainstream MT schemes: statistical machine translation (SMT) and neural machine translation (NMT), including their basic rationales and developments. Meanwhile, the detailed translation models are also presented, such as the word-based model, syntax-based model, and phrase-based model in statistical machine translation. Similarly, approaches in NMT, such as the recurrent neural network-based, attention mechanism-based, and transformer-based models are presented. Last but not least, the evaluation approaches also play an important role in helping developers to improve their methods better in MT. The prevailing machine translation evaluation methodologies are also presented in this article.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30
Fig. 31
Fig. 32
Fig. 33
Fig. 34
Fig. 35
Fig. 36
Fig. 37
Fig. 38
Fig. 39
Fig. 40
Fig. 41
Fig. 42
Fig. 43
Fig. 44

Similar content being viewed by others

Notes

  1. SYSTRAN https://www.systran.net/en/translate/.

  2. Apertium https://www.apertium.org/index.eng.html.

  3. GramTrans https://gramtrans.com/.

  4. Google AI Blog https://ai.googleblog.com/2016/09/a-neural-network-for-machine.html.

  5. Microsoft Translator Blog https://www.microsoft.com/en-us/translator/blog/2016/11/15/microsoft-translator-launching-neural-network-based-translations-for-all-its-speech-languages/.

  6. Moses https://www.statmt.org/moses/.

  7. www.promt.com.

  8. www.systransoft.com.

  9. Machine Translation Research at Google https://research.google/pubs/?area=machine-translation.

  10. Machine Translation Research at Microsoft https://www.microsoft.com/en-us/research/group/machine-translation-group/.

  11. Machine Translation Research at Meta AI https://ai.facebook.com/research/NLP.

  12. OpenNMT https://opennmt.net/.

  13. Google Translate https://translate.google.com/.

  14. Europarl Datasets https://www.statmt.org/europarl/.

  15. WMT’14 Translation Task https://www.statmt.org/wmt14/translation-task.html.

  16. WMT’15 Translation Task https://www.statmt.org/wmt15/translation-task.html.

  17. Hugging Face Transformers Index https://huggingface.co/docs/transformers/index.

  18. C4 Dataset https://www.tensorflow.org/datasets/catalog/c4.

  19. WMT Workshop https://www.statmt.org/.

  20. Europarl Datasets https://www.statmt.org/europarl/.

  21. Statistical and Neural Machine Translation \(\rightarrow\) Events https://www.statmt.org/.

  22. WMT22 https://www.statmt.org/wmt22/, WMT21, WMT20, and so on.

  23. Google Research: Machine Translation https://research.google/research-areas/machine-translation/.

Abbreviations

MT :

Machine translation

NLP :

Natural Language Processing

RBMT :

Rule-based Machine Translation

CBMT :

Corpus-based Machine Translation

SMT :

Statistical machine translation

EBMT :

Example-based Machine Translation

HMT :

Hybrid Machine Translation

NMT :

Neural machine translation

EM :

Expectation-Maximization

WBMT :

Word-based Machine Translation

SBMT :

Syntax-based Machine Translation

PBMT :

Phrase-based Machine Translation

CFG :

Context-Free Grammar

SCFG :

Synchronous Context-Free Grammar

ITG :

Inversion Transduction Grammar

TER :

Translation Edit Rat

HTER :

Human-targeted Translation Edit Rat

mTER :

Multi-reference TER

GNMT :

Google’s Neural Machine Translation

BP :

Backpropagation

BT :

Back-Translation

NN :

Neural Network

CNN :

Convolutional Neural Network

RNN :

Recurrent Neural Network

TNN :

Transformer Neural Network

GRU :

Gate Recurrent Unit

LSTM :

Long-Short Term Memory

BERT :

Bidirectional Encoder Representations from Transformers

LRL :

Low-Resource Language

HRL :

High-Resource Language

Enc :

Encoder

Dec :

Decoder

ALPAC :

Automatic Language Processing Advisory Committee

DARPA :

Defense Advanced Research Projects Agency

BLEU :

Bilingual Evaluation Understudy

NIST :

National Institute of Standards and Technology

METEOR :

Metric for Evaluation of Translation with Explicit ORdering

ROUGE :

Recall-Oriented Understudy for Gisting Evaluation

OpenMT :

Open Machine Translation Evaluation

T5 :

Text-To-Text Transfer Transformer

UNK :

Unknown

References

  • ACL (2022) ACL 2014 NINTH WORKSHOP ON STATISTICAL MACHINE TRANSLATION. Available at https://www.statmt.org/wmt14/translation-task.html. Accessed 05 Apr 2022

  • ACL 2015 NINTH WORKSHOP ON STATISTICAL MACHINE TRANSLATION (2022) https://www.statmt.org/wmt15/translation-task.html. Accessed 05 Apr 2022

  • Abid A, Farooqi M, Zou J (2021) Persistent anti-muslim bias in large language models. In: Proceedings of the 2021 AAAI/ACM conference on AI, Ethics, and Society, pp 298–306

  • Agency DARP (2003) Program Translingual Information Detection, Extraction and Summarization. http://www.darpa.mil/ipto/programs/tides/. Accessed 08 Apr 2022

  • Ahmadnia B, Dorr BJ (2019) Augmenting neural machine translation through round-trip training approach. Open Comput Sci 9(1):268–278

    Google Scholar 

  • Ahmadnia B, Serrano J, Haffari G (2017) Persian-Spanish low-resource statistical machine translation through English as pivot language. Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017:24–30

    Google Scholar 

  • Ahmadnia B, Haffari G, Serrano J (2019) Round-trip training approach for bilingually low-resource statistical machine translation systems. Int J Artif Intell 17(1):167–185

    Google Scholar 

  • Ahmadnia B, Dorr BJ, Kordjamshidi P (2020) Knowledge graphs effectiveness in neural machine translation improvement. Comput Sci 21:299–318

    Google Scholar 

  • Ahmed A, Hanneman G (2005) Syntax-based statistical machine translation: a review. Comput Linguist 1:1

  • Álvaro Rocha, Adeli H, Reis LP, Costanzo S (2018) Trends and advances in information systems and technologies, vol 2. Springer, Berlin

    Google Scholar 

  • Auer S, Bizer C, Kobilarov G, Lehmann J, Cyganiak R, Ives Z (2007) Dbpedia: a nucleus for a web of open data. In: The semantic web, Springer, pp 722–735

  • Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473

  • Banerjee S, Lavie A (2005) METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp 65–72

  • Bapna A, Firat O, Wang P, Macherey W, Cheng Y, Cao Y (2022) Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation. In: ACL 2022

  • Barzilay R, Koehn P (2004) Natural Language Processing, Fall 2004, Machine Translation I, Lecture 20. CS an AI Lab, MIT, New York

    Google Scholar 

  • Bender EM (2019) A typology of ethical risks in language technology with an eye towards where transparent documentation can help. In: Future of artificial intelligence: language, ethics, technology workshop

  • Benesty J, Chen J, Huang Y, Cohen I (2009) Pearson correlation coefficient. In: Noise reduction in speech processing, Springer, pp 1–4

  • Bentivogli L, Bisazza A, Cettolo M, Federico M (2016) Neural versus phrase-based machine translation quality: a case study. arXiv preprint arXiv:1608.04631

  • Besacier L, Blanchon H (2017) Comparing statistical machine translation and neural machine translation performances. https://evaluerlata.hypotheses.org/files/2017/07/Laurent-Besacier-NMTvsSMT.pdf, laboratoire LIG, Université Grenoble Alpes, France

  • Bick E (2007) Dan2eng: wide-coverage Danish-English machine translation. In: Proceedings of Machine Translation Summit XI: Papers

  • Bollacker K, Evans C, Paritosh P, Sturge T, Taylor J (2008) Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the 2008 ACM SIGMOD international conference on management of data, pp 1247–1250

  • Bouchard G (2007) Efficient bounds for the softmax function and applications to approximate inference in hybrid models. In: NIPS 2007 workshop for approximate Bayesian inference in continuous/hybrid systems

  • Brown PF, Pietra VJD, Pietra SAD, Mercer RL (1993) The mathematics of statistical machine translation: parameter estimation. Comput Linguist 19(2):263–311

    Google Scholar 

  • Buck C, Heafield K, Van Ooyen B (2014) N-gram counts and language models from the common crawl. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pp 3579–3584

  • Canfora C, Ottmann A (2020) Risks in neural machine translation. Trans Spaces 9(1):58–77

    Google Scholar 

  • Casas N, Costa-jussà MR, Fonollosa JA, Alonso JA, Fanlo R (2021) Linguistic knowledge-based vocabularies for Neural Machine Translation. Nat Lang Eng 27(4):485–506

    Google Scholar 

  • Caswell I, Liang B (2022) Recent Advances in Google Translate. Tutorial https://ai.googleblog.com/2020/06/recent-advances-in-google-translate.html. Accessed 05 Apr 2022

  • Chah N (2018) OK Google, what is your ontology? Or: exploring freebase classification to understand Google’s Knowledge Graph. arXiv preprint arXiv:1805.03885

  • Chapelle O, Weston J, Bottou L, Vapnik V (2000) Vicinal risk minimization. Adv Neural Inf Process Syst 13:1

  • Cheng Y (2019) Joint training for pivot-based neural machine translation. In: Joint Training for Neural Machine Translation, Springer, pp 41–54

  • Cheng Y, Jiang L, Macherey W (2019) Robust Neural Machine Translation with Doubly Adversarial Inputs. In: ACL

  • Cheng Y, Jiang L, Macherey W, Eisenstein J (2020) AdvAug: Robust Adversarial Augmentation for Neural Machine Translation. In: ACL, https://arxiv.org/abs/2006.11834

  • Chiang D, Knight K (2006) An introduction to synchronous grammars. Tutorial https://www3.nd.edu/~dchiang/papers/synchtut.pdf. Accessed 4 Jan 2022

  • Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014a) Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078

  • Cho K, Van Merriënboer B, Bahdanau D, Bengio Y (2014b) On the properties of neural machine translation: encoder-decoder approaches. arXiv preprint arXiv:1409.1259

  • Colah (2015) Understanding LSTM networks. http://colah.github.io/posts/2015-08-Understanding-LSTMs/. Accessed 08 Apr 2022

  • Common Crawl corpus (2022) https://commoncrawl.org/. Accessed 30 Sep 2022

  • Corbí-Bellot AM, Forcada ML, Ortiz-Rojas S, Pérez-Ortiz JA, Ramírez-Sánchez G, Sánchez-Martínez F, Alegria I, Mayor A, Sarasola K (2005) An open-source shallow-transfer machine translation engine for the Romance languages of Spain. In: Proceedings of the 10th EAMT conference: practical applications of machine translation

  • Costa-Jussa MR, Fonollosa JA (2015) Latest trends in hybrid machine translation and its applications. Comput Speech Lang 32(1):3–10

    Google Scholar 

  • Council NR (1416) Language and Machines: Computers in Translation and Linguistics; a Report. National Research Council, https://books.google.com/books?hl=zh-CN &lr= &id=Q0ErAAAAYAAJ &oi=fnd &pg=PA1 &dq=Languages+and+machines:+computers+in+translation+and+linguistics &ots=NgytafcXa- &sig=Hc733OYAAT89yd4U-3xLdh77gEM#v=onepage &q &f=false. Accessed 08 Apr 2022

  • Cui Y, Chen Z, Wei S, Wang S, Liu T, Hu G (2016) Attention-over-attention neural networks for reading comprehension. arXiv preprint arXiv:1607.04423

  • Currey A, Heafield K (2019) Zero-resource neural machine translation with monolingual pivot data. In: Proceedings of the 3rd workshop on neural generation and translation, pp 99–107

  • Dabre R, Cromieres F, Kurohashi S, Bhattacharyya P (2015) Leveraging small multilingual corpora for smt using many pivot languages. In: Proceedings of the 2015 conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp 1192–1202

  • Dabre R, Imankulova A, Kaneko M, Chakrabarty A (2021) Simultaneous multi-pivot neural machine translation. arXiv preprint arXiv:2104.07410

  • Dajun Z, Yun W (2015) Corpus-based machine translation: Its current development and perspectives. In: International Forum of Teaching and Studies, American Scholars Press, Inc., vol 11, p 90

  • Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc B 39(1):1–22

    MathSciNet  MATH  Google Scholar 

  • Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

  • Doddington G (2002) Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In: Proceedings of the second international conference on Human Language Technology Research, Morgan Kaufmann Publishers Inc., pp 138–145

  • Edunov S, Ott M, Auli M, Grangier D (2018) Understanding back-translation at scale. arXiv preprint arXiv:1808.09381

  • Esplà-Gomis M, Forcada ML, Ramírez-Sánchez G, Hoang H (2019) ParaCrawl: Web-scale parallel corpora for the languages of the EU. In: Proceedings of Machine Translation Summit XVII: Translator, Project and User Tracks, pp 118–119

  • Europarl (2022) Europarl: European Parliament Proceedings Parallel Corpus. https://www.statmt.org/europarl/. Accessed 30 Sep 2022

  • Forcada ML, Ginestí-Rosell M, Nordfalk J, O’Regan J, Ortiz-Rojas S, Pérez-Ortiz JA, Sánchez-Martínez F, Ramírez-Sánchez G, Tyers FM (2011) Apertium: a free/open-source platform for rule-based machine translation. Mach Trans 25(2):127–144

    Google Scholar 

  • Freitag M, Torres DV, Grangier D, Cherry C, Foster G (2022) A Natural Diet: Towards Improving Naturalness of Machine Translation Output. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Online

  • Furuse O, Iida H (1992) An example-based method for transfer-driven machine translation. TMI 1992:139–150

    Google Scholar 

  • Färber M, Ell B, Menne C, Rettinger A (2015) A comparative survey of dbpedia, freebase, opencyc, wikidata, and yago. Semantic Web J 1(1):1–5

    Google Scholar 

  • Gehring J, Auli M, Grangier D, Yarats D, Dauphin YN (2017) Convolutional sequence to sequence learning. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, JMLR. org, pp 1243–1252

  • Gheini M, Ren X, May J (2021a) Cross-attention is all you need: adapting pretrained transformers for machine translation. In: Proceedings of the 2021 conference on Empirical Methods in Natural Language Processing, pp 1754–1765

  • Gheini M, Ren X, May J (2021b) On the strengths of cross-attention in pretrained transformers for machine translation

  • Gispert Ramis A (2007) Introducing linguistic knowledge into statistical machine translation. Universitat Politècnica de Catalunya

  • Graves A (2012) Long short-term memory. In: Supervised sequence labelling with recurrent neural networks. Springer, Berlin. pp 37–45

  • Guo Z, Huang Z, Zhu KQ, Chen G, Zhang K, Chen B, Huang F (2021) Automatically paraphrasing via sentence reconstruction and round-trip translation. IJCAI

  • Habash N, Zalmout N, Taji D, Hoang H, Alzate M (2017) A Parallel Corpus for Evaluating Machine Translation between Arabic and European Languages. In: Proceedings of the 15th conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, Association for Computational Linguistics, Valencia, Spain, pp 235–241. https://aclanthology.org/E17-2038

  • Hardmeier C (2012) Discourse in statistical machine translation. a survey and a case study. Discours Revue de linguistique, psycholinguistique et informatique A journal of linguistics, psycholinguistics and computational linguistics (11)

  • He D, Xia Y, Qin T, Wang L, Yu N, Liu TY, Ma WY (2016) Dual learning for machine translation. Adv Neural Inf Process Syst 29

  • Hecht-Nielsen R (1992) Theory of the backpropagation neural network. In: Neural networks for perception, Elsevier, pp 65–93

  • Hill DC, Gombay C, Sanchez O, Woappi B, Romero Vélez AS, Davidson S, Richardson EZ (2022) Lost in machine translation: The promises and pitfalls of machine translation for multilingual group work in global health education. Discov Educ 1(1):1–5

    Google Scholar 

  • Horváth I (2022) AI in interpreting: ethical considerations. Across Lang Cult 23(1):1–13

    Google Scholar 

  • Huck M, Birch A (2015) The Edinburgh machine translation systems for IWSLT 2015. In: Proceedings of the 12th International Workshop on Spoken Language Translation: Evaluation Campaign

  • Hull DA, Grefenstette G (1996) Querying across languages: a dictionary-based approach to multilingual information retrieval. In: Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, ACM, pp 49–57

  • IWSLT (2018) International Workshop on Spoken Language Translation. https://workshop2018.iwslt.org/. Accessed 08 Apr 2022

  • Imankulova A, Sato T, Komachi M (2019) Filtered pseudo-parallel corpus improves low-resource neural machine translation. ACM Trans Asian Low-Resour Lang Inf Process 19(2):1–16

    Google Scholar 

  • Islam M, Anik M, Hoque S, Islam A et al (2021) Towards achieving a delicate blending between rule-based translator and neural machine translator. Neural Comput Appl 33(18):12141–12167

    Google Scholar 

  • Jean S, Cho K, Memisevic R, Bengio Y (2015) On Using Very Large Target Vocabulary for Neural Machine Translation. In: Proceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp 1–10

  • Jehl L, Simianer P, HIrschler J, Riezler S (2015) The Heidelberg university English-German translation system for IWSLT 2015. In: Proceedings of the 12th International Workshop on Spoken Language Translation: Evaluation Campaign

  • Jian Zhang MZ Ji Wu (2003) The improvement of automatic machine translation evaluation. J Chin Inf Process 17(6):2, http://jcip.cipsc.org.cn/CN/abstract/article_1823.shtml

  • Johnson M (2020) A scalable approach to reducing gender bias in Google translate, https://ai.googleblog.com/2020/04/a-scalable-approach-to-reducing-gender.html. Google AI Blog Accessed on 30 Sep 2022

  • Jurafsky D, Martin JH (2022) Speech and language processing: an introduction to natural language processing, computational linguistics, and speech recognition, chapter 10: machine translation and encoder-decoder models, 10.9 Bias and Ethical Issues (3rd (draft) ed.). Accessed on 08 Apr 2022

  • Kalchbrenner N, Espeholt L, Simonyan K, Oord Avd, Graves A, Kavukcuoglu K (2016) Neural machine translation in linear time. arXiv preprint arXiv:1610.10099

  • Kaljahi RSZ, Rubino R, Roturier J, Foster J, Park BB (2012) A detailed analysis of phrase-based and syntax-based machine translation: The search for systematic differences. In: Proceedings of AMTA

  • Karim R (2019) Illustrated attention. https://towardsdatascience.com/attn-illustrated-attention-5ec4ad276ee3. Accessed 08 Apr 2022

  • Kenny D, Moorkens J, Do Carmo F (2020) Fair MT: towards ethical, sustainable machine translation. Trans Spaces 9(1):1–11

    Google Scholar 

  • Kharitonova K (2021) Linguistics4fairness: neutralizing Gender Bias in neural machine translation by introducing linguistic knowledge. Master’s thesis, Universitat Politècnica de Catalunya

  • Klakow D, Peters J (2002) Testing the correlation of word error rate and perplexity. Speech Commun 38(1–2):19–28

    MATH  Google Scholar 

  • Klein G, Kim Y, Deng Y, Senellart J, Rush AM (2017) Opennmt: Open-source toolkit for neural machine translation. arXiv preprint arXiv:1701.02810

  • Ko WJ, El-Kishky A, Renduchintala A, Chaudhary V, Goyal N, Guzmán F, Fung P, Koehn P, Diab M (2021) Adapting high-resource NMT models to translate low-resource related languages without parallel data. arXiv preprint arXiv:2105.15071

  • Koehn P (2009) Statistical machine translation. Cambridge University Press

    MATH  Google Scholar 

  • Koehn P, Chiang D (2019) Special interest group of machine translation. http://www.sigmt.org. Accessed 08 Apr 2022

  • Koehn P, Hoang H, Birch A, Callison-Burch C, Federico M, Bertoldi N, Cowan B, Shen W, Moran C, Zens R et al (2007) Moses: Open source toolkit for statistical machine translation. In: Proceedings of the 45th annual meeting of the association for computational linguistics companion volume proceedings of the demo and poster sessions, pp 177–180

  • Koehn P (2009) Chapter 4 Word-based models - Statistical Machine Translation. Cambridge University Press, Cambridge. Accessed 08 Apr 2022

  • Koehn P (2009) Statistical Machine Translation Lecture 5 Syntax-Based Models. Cambridge University Press, Cambridge. Accessed 08 Apr 2022

  • Koehn P, Och FJ, Marcu D (2003) Statistical phrase-based translation. In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, Association for Computational Linguistics, pp 48–54

  • Koehn P (2005) Europarl: A parallel corpus for statistical machine translation. In: Proceedings of machine translation summit x: papers, pp 79–86

  • Koehn P (2009) Statistical Machine Translation Lecture 6 Decoding. Cambridge University Press, Cambridge. Accessed 08 Apr 2022

  • Kussul E, Baidyk T, Kasatkina L, Lukovich V (2001) Rosenblatt perceptrons for handwritten digit recognition. In: IJCNN’01. International Joint Conference on Neural Networks. Proceedings (Cat. No. 01CH37222), IEEE, vol 2, pp 1516–1520

  • Labaka G, España-Bonet C, Màrquez L, Sarasola K (2014) A hybrid machine translation architecture guided by syntax. Mach Trans 28(2):91–125

    Google Scholar 

  • Lample G, Conneau A (2019) Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291

  • Lavie A, Sagae K, Jayaraman S (2004) The significance of recall in automatic metrics for MT evaluation. In: Conference of the Association for Machine Translation in the Americas, Springer, pp 134–143

  • Li Y, Xiong D, Zhang M (2018) A survey of neural machine translation. Chinese Journal of Computers 12:2734

    MathSciNet  Google Scholar 

  • Li Q, Zhang X, Xiong J, Hwu Wm, Chen D (2019) Implementing neural machine translation with bi-directional GRU and attention mechanism on FPGAs using HLS. In: Proceedings of the 24th Asia and South Pacific Design Automation Conference, pp 693–698

  • Lin CY (2004) Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out, pp 74–81

  • Lin CY, Hovy E (2003) Automatic evaluation of summaries using n-gram co-occurrence statistics. In: Proceedings of the 2003 human language technology conference of the North American chapter of the association for computational linguistics, pp 150–157

  • Lin CY, Och FJ (2004) Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In: Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pp 605–612

  • Liu J (2020) Comparing and analyzing cohesive devices of SMT and NMT from Chinese to English: a diachronic approach. Open J Mod Linguist 10(6):765–772

    Google Scholar 

  • Liu CH, Silva CC, Wang L, Way A (2018) Pivot machine translation using chinese as pivot language. In: China workshop on machine translation. Springer, Berlin. pp 74–85

  • Liu X, Wang Y, Wang X, Xu H, Li C, Xin X (2021) Bi-directional gated recurrent unit neural network based nonlinear equalizer for coherent optical communication system. Opt Express 29(4):5923–5933

    Google Scholar 

  • Lu Y, Zhang J, Zong C (2018) Exploiting knowledge graph in neural machine translation. In: China workshop on machine translation. Springer, Berlin. pp 27–38

  • Luccioni A, Viviano J (2021) What’s in the box? an analysis of undesirable content in the Common Crawl corpus. In: Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp 182–189

  • Luong MT, Sutskever I, Le QV, Vinyals O, Zaremba W (2014) Addressing the rare word problem in neural machine translation. arXiv preprint arXiv:1410.8206

  • Luong MT, Pham H, Manning CD (2015) Effective Approaches to Attention-based Neural Machine Translation. In: Proceedings of the 2015 conference on Empirical Methods in Natural Language Processing, pp 1412–1421

  • Ma S, Sun X, Wang Y, Lin J (2018) Bag-of-words as target for neural machine translation. arXiv preprint arXiv:1805.04871

  • Marcus MP, Marcinkiewicz MA, Santorini B (1993) Building a large annotated corpus of English: the penn treebank. Comput Linguist 19(2):313–330

    Google Scholar 

  • Mehandru N, Robertson S, Salehi N (2022) Reliable and Safe Use of Machine Translation in Medical Settings. In: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (Seoul, South Korea)(FAccT’22). Association for Computing Machinery, New York, NY, USA

  • Meng F, Lu Z, Li H, Liu Q (2016) Interactive attention for neural machine translation. arXiv preprint arXiv:1610.05011

  • Moon TK (1996) The expectation-maximization algorithm. IEEE Signal Process Mag 13(6):47–60

    Google Scholar 

  • Moon J, Cho H, Park EL (2020) Revisiting round-trip translation for quality estimation. arXiv preprint arXiv:2004.13937

  • Moorkens J (2022) Ethics and machine translation. Machine translation for everyone: empowering users in the age of artificial intelligence 18:121

    Google Scholar 

  • Moussallem D, Ngonga Ngomo AC, Buitelaar P, Arcan M (2019) Utilizing knowledge graphs for neural machine translation augmentation. In: Proceedings of the 10th international conference on knowledge capture, pp 139–146

  • Mueller V (2022) An introduction to synchronous grammars. Tutorial https://medium.com/towards-data-science/attention-please-85bd0abac41. Accessed 05 Apr 2022

  • Nießen S, Ney H (2000) Improving SMT quality with morpho-syntactic analysis. In: COLING 2000 Volume 2: the 18th international conference on computational linguistics

  • Nyberg EH, Mitamura T (1992) The KANT system: Fast, accurate, high-quality translation in practical domains. In: Proceedings of the 14th conference on Computational linguistics-Volume 3, Association for Computational Linguistics, pp 1069–1073

  • Och FJ, Ney H (2003) A systematic comparison of various statistical alignment models. Comput Linguist 29(1):19–51

    MATH  Google Scholar 

  • Och FJ, Tillmann C, Ney H (1999) Improved alignment models for statistical machine translation. In: 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora

  • Och FJ, Ueffing N, Ney H (2001) An efficient A* search algorithm for statistical machine translation. In: Proceedings of the workshop on Data-driven methods in machine translation-Volume 14, Association for Computational Linguistics, pp 1–8

  • Och FJ, Gildea D, Khudanpur S, Sarkar A, Yamada K, Fraser A, Kumar S, Shen L, Smith D, Eng K et al (2003) Syntax for statistical machine translation. In: Johns Hopkins University 2003 Summer Workshop on Language Engineering, Center for Language and Speech Processing, Baltimore, MD, Tech. Rep

  • Of Standards NNI, Technology) (2010) Open Machine Translation Evaluation. https://www.nist.gov/itl/iad/mig/open-machine-translation-evaluation. Accessed 08 Apr 2022

  • Ortega JE, Castro Mamani R, Cho K (2020) Neural machine translation with a polysynthetic low resource language. Mach Trans 34(4):325–346

    Google Scholar 

  • Pal SK, Mitra S (1992) Multilayer perceptron, fuzzy sets, and classification. IEEE Trans Neural Netw 3(5):683–697

    Google Scholar 

  • Papadimitriou CH (2003) Computational complexity. Wiley, New York

    MATH  Google Scholar 

  • Papineni K, Roukos S, Ward T, Zhu WJ (2002) BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting on association for computational linguistics, Association for Computational Linguistics, pp 311–318

  • ParaCrwal (2022) ParaCrawl: Web-scale parallel corpora for the languages of the EU. https://paracrawl.eu/. Accessed 30 Sep 2022

  • Pierce J (1966) Language and machines: computers in translation and linguistics; a Report. National Research Council

  • Popović M (2017) chrF++: words helping character n-grams. In: Proceedings of the second conference on machine translation, pp 612–618

  • Prates MO, Avelar PH, Lamb LC (2020) Assessing gender bias in machine translation: a case study with google translate. Neural Comput Appl 32(10):6363–6381

    Google Scholar 

  • Pryzant R, Chung Y, Jurafsky D, Britz D (2018) JESC: Japanese-English Subtitle Corpus. In: Proceedings of the eleventh International Conference on Language Resources and Evaluation (LREC 2018)

  • Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu PJ (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res 21:1–67

    MathSciNet  MATH  Google Scholar 

  • Ramnath S, Johnson M, Gupta A, Raghuveer A (2021) HintedBT: Augmenting Back-Translation with Quality and Transliteration Hints. In: EMNLP 2021

  • Ranathunga S, Lee ESA, Skenduli MP, Shekhar R, Alam M, Kaur R (2021) Neural machine translation for low-resource languages: a survey. arXiv preprint arXiv:2106.15115

  • Ravikumar D, Kodge S, Garg I, Roy K (2020) Exploring Vicinal Risk Minimization for Lightweight Out-of-Distribution Detection. arXiv preprint arXiv:2012.08398

  • Rebele T, Suchanek F, Hoffart J, Biega J, Kuzey E, Weikum G (2016) YAGO: A multilingual knowledge base from wikipedia, wordnet, and geonames. In: International semantic web conference, Springer, pp 177–185

  • Rescigno AA, Vanmassenhove E, Monti J, Way A (2020) A case study of natural gender phenomena in translation. A comparison of Google Translate, Bing Microsoft Translator and DeepL for English to Italian, French and Spanish. In: CLiC-it

  • Richards C, Bouman WP, Seal L, Barker MJ, Nieder TO, T’Sjoen G (2016) Non-binary or genderqueer genders. Int Rev Psychiatry 28(1):95–102

    Google Scholar 

  • Ringler D, Paulheim H (2017) One knowledge graph to rule them all? Analyzing the differences between DBpedia, YAGO, Wikidata & co. In: Joint German/Austrian Conference on Artificial Intelligence (Künstliche Intelligenz), Springer, pp 366–372

  • Rocktäschel T, Grefenstette E, Hermann KM, Kočiskỳ T, Blunsom P (2015) Reasoning about entailment with neural attention. arXiv preprint arXiv:1509.06664

  • Rothman D (2021) Transformers for Natural Language Processing: build innovative deep neural network architectures for NLP with Python, PyTorch, TensorFlow, BERT, RoBERTa, and more. Packt Publishing Ltd

  • Sakamoto A (2019) Unintended consequences of translation technologies: from project managers’ perspectives. Perspectives 27(1):58–73

    Google Scholar 

  • Sara Stymne GT (2022) Phrase Based Machine Translation. Tutorial https://cl.lingfil.uu.se/kurs/MT19/slides/pbsmt.pdf. Accessed 05 Apr 2022

  • Saunders D, Sallis R, Byrne B (2020) Neural machine translation doesn’t translate gender coreference right unless you make it. In: Proceedings of the second workshop on gender bias in natural language processing, pp 35–43

  • Savoldi B, Gaido M, Bentivogli L, Negri M, Turchi M (2021) Gender bias in machine translation. Trans Assoc Comput Linguist 9:845–874

    Google Scholar 

  • Schiebinger L (2014) Scientific research must take gender into account. Nature 507(7490):9–9

    Google Scholar 

  • Schuster M, Paliwal KK (1997) Bidirectional recurrent neural networks. IEEE Trans Signal Process 45(11):2673–2681

    Google Scholar 

  • Schwenk H, Chaudhary V, Sun S, Gong H, Guzmán F (2019) Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia. arXiv preprint arXiv:1907.05791

  • Sennrich R, Haddow B, Birch A (2015) Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709

  • Sennrich R, Haddow B, Birch A (2015) Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909

  • Shannon CE, Weaver W (1949) The mathematical theory of information. University of Illinois Press

  • Sharma S, Sharma S, Athaiya A (2017) Activation functions in neural networks towards data science 6(12):310–316

    Google Scholar 

  • Shazeer N, Mirhoseini A, Maziarz K, Davis A, Le Q, Hinton G, Dean J (2017) Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538

  • Shieber SM, Schabes Y (1990) Synchronous tree-adjoining grammars. In: Proceedings of the 13th conference on Computational linguistics-Volume 3, pp 253–258

  • Shieber SM, Schabes Y (1991) Generation and synchronous tree-adjoining grammars. Comput Intell 7(4):220–228

    Google Scholar 

  • Singh SP, Kumar A, Darbari H, Singh L, Rastogi A, Jain S (2017) Machine translation using deep learning: An overview. In: 2017 international conference on computer, communications and electronics (comptelix), IEEE, pp 162–167

  • Snover M, Dorr B, Schwartz R, Micciulla L, Makhoul J (2006) A study of translation edit rate with targeted human annotation. In: Proceedings of association for machine translation in the Americas, vol 200

  • Somers H (2005) Round-trip translation: What is it good for? Proc Austral Lang Technol Workshop 2005:127–133

    Google Scholar 

  • Stanovsky G, Smith NA, Zettlemoyer L (2019) Evaluating gender bias in machine translation. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Florence, Italy, pp 1679–1684, https://doi.org/10.18653/v1/P19-1164, https://aclanthology.org/P19-1164

  • Stasimioti M, Sosoni V, Kermanidis KL, Mouratidis D (2020) Machine Translation Quality: a comparative evaluation of SMT, NMT and tailored-NMT outputs. In: Proceedings of the 22nd annual conference of the European Association for Machine Translation, pp 441–450

  • Staudemeyer RC, Morris ER (2019) Understanding LSTM–a tutorial into long short-term memory recurrent neural networks. arXiv preprint arXiv:1909.09586

  • Steinberger R, Pouliquen B, Widiger A, Ignat C, Erjavec T, Tufis D, Varga D (2006) The JRC-Acquis: A multilingual aligned parallel corpus with 20+ languages. arXiv preprint arXiv:cs/0609058

  • Strubell E, Ganesh A, McCallum A (2019) Energy and Policy Considerations for Deep Learning in NLP. In: Proceedings of the 57th annual meeting of the Association for Computational Linguistics, pp 3645–3650

  • Sun T, Gaut A, Tang S, Huang Y, ElSherief M, Zhao J, Mirza D, Belding E, Chang KW, Wang WY (2019) Mitigating gender bias in natural language processing: literature review. In: Proceedings of the 57th annual meeting of the Association for Computational Linguistics, pp 1630–1640

  • Sun T, Shah A, Webster K, Johnson M (eds) (2021) They, them, theirs: rewriting with gender-neutral English, arXiv:2102.06788

  • Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Advances in neural information processing systems, pp 3104–3112

  • Tixier AJP (2018) Notes on deep learning for nlp. arXiv preprint arXiv:1808.09772

  • Toma P (1977) Systran as a multilingual machine translation system. In: Proceedings of the Third European Congress on Information Systems and Networks, overcoming the language barrier, pp 569–581

  • Tomalin M, Byrne B, Concannon S, Saunders D, Ullmann S (2021) The practical ethics of bias reduction in machine translation: why domain adaptation is better than data debiasing. Ethics Inf Technol 23(3):419–433

    Google Scholar 

  • Ullmann S (2022) Gender bias in machine translation systems. In: Artificial Intelligence and Its Discontents, Springer, pp 123–144

  • United Nations Parallel Corpus (2022) https://conferences.unite.un.org/uncorpus. Accessed 30 Sep 2022

  • van Wynsberghe A (2021) Sustainable AI: AI for sustainability and the sustainability of AI. AI Ethics (2021). https://doi.org/10.1007/s43681-021-00043-6

  • Vandeghinste V, Martens S, Kotzé G, Tiedemann J, Van den Bogaert J, De Smet K, Van Eynde F, Van Noord G (2013) Parse and corpus-based machine translation. Essential Speech and Language Technology for Dutch. Springer, Berlin, pp 305–319

    Google Scholar 

  • Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst 30:1

  • Vieira LN, O’Hagan M, O’Sullivan C (2021) Understanding the societal impacts of machine translation: a critical review of the literature on medical and legal use cases. Inf Commun Soc 24(11):1515–1532

    Google Scholar 

  • Wahler ME (2018) A word is worth a thousand words: legal implications of relying on machine translation technology. Stetson L Rev 48:109

    Google Scholar 

  • Wang S (2005) Computers and translation: a translator’s guide. Language 81(2):544–545

    Google Scholar 

  • Wang W, Pan SJ, Dahlmeier D, Xiao X (2017) Coupled multi-layer attentions for co-extraction of aspect and opinion terms. In: Thirty-First AAAI Conference on Artificial Intelligence

  • Weidinger L, Mellor J, Rauh M, Griffin C, Uesato J, Huang PS, Cheng M, Glaese M, Balle B, Kasirzadeh A et al (2021) Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359

  • White JS (1995) Approaches to black box MT evaluation. Citeseer, proceedings of machine translation summit V, vol 10

  • Wiki (2019) Automatic Language Processing Advisory Committee. https://en.wikipedia.org/wiki/ALPAC. Accessed 08 Apr 2022

  • Wiki (2019) Long-short term memory. https://en.wikipedia.org/wiki/Long_short-term_memory. Accessed 08 Apr 2022

  • Wikimatrix (2022) Mining 135m parallel sentences in 1620 language pairs from wikipedia. https://github.com/facebookresearch/LASER/tree/main/tasks/WikiMatrix. Accessed 30 Sep 2022

  • Wong YW, Mooney R (2007) Learning synchronous grammars for semantic parsing with lambda calculus. In: Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp 960–967

  • Wu D (1997) Stochastic inversion transduction grammars and bilingual parsing of parallel corpora. Comput Linguist 23(3):377–403

    Google Scholar 

  • Wu H, Wang H (2007) Pivot language approach for phrase-based statistical machine translation. Mach Trans 21(3):165–181

    Google Scholar 

  • Wu Y, Schuster M, Chen Z, Le QV, Norouzi M, Macherey W, Krikun M, Cao Y, Gao Q, Macherey K et al (2016) Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144

  • Xu K, Ba J, Kiros R, Cho K, Courville A, Salakhudinov R, Zemel R, Bengio Y (2015) Show, attend and tell: Neural image caption generation with visual attention. In: International conference on machine learning, pp 2048–2057

  • Yamada M (2019) The impact of Google Neural Machine Translation on Post-editing by student translators. J Special Trans 31:87–106

    Google Scholar 

  • Yamada K, Knight K (2001) A syntax-based statistical translation model. In: Proceedings of the 39th annual meeting of the Association for Computational Linguistics, pp 523–530

  • Yang Z, Yang D, Dyer C, He X, Smola A, Hovy E (2016) Hierarchical attention networks for document classification. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp 1480–1489

  • Yang B, Wang L, Wong DF, Chao LS, Tu Z (2019) Convolutional Self-Attention Networks. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp 4040–4045

  • Yun Huang QL Yang Liu (2008) Introduction on machine translation evaluation

  • Zakir HM, Nagoor MS (2017) A brief study of challenges in machine translation. Int J Comput Sci Issues 14(2):54

    Google Scholar 

  • Zhang D, Li M, Li CH, Zhou M (2007) Phrase reordering model integrating syntactic knowledge for SMT. In: Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pp 533–540

  • Zhao Y, Xiang L, Zhu J, Zhang J, Zhou Y, Zong C (2020) Knowledge graph enhanced neural machine translation via multi-task learning on sub-entity granularity. In: Proceedings of the 28th International Conference on Computational Linguistics, pp 4495–4505

  • Zhao Y, Zhang J, Zhou Y, Zong C (2020) Knowledge graphs enhanced neural machine translation. In: IJCAI, pp 4039–4045

  • Zhou J, Cao Y, Wang X, Li P, Xu W (2016) Deep recurrent models with fast-forward connections for neural machine translation. Trans Assoc Comput Linguist 4:371–383

    Google Scholar 

  • Zhou W, Ge T, Mu C, Xu K, Wei F, Zhou M (2019) Improving grammatical error correction with machine translation pairs. arXiv preprint arXiv:1911.02825

  • Ziemski M, Junczys-Dowmunt M, Pouliquen B (2016) The united nations parallel corpus v1. 0. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pp 3530–3534

  • Zollmann A, Venugopal A, Och FJ, Ponte J (2008) A Systematic Comparison of Phrase-Based, Hierarchical and Syntax-Augmented Statistical MT. In: Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pp 1145–1152

Download references

Acknowledgements

The authors would like to thank the anonymous reviewers for their quality reviews and suggestions. This work was supported in part by The Science and Technology Development Fund of Macao, Macao SAR, China under Grant 0033/2022/ITP and in part by The Faculty Research Grant Projects of Macau University of Science and Technology, Macao SAR, China under Grant FRG-22-020-FI.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Macau University of Science and Technology, Taipa, 999078, Macao, China

    Subrota Kumar Mondal, Haoxi Zhang & Kan Ni

  2. Deakin University, Geelong, Australia

    H. M. Dipu Kabir

  3. The Department of Computer Science, Hong Kong Baptist University, Hong Kong, China

    Hong-Ning Dai

Authors
  1. Subrota Kumar Mondal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haoxi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. H. M. Dipu Kabir
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kan Ni
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hong-Ning Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Subrota Kumar Mondal.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mondal, S.K., Zhang, H., Kabir, H.M.D. et al. Machine translation and its evaluation: a study. Artif Intell Rev 56, 10137–10226 (2023). https://doi.org/10.1007/s10462-023-10423-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-023-10423-5

Keywords

Search

Navigation