Skip to main content
SpringerLink
Log in

Otem&Utem: Over- and Under-Translation Evaluation Metric for NMT

  • Conference paper
  • First Online:
Natural Language Processing and Chinese Computing (NLPCC 2018)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11108))

Abstract

Although neural machine translation (NMT) yields promising translation performance, it unfortunately suffers from over- and under-translation issues [31], of which studies have become research hotspots in NMT. At present, these studies mainly apply the dominant automatic evaluation metrics, such as BLEU, to evaluate the overall translation quality with respect to both adequacy and fluency. However, they are unable to accurately measure the ability of NMT systems in dealing with the above-mentioned issues. In this paper, we propose two quantitative metrics, the Otem and Utem, to automatically evaluate the system performance in terms of over- and under-translation respectively. Both metrics are based on the proportion of mismatched n-grams between gold reference and system translation. We evaluate both metrics by comparing their scores with human evaluations, where the values of Pearson Correlation Coefficient reveal their strong correlation. Moreover, in-depth analyses on various translation systems indicate some inconsistency between BLEU and our proposed metrics, highlighting the necessity and significance of our metrics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://nlp.stanford.edu/software/segmenter.html.

  2. 2.

    https://github.com/thumt/THUMT.

References

  1. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: ICLR (2015)

    Google Scholar 

  2. Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: ACL Workshop (2005)

    Google Scholar 

  3. Chen, B., Guo, H.: Representation based translation evaluation metrics. In: ACL (2015)

    Google Scholar 

  4. Chen, B., Kuhn, R.: AMBER: a modified BLEU, enhanced ranking metric. In: WMT (2011)

    Google Scholar 

  5. Chen, B., Kuhn, R., Foster, G.: Improving AMBER, an MT evaluation metric. In: WMT (2012)

    Google Scholar 

  6. Chiang, D.: Hierarchical phrase-based translation. Comput. Linguist. 33(2), 201–228 (2007)

    Article  Google Scholar 

  7. Cohn, T., Hoang, C.D.V., Vymolova, E., Yao, K., Dyer, C., Haffari, G.: Incorporating structural alignment biases into an attentional neural translation model. In: NAACL (2016)

    Google Scholar 

  8. Doddington, G.: Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In: HLT (2002)

    Google Scholar 

  9. Feng, S., Liu, S., Yang, N., Li, M., Zhou, M., Zhu, K.Q.: Improving attention modeling with implicit distortion and fertility for machine translation. In: COLING (2016)

    Google Scholar 

  10. Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin, Y.N.: Convolutional sequence to sequence learning. In: ICML (2017)

    Google Scholar 

  11. Giménez, J., Màrquez, L.: Linguistic features for automatic evaluation of heterogenous MT systems. In: WMT (2007)

    Google Scholar 

  12. Gupta, R., Orasan, C., van Genabith, J.: ReVal: a simple and effective machine translation evaluation metric based on recurrent neural networks. In: EMNLP (2015)

    Google Scholar 

  13. Guzmán, F., Joty, S., Màrquez, L., Nakov, P.: Pairwise neural machine translation evaluation. In: ACL (2015)

    Google Scholar 

  14. Koehn, P., Och, F.J., Marcu, D.: Statistical phrase-based translation. In: NAACL (2003)

    Google Scholar 

  15. Leusch, G., Ueffing, N., Ney, H.: A novel string to string distance measure with applications to machine translation evaluation. In: Mt Summit IX (2003)

    Google Scholar 

  16. Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: ACL Workshop (2004)

    Google Scholar 

  17. Liu, D., Gildea, D.: Syntactic features for evaluation of machine translation. In: ACL Workshop (2005)

    Google Scholar 

  18. Lo, C.k., Tumuluru, A.K., Wu, D.: Fully automatic semantic MT evaluation. In: WMT (2012)

    Google Scholar 

  19. Lo, C.k., Wu, D.: MEANT: an inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility based on semantic roles. In: ACL (2011)

    Google Scholar 

  20. Lo, C.k., Wu, D.: MEANT at WMT 2013: a tunable, accurate yet inexpensive semantic frame based MT evaluation metric. In: WMT (2013)

    Google Scholar 

  21. Malaviya, C., Ferreira, P., Martins, A.F.: Sparse and constrained attention for neural machine translation. arXiv preprint arXiv:1805.08241 (2018)

  22. Mehay, D.N., Brew, C.: BLEUÂTRE: flattening syntactic dependencies for MT evaluation. In: MT Summit (2007)

    Google Scholar 

  23. Nießen, S., Och, F.J., Leusch, G., Ney, H.: An evaluation tool for machine translation: fast evaluation for MT research. In: LREC (2000)

    Google Scholar 

  24. Papineni, K., Roukos, S., Ward, T., Zhu, W.: BLEU: a method for automatic evaluation of machine translation. In: ACL (2002)

    Google Scholar 

  25. Popović, M., Ney, H.: Towards automatic error analysis of machine translation output. Comput. Linguist. 37(4), 657–688 (2011)

    Article  MathSciNet  Google Scholar 

  26. Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: ACL (2016)

    Google Scholar 

  27. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., Makhoul, J.: A study of translation edit rate with targeted human annotation. In: AMTA (2006)

    Google Scholar 

  28. Sundermeyer, M., Alkhouli, T., Wuebker, J., Ney, H.: Translation modeling with bidirectional recurrent neural networks. In: EMNLP (2014)

    Google Scholar 

  29. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: NIPS (2014)

    Google Scholar 

  30. Owczarzak, K., van Genabith, J., Way, A.: Dependency-based automatic evaluation for machine translation. In: SSST (2007)

    Google Scholar 

  31. Tu, Z., Lu, Z., Liu, Y., Liu, X., Li, H.: Modeling coverage for neural machine translation. In: ACL (2016)

    Google Scholar 

  32. Vaswani, A., et al.: Attention is all you need. In: NIPS (2017)

    Google Scholar 

  33. Yang, Z., Hu, Z., Deng, Y., Dyer, C., Smola, A.: Neural machine translation with recurrent attention modeling. In: EACL (2017)

    Google Scholar 

  34. Yu, H., Wu, X., Xie, J., Jiang, W., Liu, Q., Lin, S.: RED: a reference dependency based MT evaluation metric. In: COLING (2014)

    Google Scholar 

  35. Zhang, B., Xiong, D., Su, J.: A GRU-gated attention model for neural machine translation. arXiv preprint arXiv:1704.08430 (2017)

Download references

Acknowledgement

The authors were supported by Natural Science Foundation of China (No. 61672440), the Fundamental Research Funds for the Central Universities (Grant No. ZK1024), Scientific Research Project of National Language Committee of China (Grant No. YB135-49), and Research Fund of the Provincial Key Laboratory for Computer Information Processing Technology in Soochow University (Grant No. KJS1520). We also thank the reviewers for their insightful comments.

Author information

Authors and Affiliations

  1. Xiamen University, Xiamen, China

    Jing Yang, Biao Zhang, Yue Qin, Xiangwen Zhang, Qian Lin & Jinsong Su

  2. Provincial Key Laboratory for Computer Information Processing Technology, Soochow University, Suzhou, China

    Jing Yang & Jinsong Su

Authors
  1. Jing Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Biao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yue Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiangwen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qian Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jinsong Su
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinsong Su .

Editor information

Editors and Affiliations

  1. Soochow University, Suzhou, China

    Min Zhang

  2. The University of Texas at Dallas, Richardson, Texas, USA

    Vincent Ng

  3. Peking University, Beijing, China

    Dongyan Zhao

  4. Peking University, Beijing, China

    Sujian Li

  5. Zhengzhou University, Zhengzhou, China

    Hongying Zan

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yang, J., Zhang, B., Qin, Y., Zhang, X., Lin, Q., Su, J. (2018). Otem&Utem: Over- and Under-Translation Evaluation Metric for NMT. In: Zhang, M., Ng, V., Zhao, D., Li, S., Zan, H. (eds) Natural Language Processing and Chinese Computing. NLPCC 2018. Lecture Notes in Computer Science(), vol 11108. Springer, Cham. https://doi.org/10.1007/978-3-319-99495-6_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-99495-6_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-99494-9

  • Online ISBN: 978-3-319-99495-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation