Skip to main content
SpringerLink
Log in
Book cover

China Conference on Machine Translation

CCMT 2020: Machine Translation pp 24–36Cite as

Unsupervised Machine Translation Quality Estimation in Black-Box Setting

  • Conference paper
  • First Online:

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1328))

Abstract

Machine translation quality estimation (Quality Estimation, QE) aims to evaluate the quality of machine translation automatically without golden reference. QE is an important component in making machine translation useful in real-world applications. Existing approaches require large amounts of expert annotated data. Recently, there are some trials to perform QE in an unsupervised manner, but these methods are based on glass-box features which demands probation inside the machine translation system. In this paper, we propose a new paradigm to perform unsupervised QE in black-box setting, without relying on human-annotated data or model-related features. We create pseudo-data based on Machine Translation Evaluation (MTE) metrics from existing machine translation parallel dataset, and the data are used to fine-tune multilingual pre-trained language models to fit human evaluation. Experiment results show that our model surpasses the previous unsupervised methods by a large margin, and achieve state-of-the-art results on MLQE Dataset.

H. Huang---Work was done when Hui Huang was an intern at Research and Develop Center, Toshiba (China) Co., Ltd., China.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    https://github.com/facebookresearch/mlqe.

  2. 2.

    http://www.statmt.org/wmt20/translation-task.html.

References

  1. John, B., et al.: Confidence estimation for machine translation. In: Proceedings of the International Conference on Computational Linguistics, p. 315 (2004)

    Google Scholar 

  2. Matthew, S., Bonnie, D., Richard, S., Linnea, M., John, M.: A study of translation edit rate with targeted human annotation. In: Proceedings of association for machine translation in the Americas, vol. 200, No. 6 (2006)

    Google Scholar 

  3. Yvette, G., Timothy, B., Alistair, M., Justin, Z.: Can machine translation systems be evaluated by the crowd alone. Nat. Lang. Eng. 23, 1–28 (2015)

    Google Scholar 

  4. Hyun, K., Jong-Hyeok, L., Seung-Hoon, N.: Predictor-estimator using multilevel task learning with stack propagation for neural quality estimation. In: Proceedings of the Second Conference on Machine Translation, vol. 2, Shared Tasks Papers, pp. 562–568 (2017)

    Google Scholar 

  5. Erick, F., Lisa, Y., André, M., Mark, F., Christian, F.: Findings of the WMT 2019 shared tasks on quality estimation. In: Proceedings of the Fourth Conference on Machine Translation (Shared Task Papers, Day 2), vol. 3, pp. 1–10 (2019)

    Google Scholar 

  6. Fomicheva, M., et al.: Unsupervised Quality Estimation for Neural Machine Translation. arXiv preprint arXiv:2005.10608 (2020)

  7. Kishore, P., Salim, R., Todd, W., Wei-Jing, Z.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the Association for Computa-tional Linguistics, pp. 311–318 (2002)

    Google Scholar 

  8. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., Artzi, Y.: Bertscore: evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)

  9. Sellam, T., Das, D., Parikh, A.P.: BLEURT: Learning Robust Metrics for Text Generation. arXiv preprint arXiv:2004.04696 (2020)

  10. Lucia, S.: Exploiting objective annotations for measuring translation post-editing effort. In: Proceedings of the 15th Conference of the European Association for Machine Translation, pp. 73–80 (2011)

    Google Scholar 

  11. Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Logacheva, V.: Findings of the 2017 conference on machine translation. In: Proceedings of the Second Conference on Machine Translation, pp. 169–214 (2017)

    Google Scholar 

  12. Kim, H., Jung, H.-Y., Kwon, H., Lee, J.H., Na, S.-H.: Predictor-estimator: neural quality estimation based on target word prediction for machine translation. ACM Trans. Asian and Low-Resour. Lang. Inf. Proc. (TALLIP) 17(1), 3 (2017)

    Google Scholar 

  13. Kai, F., Bo, L., Fengming, Z., Jiayi W.: “Bilingual Expert” Can Find Translation Errors. arXiv preprint arXiv:1807.09433 (2018)

  14. Kepler, F., et al.: Unbabel’s Participation in the WMT19 Translation Quality Estimation Shared Task. arXiv preprint arXiv:1907.10352 (2019)

  15. Frédéric, B., Nikolaos, A., Lucia, S.: Quality in, quality out: Learning from actual mistakes. In: Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (2020)

    Google Scholar 

  16. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

  17. Lample, G., Conneau, A.: Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291 (2019)

  18. Conneau, A., et al.: Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116 (2019)

  19. Pires, T., Schlinger, E., Garrette, D. How multilingual is Multilingual BERT?. arXiv preprint arXiv:1906.01502 (2019)

  20. Barrault, L., et al.: Findings of the 2019 conference on machine translation (wmt19). In: Proceedings of the Fourth Conference on Machine Translation (Shared Task Papers, Day 1), vol. 2, pp. 1–61 (2019)

    Google Scholar 

  21. Ma, Q., Wei, J., Bojar, O., Graham, Y.: Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In: Proceedings of the Fourth Conference on Machine Translation (Shared Task Papers, Day 1), vol. 2, pp. 62–90 (2019)

    Google Scholar 

  22. Niven, T., Kao, H.Y.: Probing neural network comprehension of natural language arguments. arXiv preprint arXiv:1907.07355 (2019)

  23. Tandon, N., Varde, A.S., de Melo, G.: Commonsense knowledge in machine intelligence. ACM SIGMOD Rec. 46(4), 49–52 (2018)

    Article  Google Scholar 

  24. Zhang, J., Liu, Y., Luan, H., Xu, J., Sun, M.: Prior knowledge integration for neural machine translation using posterior regularization. arXiv preprint arXiv:1811.01100 (2018)

Download references

Acknowledgement

This work is supported by the National Natural Science Foundation of China (Contract 61976015, 61976016, 61876198 and 61370130), and the Beijing Municipal Natural Science Foundation (Contract 4172047), and the International Science and Technology Cooperation Program of the Ministry of Science and Technology (K11F100010), and Toshiba (China) Co., Ltd.

Author information

Authors and Affiliations

  1. Beijing Jiaotong University, Beijing, China

    Hui Huang, Jin’an Xu & Yufeng Chen

  2. Toshiba (China) Co., Ltd, Beijing, China

    Hui Di & Kazushige Ouchi

Authors
  1. Hui Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hui Di
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jin’an Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kazushige Ouchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yufeng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jin’an Xu .

Editor information

Editors and Affiliations

  1. Soochow University, Suzhou, China

    Junhui Li

  2. Dublin City University, Dublin, Ireland

    Andy Way

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Huang, H., Di, H., Xu, J., Ouchi, K., Chen, Y. (2020). Unsupervised Machine Translation Quality Estimation in Black-Box Setting. In: Li, J., Way, A. (eds) Machine Translation. CCMT 2020. Communications in Computer and Information Science, vol 1328. Springer, Singapore. https://doi.org/10.1007/978-981-33-6162-1_3

Download citation

  • DOI: https://doi.org/10.1007/978-981-33-6162-1_3

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-33-6161-4

  • Online ISBN: 978-981-33-6162-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation