Skip to main content
SpringerLink
Log in

Packing It All Up in Search for a Language Independent MT Quality Measure Tool – Part Two

  • Conference paper
Human Language Technology. Challenges for Computer Science and Linguistics (LTC 2009)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 6562))

Included in the following conference series:

  • 1065 Accesses

Abstract

This study describes first usage of a particular implementation of Normalized Compression Distance (NCD) as a machine translation quality evaluation tool. NCD has been introduced and tested for clustering and classification of different types of data and found a reliable and general tool. As far as we know NCD in its Complearn implementation has not been evaluated as a MT quality tool yet, and we wish to show that it can also be used for this purpose. We show that NCD scores given for MT outputs in different languages correlate highly with scores of a state-of-the-art MT evaluation metrics, METEOR 0.6. Our experiments are based on translations between one source and three target languages with a smallish sample that has available reference translations, UN’s Universal Declaration of Human Rights. Secondly we shall also briefly describe and discuss results of a larger scale evaluation of NCD as an MT metric with WMT08 Shared Task Evaluation Data. These evaluations confirm further that NCD is a noteworthy MT metric both in itself and also enriched with basic language tools, stemming and Wordnet.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: BLEU: a Method for Automatic Evaluation of Machine Translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL 2002), pp. 311–318 (2002)

    Google Scholar 

  2. Doddington, G.: Automatic Evaluation of Machine Translation Quality Using N-Gram Co-Occurrence Statistics. In: Proceedings of the Second International Conference on Human Language Technology Research, pp. 138-145 (2002)

    Google Scholar 

  3. Lavie, A., Agarwal, A.: METEOR: An automatic Metric for MT Evaluation with High Levels of Correlation with Human judgements. In: Proceedings of the Second Workshop on Statistical Machine Translation, Prague, pp. 228–231 (June 2007)

    Google Scholar 

  4. Giménez, J., Amigó, E.: IQMT: A Framework for Automatic Machine Translation Evaluation. In: Proceedings of the 5th LREC (2006)

    Google Scholar 

  5. Callison-Burch, C., Osborne, M., Koehn, P.: Re-evaluating the Role of BLEU in Machine Translation Research. In: EACL 2006, pp. 249–256 (2006)

    Google Scholar 

  6. Turian, J.P., Shen, L., Melamed, D.: Evaluation of Machine Translation and its Evaluation. In: MT Summit IX, New Orleans, USA, September 23-27, pp. 386–393 (2003)

    Google Scholar 

  7. Cilibrasi, R., Vitányi, P.M.B.: Clustering by Compression. IEEE Transactions on Information Theory 51, 1523–1545 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  8. Cilibrasi, R., Vitányi, P.M.B.: The Google Similarity Distance. IEEE Transactions on Knowledge and Data Engineering 19, 370–383 (2007)

    Article  Google Scholar 

  9. Li, M., Chen, X., Li, X., Ma, B., Vitányi, P.: The Similarity Metric. IEEE Transactions on Information Theory 50, 3250–3264 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  10. Callison-Burch, C., Fordyce, C., Koehn, P., Monz, C., Schroeder, J.: Further Meta-Evaluation of Machine Translation. In: Proceedings of the Third Workshop on Statistical Machine Translation, pp. 70–106 (2008)

    Google Scholar 

  11. Väyrynen, J., Tapiovaara, T., Kettunen, K., Dobrinkat, M.: Normalized Compression Distance as an Automatic MT Evaluation Metric. In: Twenty-Five Years on Machine Translation, Cranfield, U.K (2009)

    Google Scholar 

  12. Dobrinkat, M., Tapiovaara, T., Väyrynen, J., Kettunen, K.: Normalized Compression Distance Based Measures for MetricsMATR 2010. In: ACL 2010: Joint Fifth Workshop on Statistical Machine Translation and Metrics MATR. Proceedings of the Workshop, Uppsala University, Uppsala, Sweden, July 15-16, pp. 343–348 (2010)

    Google Scholar 

  13. Dobrinkat, M., Tapiovaara, T., Väyrynen, J., Kettunen, K.: Evaluating Machine translations Using mNCD. In: ACL 2010: the 48th Annual Meeting of the Association for Computational Linguistics: Proceedings of the Conference Short Papers, Uppsala, Sweden, July 11-16, pp. 80–85 (2010)

    Google Scholar 

  14. Lavie, A., Agarwal, A.: The METEOR Automatic Machine Translation Evaluation System, http://www.cs.cmu.edu/~alavie/METEOR/

  15. Banerjee, S., Lavie, A.: METEOR: Automatic Metric for MT Evaluation with Improved Correlation with Human Judgements. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Ann Arbor, pp. 65–72 (2005)

    Google Scholar 

  16. Li, M., Vitányi, P.M.B.: An Introduction to Kolmogorov Complexity and its Applications, 2nd edn. Springer, Heidelberg (1997)

    Book  MATH  Google Scholar 

  17. What is NCD?, http://www.complearn.org/ncd.html

  18. Parker, S.: BADGER: a New Machine Translation Metric (2008), http://www.amtaweb.org/papers/badger_babblequest_matr08.pdf

  19. Lavie, A., Sagae, K., Jayaraman, S.: The significance of recall in automatic metrics for MT evaluation. In: Frederking, R.E., Taylor, K.B. (eds.) AMTA 2004. LNCS (LNAI), vol. 3265, pp. 134–143. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  20. Kettunen, K.: Choosing the best MT programs for CLIR purposes – can MT metrics be helpful? In: Boughanem, M., Berrut, C., Mothe, J., Soule-Dupuy, C. (eds.) ECIR 2009. LNCS, vol. 5478, pp. 706–712. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  21. Kettunen, K.: Facing The Machine Translation Babel in CLIR - Can MT Metrics Help in Choosing CLIR Resources? In: Klopotek, M.A., Przepiórkowski, A., Wierzchon, S.T., Trojanowski, K. (eds.) Recent Advances in Intelligent Information Systems, IIS 2009, pp. 103–116 (2009)

    Google Scholar 

  22. Kishida, K.: Prediction of Performance of Cross-language Information Retrieval System Using Automatic Evaluation of Translation. Library & Information Science Research 30, 138–144 (2008)

    Article  Google Scholar 

  23. Zhu, J., Wang, H.: The Effect of Translation Quality in MT-Based Cross-Language Information Retrieval. In: Proceedings of the 21st International Conference on Computational Linguistics and 44th annual Meeting of the ACL, pp. 593–600 (2006)

    Google Scholar 

  24. Cebrián, M., Alfonseca, M., Ortega, M.: Common Pitfalls Using The Normalized Compression Distance: What To Watch Out For In A Compressor. Communications in Information and Systems 5, 367–384 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  25. Callison-Burch, C., Koehn, P., Monz, C., Peterson, K., Przybocki, M., Zaidan, O.F.: Findings of the 2010 Joint Workshop on Statistical Machine Translation and Metrics for Machine Translation. In: Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, Uppsala, Sweden, July 15-16, pp. 17–53 (2010), Updated version: http://statmt.org/wmt10/pdf/wmt10-overview.pdf

  26. Lopez, A.: Statistical Machine Translation. ACM Computing Surveys 40, 8–49 (2008)

    Article  Google Scholar 

  27. Vitányi, P.M.B., Balbach, F.J., Cilibrasi, R., Ming, L.: Normalized Information Distance. In: Emmert-Streib, F., Dehmer, M. (eds.) Information Theory and Statistical Learning, pp. 39–71. Springer, Heidelberg (2009)

    Google Scholar 

  28. Culy, C., Riehemann, S.Z.: The Limits of N-gram Translation Metrics. In: MT Summit IX, New Orleans, USA, September 23-27, pp. 71–78 (2003)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Kymenlaakso University of Applied Sciences, Paraatikenttä 7, FIN-045 100, Kouvola, Finland

    Kimmo Kettunen

Authors
  1. Kimmo Kettunen
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Faculty of Mathematics and Computer Science, Adam Mickiewicz University in Poznan, ul. Umultowska 87, 61614, Poznan, Poland

    Zygmunt Vetulani

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Kettunen, K. (2011). Packing It All Up in Search for a Language Independent MT Quality Measure Tool – Part Two. In: Vetulani, Z. (eds) Human Language Technology. Challenges for Computer Science and Linguistics. LTC 2009. Lecture Notes in Computer Science(), vol 6562. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-20095-3_52

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-20095-3_52

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-20094-6

  • Online ISBN: 978-3-642-20095-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation