Abstract
In this study we aim at the automatic evaluation of machine translation through the residual analysis at the sentence level. We created a dataset, which covered one translation direction- a translation from an inflective language (Slovak) into an analytical language (English). BLEU (Bilingual Evaluation Understudy) as a state-of-the-art automatic metric for machine translation evaluation was used. The main contribution consists of rigorous technique (statistical method), novel to research of MT evaluation given by the residual analysis to identify differences between MT output and post-edited machine translation output.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
White, J.S.: How to evaluate machine translation. In: Somers, H. (ed.) Computers and Translation: a Translator’s Guide, pp. 211–244. J. Benjamins, Amsterdam (2003)
Munková, D., Munk, M.: An automatic evaluation of machine translation and Slavic languages. In: Proceedings of the 8th International Conference on Application of Information and Communication Technologies (AICT-2014), Astana, pp. 447–451 (2014)
Papineni, K., Roukos, S., Ward, T., Zhu, W.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual meeting of the Association for Computational Linguistics (ACL-02), Philadelphia, pp. 311–318 (2002)
Vilar, D., Xu, J., D’Haro, L.F., Ney, H.: Error analysis of statistical machine translation output. In: Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC-2006), Genoa, pp. 697–702 (2006)
Banerjee, S., Lavie, A.: METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for MT and/or Summarization (ACL-05), Michigan, pp. 65–72 (2005)
Doddington, G.: Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In: Proceedings of the 2nd International Conference on Human Language Technology Research (HLT-2002), pp. 138–145 (2002)
Koehn, P.: Statistical Machine Translation. Cambridge University Press, Cambridge (2010)
Snover, M., Dorr, B., Schwartz, R., Micciulla, L., Makhoul, J.: A study of translation edit rate with targeted human annotation. In: Proceedings of Association for Machine Translation in the Americas (ASMTA-06), pp. 223–231 (2006)
Kapusta, J., Munk, M., Drlík, M.: Analysis of differences between expected and observed probability of accesses to web pages. In: Hwang, D., Jung, J.J., Nguyen, N.-T. (eds.) ICCCI 2014. LNCS, vol. 8733, pp. 673–683. Springer, Heidelberg (2014)
Seljan, S., Brkić, M., Kučiš, V.: Evaluation of free online machine translations for Croatian-English and English-Croatian language pairs. In: Proceedings of the 3rd International Conference on the Future of Information Sciences: INFuture 2011-Information Sciences and e-Society, Zagreb, pp. 331–345 (2011)
Dis Brandt, M.: Developing Icelandic to English shallow transfer machine translation system. Ms. thesis. Reykjavik University (2011)
Babych, B., Hartley, A., Sharoff, S.: Translating from under-resourced languages: comparing direct transfer against pivot translation. In: Proceedings of the MT Summit XI, Copenhagen (2007)
Adly, N., Al Ansary, S.: Evaluation of Arabic machine translation system based on the universal networking language. In: Horacek, H., Métais, E., Muñoz, R., Wolska, M. (eds.) NLDB 2009. LNCS, vol. 5723, pp. 243–257. Springer, Heidelberg (2010)
Munk, M., Drlík, M.: Analysis of stakeholders’ behaviour depending on time in virtual learning environment. Appl. Math. Inf. Sci. 8(2), 773–785 (2014)
Munk, M., Drlik, M., Vrábelová, M.: Probability modelling of accesses to the course activities in the web-based educational system. In: Murgante, B., Gervasi, O., Iglesias, A., Taniar, D., Apduhan, B.O. (eds.) ICCSA 2011, Part V. LNCS, vol. 6786, pp. 485–499. Springer, Heidelberg (2011)
Pilkova, A., Volna, J., Papula, J., Holienka, M.: The influence of intellectual capital on firm performance among slovak SMEs. In: Proceedings of the 10th International Conference on Intellectual Capital, Knowledge Management and Organisational Learning (ICICKM-2013), Washington, DC, pp. 329–338 (2013)
Lin, Ch.Y., Och, F.J.: Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In: Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL-2004), Barcelona, pp. 605–612 (2004)
Acknowledgments
This work was supported by the Slovak Research and Development Agency under the contract No. APVV-0451-10 and Scientific Grant Agency of the Ministry of Education of the Slovak Republic (ME SR) and of Slovak Academy of Sciences (SAS) under the contract No. VEGA-1/0559/14.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Munková, D., Munk, M. (2015). Automatic Evaluation of Machine Translation Through the Residual Analysis. In: Huang, DS., Han, K. (eds) Advanced Intelligent Computing Theories and Applications. ICIC 2015. Lecture Notes in Computer Science(), vol 9227. Springer, Cham. https://doi.org/10.1007/978-3-319-22053-6_51
Download citation
DOI: https://doi.org/10.1007/978-3-319-22053-6_51
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-22052-9
Online ISBN: 978-3-319-22053-6
eBook Packages: Computer ScienceComputer Science (R0)