Abstract
In this article we report the initial results of experiments using HMEANT metric (semi-automatic evaluation metric used for scoring translation quality by matching semantic role fillers) on the Polish language. The metric is evaluated in the task of Machine Translation (MT) and in re-speaking quality assessment. GUI-based annotation interface was developed and with this tool (https://github.com/krzwolk/HMEANT-metric-for-Polish) evaluation was conducted practically by not IT-related personnel. Reliability, correlation with automatic metrics, language independence and time costs were analysed as well. Role labelling and alignment using GUI interface were done by two annotators with no related background (they were only instructed for about 10 min). The results of our experiments showed high inter-annotator agreement as far as role labelling was concerned and a good correlation of the HMEANT metric with human judgements based on re-speaking evaluation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Papineni, K., et al.: BLEU: a method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311–318. Association for Computational Linguistics (2002)
Doddington, G.: Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In: Proceedings of the Second International Conference on Human Language Technology Research, pp. 138–145. Morgan Kaufmann Publishers Inc. (2002)
Koehn, P., et al.: Moses: Open source toolkit for statistical machine translation. In: Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, pp. 177–180. Association for Computational Linguistics (2007)
Banerjee, S., Lavie, A.: METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65–72 (2005)
Lo, C., Wu, D.: MEANT: an inexpensive, high-accuracy, semi-automatic metric for evaluating translation utility via semantic frames. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 220–229. Association for Computational Linguistics (2011)
Snover, M., et al.: A study of translation edit rate with targeted human annotation. In: Proceedings of Association for Machine Translation in the Americas, pp. 223–231 (2006)
Birch, A., et al.: The feasibility of HMEANT as a human MT evaluation metric. In: Proceedings of the Eighth Workshop on Statistical Machine Translation, pp. 52–61 (2013)
Bojar, O., Wu, D.: Towards a predicate-argument evaluation for MT. In: Proceedings of the Sixth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 30–38. Association for Computational Linguistics (2012)
Pradhan, S., et al.: Shallow Semantic Parsing using Support Vector Machines. HLT-NAACL, pp. 233–240 (2004)
Wołk, K., Korzinek, D.: Comparison and Adaptation of Automatic Evaluation Metrics for Quality Assessment of Re-Speaking. arXiv:1601.02789 (2016)
Acknowledgments
This research was supported by Polish-Japanese Academy of Information Technology statutory resources (ST/MUL/2016) and resources for young researchers.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing Switzerland
About this paper
Cite this paper
Wołk, K., Korzinek, D., Marasek, K. (2017). Semi-automatic and Human-Aided Translation Evaluation Metric (HMEANT) for Polish Language in Re-speaking and MT Assessment. In: Zgrzywa, A., Choroś, K., Siemiński, A. (eds) Multimedia and Network Information Systems. Advances in Intelligent Systems and Computing, vol 506. Springer, Cham. https://doi.org/10.1007/978-3-319-43982-2_21
Download citation
DOI: https://doi.org/10.1007/978-3-319-43982-2_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-43981-5
Online ISBN: 978-3-319-43982-2
eBook Packages: EngineeringEngineering (R0)