Hyun Kim
Abstract
Recently, quality estimation has been attracting increasing interest from machine translation researchers, aiming at finding a good estimator for the “quality” of machine translation output. The common approach for quality estimation is to treat the problem as a supervised regression/classification task using a quality-annotated noisy parallel corpus, called quality estimation data, as training data. However, the available size of quality estimation data remains small, due to the too-expensive cost of creating such data. In addition, most conventional quality estimation approaches rely on manually designed features to model nonlinear relationships between feature vectors and corresponding quality labels. To overcome these problems, this article proposes a novel neural network architecture for quality estimation task—called the predictor-estimator—that considers word prediction as an additional pre-task. The major component of the proposed neural architecture is a word prediction model based on a modified neural machine translation model—a probabilistic model for predicting a target word conditioned on all the other source and target contexts. The underlying assumption is that the word prediction model is highly related to quality estimation models and is therefore able to transfer useful knowledge to quality estimation tasks. Our proposed quality estimation method sequentially trains the following two types of neural models: (1) Predictor: a neural word prediction model trained from parallel corpora and (2) Estimator: a neural quality estimation model trained from quality estimation data. To transfer word a prediction task to a quality estimation task, we generate quality estimation feature vectors from the word prediction model and feed them into the quality estimation model. The experimental results on WMT15 and 16 quality estimation datasets show that our proposed method has great potential in the various sub-challenges.
- Michael Auli, Michel Galley, Chris Quirk, and Geoffrey Zweig. 2013. Joint language and translation modeling with recurrent neural networks. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 1044--1054. http://www.aclweb.org/anthology/D13-1106Google Scholar
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR 2015.Google Scholar
- John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2004. Confidence estimation for machine translation. In Proceedings of the 20th International Conference on Computational Linguistics (COLING’04). Association for Computational Linguistics, Article 315. Google ScholarDigital Library
- Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the 9th Workshop on Statistical Machine Translation. Association for Computational Linguistics, 12--58.Google Scholar
- Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the 1st Conference on Machine Translation. Association for Computational Linguistics, 131--198.Google Scholar
- Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 workshop on statistical machine translation. In Proceedings of the 10th Workshop on Statistical Machine Translation. Association for Computational Linguistics, 1--46.Google Scholar
- Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder--decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP’14). Association for Computational Linguistics, 1724--1734.Google ScholarCross Ref
- Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res. 12 (Nov. 2011), 2493--2537. 1532-4435 Google ScholarDigital Library
- Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 1370--1380.Google ScholarCross Ref
- Mariano Felice and Lucia Specia. 2012. Linguistic features for quality estimation. In Proceedings of the 7th Workshop on Statistical Machine Translation. Association for Computational Linguistics, 96--103. Google ScholarDigital Library
- Jesús González-Rubio, J. Ramón Navarro-Cerdán, and Francisco Casacuberta. 2013. Dimensionality reduction methods for machine translation quality estimation. Mach. Transl. 27, 3 (2013), 281--301. 1573-0573 Google ScholarDigital Library
- Jesús González-Rubio, Alberto Sanchís, and Francisco Casacuberta. 2012. PRHLT submission to the WMT12 quality estimation task. In Proceedings of the 7th Workshop on Statistical Machine Translation. Association for Computational Linguistics, 104--108. Google ScholarDigital Library
- Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. 2005. Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition. Springer, Berlin, 799--804. Google ScholarDigital Library
- Tianxing He, Yu Zhang, Jasha Droppo, and Kai Yu. 2016. On training bi-directional neural network language model with noise contrastive estimation. Retrieved from http://sigport.org/1255.Google Scholar
- Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. CoRR abs/1508.01991 (2015). http://arxiv.org/abs/1508.01991.Google Scholar
- Hyun Kim and Jong-Hyeok Lee. 2016a. A recurrent neural networks approach for estimating the quality of machine translation output. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 494--498.Google ScholarCross Ref
- Hyun Kim and Jong-Hyeok Lee. 2016b. Recurrent neural network based translation quality estimation. In Proceedings of the 1st Conference on Machine Translation. Association for Computational Linguistics, 787--792.Google ScholarCross Ref
- Hyun Kim and Jong-Hyeok Lee. 2016c. Sentence vector summarization for deep learning based quality estimation of MT. In Proceedings of the Korea Computer Congress (KCC’16). {in Korean}Google Scholar
- Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT Summit 5, 79--86.Google Scholar
- Anna Kozlova, Mariya Shmatova, and Anton Frolov. 2016. YSDA participation in the WMT’16 quality estimation shared task. In Proceedings of the 1st Conference on Machine Translation. Association for Computational Linguistics, 793--799.Google ScholarCross Ref
- Julia Kreutzer, Shigehiko Schamoni, and Stefan Riezler. 2015. QUality estimation from ScraTCH (QUETCH): Deep learning for word-level translation quality estimation. In Proceedings of the 10th Workshop on Statistical Machine Translation. Association for Computational Linguistics, 316--322.Google ScholarCross Ref
- André F. T. Martins, Ramón Astudillo, Chris Hokamp, and Fabio Kepler. 2016. Unbabel’s participation in the WMT16 word-level translation quality estimation shared task. In Proceedings of the 1st Conference on Machine Translation. Association for Computational Linguistics, 806--811.Google ScholarCross Ref
- André F. T. Martins, Junczys-Dowmunt Marcin, Fabio Kepler, and Ramón Astudillo. 2017. Pushing the limits of translation quality estimation. Transactions of the Association for Computational Linguistics 5, 205--218.Google ScholarCross Ref
- Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).Google Scholar
- Jan Niehues, Teresa Herrmann, Stephan Vogel, and Alex Waibel. 2011. Wider context by using bilingual language models in machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation. Association for Computational Linguistics, 198--206. Google ScholarDigital Library
- Raj Nath Patel and Sasikumar M. 2016. Translation quality estimation using recurrent neural network. In Proceedings of the 1st Conference on Machine Translation. Association for Computational Linguistics, 819--824.Google Scholar
- Raphael Rubino, Jose de Souza, Jennifer Foster, and Lucia Specia. 2013. Topic models for translation quality estimation for gisting purposes. In Proceedings of the XIV Machine Translation Summit. 295--302.Google Scholar
- Kashif Shah, Trevor Cohn, and Lucia Specia. 2015. A bayesian non-linear method for feature selection in machine translation quality estimation. Mach. Transl. 29, 2 (2015), 101--125. 1573-0573 Google ScholarDigital Library
- Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the Association for Machine Translation in the Americas. 223--231.Google Scholar
- Radu Soricut and Abdessamad Echihabi. 2010. TrustRank: Inducing trust in automatic translations via ranking. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 612--621. Google ScholarDigital Library
- Lucia Specia, Varvara Logacheva, and Carolina Scarton. 2016. WMT16 Quality Estimation Shared Task Training and Development Data. Retrieved from http://hdl.handle.net/11372/LRT-1646.Google Scholar
- Lucia Specia, Kashif Shah, Jose G. C. de Souza, and Trevor Cohn. 2013. QuEst - A translation quality estimation framework. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguistics, 79--84.Google Scholar
- Lucia Specia, Marco Turchi, Nicola Cancedda, Marc Dymetman, and Nello Cristianini. 2009. Estimating the sentence-level quality of machine translation systems. In Proceedings of the 13th Conference of the European Association for Machine Translation. 28--37.Google Scholar
- Matthew D Zeiler. 2012. ADADELTA: An adaptive learning rate method. arXiv preprint arXiv:1212.5701 (2012).Google Scholar
Index Terms
- Predictor-Estimator: Neural Quality Estimation Based on Target Word Prediction for Machine Translation
Recommendations
A Quality Estimation System for Hungarian
Human Language Technology. Challenges for Computer Science and LinguisticsAbstractQuality estimation is an important field of machine translation evaluation. There are automatic evaluation methods for machine translation that use reference translations created by human translators. The creation of these reference translations ...
Quality Estimation of English-Hindi Machine Translation Systems
ICTCS '16: Proceedings of the Second International Conference on Information and Communication Technology for Competitive StrategiesQuality Estimation is a new research area in natural language processing where machine learning techniques are used to estimate the quality of machine translation outputs. In this paper we have discussed our experience of performing quality estimation ...
Predicting insertion positions in word-level machine translation quality estimation
AbstractWord-level machine translation (MT) quality estimation (QE) is usually formulated as the task of automatically identifying which words need to be edited (either deleted or replaced) in a translation T produced by an MT system. The ...
Highlights- Novel appproach predicting insertions in machine translation quality estimation.
Comments