Abstract
In this paper, we address the topic of how to estimate phrase-based models from very large corpora and apply them in statistical machine translation. The great number of sentence pairs contained in recent corpora like the well-known Europarl corpus have enormously increased the memory requirements to train phrase-based models and to apply them within a decoding process. We propose a general framework that deals with this problem without introducing significant time overhead by means of the combination of different scaling techniques. This new framework is based on the use of counts instead of probabilities, and on the concept of cache memory.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Brown, P.F., Della Pietra, S.A., Della Pietra, V.J., Mercer, R.L.: The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics 19(2), 263–311 (1993)
Tomás, J., Casacuberta, F.: Monotone statistical translation using word groups. In: Procs. of the MT Summit VIII, Santiago de Compostela, Spain, pp. 357–361 (2001)
Marcu, D., Wong, W.: A phrase-based, joint probability model for statistical machine translation. In: Proc. of the EMNLP, Philadelphia, USA, pp. 1408–1414 (2002)
Koehn, P., Och, F.J., Marcu, D.: Statistical phrase-based translation. In: Proceedings of the HLT/NAACL, Edmonton, Canada (2003)
Callison-Burch, C., Bannard, C., Schroeder, J.: Scaling phrase-based statistical machine translation to larger corpora and longer sentences. In: Proc. of the ACL, Ann Arbor, pp. 255–262 (2005)
Zhang, Y., Vogel, S.: An efficient phrase-to-phrase alignment model for arbitrarily long phrase and large corpora. In: Proceedings of the Tenth EAMT, Budapest, Hungary, The European Association for Machine Translation (2005)
Koehn, P.: Pharaoh: a beam search decoder for phrase-based statistical machine translation models. User manual and description. Technical report, USC Information Science Institute (2003)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer Berlin Heidelberg
About this paper
Cite this paper
Ortiz, D., García Varea, I., Casacuberta, F. (2007). A General Framework to Deal with the Scaling Problem in Phrase-Based Statistical Machine Translation. In: Martí, J., Benedí, J.M., Mendonça, A.M., Serrat, J. (eds) Pattern Recognition and Image Analysis. IbPRIA 2007. Lecture Notes in Computer Science, vol 4478. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-72849-8_40
Download citation
DOI: https://doi.org/10.1007/978-3-540-72849-8_40
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-72848-1
Online ISBN: 978-3-540-72849-8
eBook Packages: Computer ScienceComputer Science (R0)