Abstract
Current techniques in word alignment disregard words with a low frequency because they would not be useful. Against this belief, this paper shows that, in particular, the notion of hapax legomena may contribute to word alignment to a large extent. In an experiment, we show that pairs of corpus hapaxes contribute to the majority of the best word alignments. In addition, we show that the notion of sentence hapax justifies a practical and common simplification of standard alignment methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Schrader, B.: How does morphological complexity translate? A cross-linguistic case study for word alignment, Tübingen, Germany (February 2006)
Cromières, F.: Sub-sentential alignment using substring co-occurrence counts. In: Proceedings of the COLING/ACL 2006 Student Research Workshop, Sydney, Australia, July 2006, pp. 13–18 (2006)
Giguet, E., Luquet, P.S.: Multilingual lexical database generation from parallel texts in 20 european languages with endogenous resources. In: Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, Sydney, Australia, July 2006, pp. 271–278 (2006)
Nishimoto, E.: Defining new words in corpus data: Productivity of English suffixes in the British National Corpus. In: 26th Annual Meeting of the Cognitive Science Society (CogSci 2004), Chicago, USA (August 2004)
Evert, S., Lüdeling, A.: Measuring morphological productivity: Is automatic preprocessing sufficient? In: Proceedings of Corpus Linguistics 2001 (CL2001), Lancaster, UK (April 2001)
Cartoni, B.: Constance et variabilité de l’incomplétude lexicale. In: Proceedings of TALN/RECITAL 2006, Leuven, Belgium, April 2006, pp. 661–669 (2006)
Langlais, P., Gotti, F., Cao, G.: English-Inuktitut word alignment system description, Ann Arbor, Michigan, USA, pp. 75–78 (2005)
Baayen, H., Sproat, R.: Estimating lexical priors for low-frequency morphologically ambiguous forms. Computational Linguistics 22, 155–166 (1996)
Brown, P., Pietra, S.D., Pietra, V.D., Mercer, R.: The mathematic of statistical machine translation: Parameter estimation. Computational Linguistics 19, 263–311 (1993)
Marcu, D., Wong, W.: A phrase-based, joint probability model for statistical machine translation. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), Philadelphia, Pennsylvania, USA (July 2002)
Och, F., Ney, H.: A systematic comparison of various statistical alignment models. Computational Linguistics 29, 19–51 (2003)
Matusov, E., Zens, R., Ney, H.: Symmetric word alignments for statistical machine translation. In: Proceedings of the 20th International Conference on Computational Linguistics (Coling 2004), Geneva, Switzerland, August 2004, pp. 219–225 (2004)
Zhang, Y., Vogel, S.: An efficient phrase-to-phrase alignment model for arbitrarily long phrase and large corpora. In: Proceedings of the Tenth Conference of the European Association for Machine Translation (EAMT-2005), Budapest, Hungary (May 2005)
Moore, R., tau Yih, W., Bode, A.: Improved discriminative bilingual word alignment. In: Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Sydney, Australia, July 2006, pp. 513–520 (2006)
Melamed, I.D.: Automatic evaluation and uniform filter cascades for inducing n-best translation lexicons. In: Proceedings of the Third Workshop on Very Large Corpora, Boston, Massachusetts, USA, June 1995, pp. 184–198 (1995)
Lardilleux, A., Lepage, Y.: A truly multilingual, high coverage, accurate, yet simple, sub-sentential alignment method. In: Proceedings of the 8th Conference of the Association for Machine Translation in the Americas (AMTA 2008), Waikiki, Hawai’i, USA, October 2008, pp. 125–132 (2008)
Shinyama, Y., Sekine, S.: Named entity discovery using comparable news articles. In: Proceedings of the 20th International Conference on Computational Linguistics (Coling 2004), Geneva, Switzerland, August 2004, pp. 848–853 (2004)
Lafourcade, M., Boitet, C.: UNL lexical selection with conceptual vectors. In: Proceedings of the Third International Conference on Language Resources and Evaluation (LREC), Las Palmas de Gran Canaria, Spain, 1958–1964 (2002)
Turney, P., Littman, M.: Corpus-based learning of analogies and semantic relations. Machine Learning 60, 251–278 (2005)
Manber, U., Myers, G.: Suffix array: A new method for on-line string searches. SIAM Journal on Computing 22, 935–948 (1993)
Nagao, M., Mori, S.: A new method of n-gram statistics for large number of n and automatic extraction of words and phrases from large text data of Japanese. In: Proceedings of the 15th International Conference on Computational Linguistics (COLING 1994), Kyoto, Japan (August 1994)
Fordyce, C.S.: Overview of the IWSLT 2007 evaluation campaign. In: Proceedings of the 4th International Workshop on Spoken Language Translation (IWSLT 2007), Trento, Italy, October 2007, pp. 1–12 (2007)
Takezawa, T., Sumita, E., Sugaya, F., Yamamoto, H., Yamamoto, S.: Toward a broad-coverage bilingual corpus for speech translation of travel conversation in the real world. In: Proceedings of the third International Conference on Language Resources and Evaluation (LREC) Las Palmas de Gran Canaria, Spain, pp. 147–152 (2002)
Matsumoto, Y.: Japanese morphological analyser ChaSen. Journal of Information Processing Society of Japan 41, 1208–1214 (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Lardilleux, A., Lepage, Y. (2009). Hapax Legomena: Their Contribution in Number and Efficiency to Word Alignment. In: Vetulani, Z., Uszkoreit, H. (eds) Human Language Technology. Challenges of the Information Society. LTC 2007. Lecture Notes in Computer Science(), vol 5603. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-04235-5_38
Download citation
DOI: https://doi.org/10.1007/978-3-642-04235-5_38
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-04234-8
Online ISBN: 978-3-642-04235-5
eBook Packages: Computer ScienceComputer Science (R0)