Abstract
This paper proposes a method of combining two n-gram language models, one constructed from a very small corpus of the right domain of interest, the other constructed from a large but less adequate corpus, resulting in a significantly enhanced language model. This method is based on the observation that a small corpus from the right domain has high quality n-grams but has serious sparseness problem, while a large corpus from a different domain has more n-gram statistics but inadequately biased. Two n-gram models are combined by extending the idea of Katz’s backoff. We ran experiments with 3-gram language models constructed from newspaper corpora of several million to tens of million words together with models from smaller broadcast news corpora. The target domain was broadcast news. We obtained significant improvement (30%) by incorporating a small corpus around one thirtieth size of the newspaper corpus.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Rosenfeld, R.: Adaptive Statistical Language Modeling: A Maximum Entropy Approach, Ph.D. dissertation, Carnegie-Mellon University (April 1994)
Katz, S.M.: Estimation of Probabilities from Sparse Data for the Language Model Component of a Speech Recognizer. IEEE Transactions on Acoustics, Speech and signal Processing ASSP-35, 400–401 (1987)
Good, I.J.: The Population frequencies of species and the Estimation of Population parameters. Biometrica 40, parts 3,4, 237–264
Goodman, J.T.: A Bit of Progress in Language Modeling. Computer Speech and Language 15, 403–434 (2001)
Bahl, L., Brown, P., de Souza, P., Mercer, R.: A tree-based Statistical Language Model for natural Language Speech Recognition. IEEE Tr. on Acoustics, Speech, and Signal Processing 37, 1001–1008 (1989)
Akiba, T., Itou, K., Fujii, A., Ishikawa, T.: Selective Backoff smoothing for incorporating grammatical constraints into the n-gram language model. In: Proc. International Conference on Spoken Language Processing, September 2002, pp. 881–884 (2002)
Chen, S.F., et al.: Topic Adaptation for Language Modeling Using Unnormalized Exponential Models. In: Proc. ICASSP 1998, May 12-15, vol. 2, pp. 681–684 (1998)
Jurafsky, D., Martin, J.H.: Speech and Language Processing. Prentice-Hall, Englewood Cliffs (2000)
Jelinek, F., et al.: Perplexity – A Measure of the difficulty of speech recognition tasks. Journal of the Acoustics Society of America 62, S63(suppl. 1) (1977)
Cho, S.: http://nlp.mju.ac.kr/dualsource/
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Cho, S., Kim, S., Park, J., Lee, Y. (2004). Overcoming the Sparseness Problem of Spoken Language Corpora Using Other Large Corpora of Distinct Characteristics. In: Gelbukh, A. (eds) Computational Linguistics and Intelligent Text Processing. CICLing 2004. Lecture Notes in Computer Science, vol 2945. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-24630-5_48
Download citation
DOI: https://doi.org/10.1007/978-3-540-24630-5_48
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-21006-1
Online ISBN: 978-3-540-24630-5
eBook Packages: Springer Book Archive