Skip to main content
Springer Nature Link
Log in

Stochastic Analysis of Lexical and Semantic Enhanced Structural Language Model

  • Conference paper
Grammatical Inference: Algorithms and Applications (ICGI 2006)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 4201))

Included in the following conference series:

Abstract

In this paper, we present a directed Markov random field model that integrates trigram models, structural language models (SLM) and probabilistic latent semantic analysis (PLSA) for the purpose of statistical language modeling. The SLM is essentially a generalization of shift-reduce probabilistic push-down automata thus more complex and powerful than probabilistic context free grammars (PCFGs). The added context-sensitiveness due to trigrams and PLSAs and violation of tree structure in the topology of the underlying random field model make the inference and parameter estimation problems plausibly intractable, however the analysis of the behavior of the lexical and semantic enhanced structural language model leads to a generalized inside-outside algorithm and thus to rigorous exact EM type re-estimation of the composite language model parameters.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Abney, S., McAllester, D., Pereira, F.: Relating probabilistic grammars and automata. In: Proceedings of ACL, pp. 542–549 (1999)

    Google Scholar 

  2. Baker, J.: Trainable grammars for speech recognition. In: Proceedings of the 97th Meeting of the Acoustical Society of America, pp. 547–550 (1979)

    Google Scholar 

  3. Chelba, C.: Exploiting syntactic structure for natural language modeling. Ph.D. Dissertation. Johns Hopkins University (1999)

    Google Scholar 

  4. Chelba, C., Jelinek, F.: Structured language modeling. Computer Speech and Language 14(4), 283–332 (2000)

    Article  Google Scholar 

  5. Chomsky, N.: Three models for the description of language. IRE Transactions on Information Theory 2(3), 113–124 (1956)

    Article  Google Scholar 

  6. Dempster, A., Laird, N., Rubin, D.: Maximum likelihood estimation from incomplete data via the EM algorithm. Journal of Royal Statistical Society 39, 1–38 (1977)

    MATH  MathSciNet  Google Scholar 

  7. Hofmann, T.: Unsupervised learning by probabilistic latent semantic analysis. Machine Learning 42(1), 177–196 (2001)

    Article  MATH  Google Scholar 

  8. Hopcroft, J., Ullman, J.: Introduction to Automata Theory, Languages and Computation. Addison-Wesley, Reading (1979)

    MATH  Google Scholar 

  9. Jelinek, F., Lafferty, J., Mercer, R.: Basic methods of probabilistic context-free grammars. In: Laface, P., De Mori, R. (eds.) Speech Recognition and Understanding: Recent Advances, Trends, and Applications, pp. 347–360. Springer, Heidelberg (1992)

    Google Scholar 

  10. Jelinek, F.: Statistical Methods for Speech Recognition. MIT Press, Cambridge (1998)

    Google Scholar 

  11. Jelinek, F., Chelba, C.: Putting language into language modeling. In: Proceedings of the 6th EuroSpeech Communication and Technology, pp. 1–6 (1999)

    Google Scholar 

  12. Jelinek, F.: Stochastic analysis of structured language modeling. In: Johnson, M., Khudanpur, S., Ostendorf, M., Rosenfeld, R. (eds.) Mathematical Foundations of Speech and Language Processing, pp. 37–72. Springer, Heidelberg (2004)

    Google Scholar 

  13. Khudanpur, S., Wu, J.: Maximum entropy techniques for exploiting syntactic, semantic and collocational dependencies in language modeling. Computer Speech and Language 14(4), 355–372 (2000)

    Article  Google Scholar 

  14. Kschischang, F., Frey, B., Loeliger, H.: Factor graphs and the sum-product algorithm. IEEE Transactions on Information Theory 47(2), 498–519 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  15. Lafferty, J.: A derivation of the inside-outside algorithm from the EM algorithm. IBM Research Report 21636 (2000)

    Google Scholar 

  16. Lari, K., Young, S.: The estimation of stochastic context-free grammars using the inside-outside algorithm. Computer Speech and Language 4, 35–56 (1990)

    Article  Google Scholar 

  17. Lauritzen, S.: Graphical Models. Oxford Press, Oxford (1996)

    Google Scholar 

  18. Shannon, C.: A mathematical theory of communication. Bell System Technical Journal 27(2), 379–423 (1948)

    MATH  MathSciNet  Google Scholar 

  19. Wang, S., Wang, S., Greiner, R., Schuurmans, D., Cheng, L.: Exploiting syntactic, semantic and lexical regularities in language modeling via directed Markov random fields. In: The 22nd International Conference on Machine Learning, pp. 953–960 (2005)

    Google Scholar 

  20. Younger, D.: Recognition and parsing of context free languages in time N 3. Information and Control 10, 198–208 (1967)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Wright State University, USA

    Shaojun Wang

  2. University of Alberta, Canada

    Shaojun Wang, Russell Greiner & Dale Schuurmans

  3. Oracle, USA

    Shaomin Wang

  4. National ICT, Australia

    Li Cheng

Authors
  1. Shaojun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shaomin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Li Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Russell Greiner
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dale Schuurmans
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Biosciences and Informatics, Keio University, 3-14-1 Hiyoshi, Kohoku-ku, 223-8522, Yokohama, Japan

    Yasubumi Sakakibara

  2. Dept. of Computer Science, Kyoto Sangyo University, Kamigamo Motoyama, Kita-ku, Kyoto, Japan

    Satoshi Kobayashi

  3. Japan Biological Informatics Consortium, 10F TIME24 Building, 2-45 Aomi, Koto-ku, 135-8073, Tokyo, Japan

    Kengo Sato

  4. Department of Information and Communication Engineering, Graduate School of Electro-Communications, The University of Electro-Communications, 1-5-1 Chofugaoka, Chofu-shi, 182-8585, Tokyo, Japan

    Tetsuro Nishino

  5. Department of Information and Communication Engineering, Faculty of Electro-Communications, The University of Electro-Communications, Chofugaoka 1–5–1, Chofu, 182-8585, Tokyo, Japan

    Etsuji Tomita

Rights and permissions

Reprints and permissions

Copyright information

© 2006 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Wang, S., Wang, S., Cheng, L., Greiner, R., Schuurmans, D. (2006). Stochastic Analysis of Lexical and Semantic Enhanced Structural Language Model. In: Sakakibara, Y., Kobayashi, S., Sato, K., Nishino, T., Tomita, E. (eds) Grammatical Inference: Algorithms and Applications. ICGI 2006. Lecture Notes in Computer Science(), vol 4201. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11872436_9

Download citation

  • DOI: https://doi.org/10.1007/11872436_9

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-45264-5

  • Online ISBN: 978-3-540-45265-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics