Abstract
Language model is one of the basic research issues of natural language processing, and which is the premise for realizing more complicated tasks such as speech recognition, machine translation and question answering system. In recent years, neural network language model has become a research hotspot, which greatly enhances the application effect of language model. In this paper, a recurrent neural network language model (RNNLM) based on word embedding is proposed, and the word embedding of each word is generated by pre-training the text data with skip-gram model. The n-gram language model, RNNLM based on one-hot and RNNLM based on word embedding are evaluated on three different public datasets. The experimental results show that the RNNLM based on word embedding performs best, and which can reduce the perplexity of language model significantly.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Jun, W.: The Beauty of Mathematics. Posts and Telecommunications Press, Beijing (2012)
Brown, P.F., Desouza, P.V., Mercer, R.L., Pietra, V.J.D., Lai, J.C.: Class-based n-gram models of natural language. Comput. Linguist. 18(4), 467–479 (1992)
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C.: A neural probabilistic language model. J. Mach. Learn. Res. 3(2), 1137–1155 (2003)
Mikolov, T., Karafiát, M., Burget, L., Černocký, J., Khudanpur, S.: Recurrent neural network based language model. In: Proceedings of the Eleventh Annual Conference of the International Speech Communication Association, pp. 1045–1048. International Speech Communication Association, Makuhari, Chiba (2010)
Bird, S., Loper, E.: NLTK: the natural language toolkit. In: Proceedings of the ACL 2004 on Interactive Poster and Demonstration Sessions, p. 31. Association for Computational Linguistics (2004)
Hinton, G.E.: Learning distributed representations of concepts. In: Proceedings of the Eighth Annual Conference of the Cognitive Science Society, vol. 1, p. 12. Oxford University Press, Oxford, UK (1986)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. ArXiv Preprint arXiv:1301.3781 (2013)
Theano 0.9.0 documentation. http://deeplearning.net/software/theano/. Accessed 10 Dec 2017
Jian, Z.: Research on recurrent neural network language model in continuous speech recognition. The PLA Information Engineering University, Zhengzhou, pp. 19–20 (2014)
Recurrent neural networks tutorial. http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-2-implementing-a-language-model-rnn-with-python-numpy-and-theano/. Accessed 12 Jan 2018
Marcus, M.P., Marcinkiewicz, M.A., Santorini, B.: Building a large annotated corpus of English: the Penn Treebank. Comput. Linguist. 19(2), 313–330 (1993)
Stolcke, A.: SRILM-an extensible language modeling toolkit. In: Proceedings of the Seventh International Conference on Spoken Language Processing, pp. 901–904 (2002)
Acknowledgements
This work is supported by the Beijing Natural Science Foundation under Grant No. 4162067.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Li, S., Xu, J. (2018). A Recurrent Neural Network Language Model Based on Word Embedding. In: U, L., Xie, H. (eds) Web and Big Data. APWeb-WAIM 2018. Lecture Notes in Computer Science(), vol 11268. Springer, Cham. https://doi.org/10.1007/978-3-030-01298-4_30
Download citation
DOI: https://doi.org/10.1007/978-3-030-01298-4_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-01297-7
Online ISBN: 978-3-030-01298-4
eBook Packages: Computer ScienceComputer Science (R0)