Abstract
In this paper, we propose an ESN having multiple timescale layer and working memories as a probabilistic language model. The reservoir of the proposed model is composed of three neuron groups each with an associated time constant, which enables the model to learn the hierarchical structure of language. We add working memories to enhance the effect of multiple timescale layers. As shown by the experiments, the proposed model can be trained efficiently and accurately to predict the next word from given words. In addition, we found that use of working memories is especially effective in learning grammatical structure.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Arisoy, E., Sainath, T.N., Kingsbury, B., Ramabhadran, B.: Deep neural network language models. In: Proceedings of the NAACL-HLT 2012 Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, WLM 2012, Stroudsburg, PA, USA, pp. 20–28. Association for Computational Linguistics (2012)
Russell, S.J., Norvig, P.: Artificial Intelligence: A Modern Approach, 2nd edn. Pearson Education (2003)
Jurafsky, D., Martin, J.H.: Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 1st edn. Prentice Hall PTR, Upper Saddle River (2000)
Mikolov, T., Kombrink, S., Burget, L., Cernocky, J., Khudanpur, S.: Extensions of recurrent neural network language model. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5528–5531 (2011)
Hinoshita, W., Arie, H., Tani, J., Okuno, H.G., Ogata, T.: Emergence of hierarchical structure mirroring linguistic composition in a recurrent neural network. Neural Networks 24(4), 311–320 (2011)
Tong, M.H., Bickett, A.D., Christiansen, E.M., Cottrell, G.W.: Learning grammatical structure with Echo State Networks. Neural Networks 20(3), 424–432 (2007)
Elman, J.L.: Learning and development in neural networks: The importance of starting small. Cognition 48(1), 71–99 (1993)
Jaeger, H.: The” echo state” approach to analysing and training recurrent neural networks-with an erratum note, vol. 148. German National Research Center for Information Technology GMD Technical Report, Bonn (2001)
Boccato, L., Lopes, A., Attux, R., Zuben, F.J.V.: An extended echo state network using Volterra filtering and principal component analysis. Neural Networks 32, 292–302 (2012)
Lukosevicius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Computer Science Review 3(3), 127–149 (2009)
Pascanu, R., Jaeger, H.: A neurodynamical model for working memory. Neural Networks 24(2), 199–207 (2011)
Yamashita, Y., Tani, J.: Emergence of Functional Hierarchy in a Multiple Timescale Neural Network Model: A Humanoid Robot Experiment. PLoS Comput. Biol. 4(11) (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Homma, Y., Hagiwara, M. (2013). An Echo State Network with Working Memories for Probabilistic Language Modeling. In: Mladenov, V., Koprinkova-Hristova, P., Palm, G., Villa, A.E.P., Appollini, B., Kasabov, N. (eds) Artificial Neural Networks and Machine Learning – ICANN 2013. ICANN 2013. Lecture Notes in Computer Science, vol 8131. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-40728-4_74
Download citation
DOI: https://doi.org/10.1007/978-3-642-40728-4_74
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-40727-7
Online ISBN: 978-3-642-40728-4
eBook Packages: Computer ScienceComputer Science (R0)