Abstract
In this paper we investigate parameter dependencies in the echo state network (ESN). In particular, we investigate the interplay between reservoir sizes and the choice of the average absolute output feedback connection weight values (W OFB ). We consider the multiple sine wave oscillator problem and the powered sine problem. The results show that somewhat contrary to basic intuition (1) smaller reservoir sizes often yield better networks with higher probability; (2) large W OFB values paired with comparatively large reservoirs may strongly decrease the likelihood of generating effective networks; (3) the likelihood of generating an effective ESN depends non-linearly on the choice of W OFB : very small and large weight values often yield higher likelihoods of generating effective ESNs than networks resulting from intermediate W OFB choices. While the considered test problems are rather simple, the insights gained need to be considered when designing effective ESNs for the problem at hand. Nonetheless, further studies appear necessary to be able to explain the actual reasons behind the observed phenomena.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks. GMD Report 148, German National Research Center for Information Technology (2001)
Jaeger, H.: Short term memory in echo state networks, German National Research Center for Information Technology. Technical Report GMD report 152 (2002)
Ishii, K., van der Zant, T., Becanovic, V., Plöger, P.: Identification of Motion with Echo State Network. In: OCEANS 2004, MTTS/IEEE TECHNO-OCEAN 2004, vol. 3, pp. 1205–1210 (2004)
Verstraeten, D., Schrauwen, B., D’Haene, M., Stroobandt, D.: An experimental unification of reservoir computing methods. Neural Networks 20(3), 391–403 (2007)
Rodan, A., Tino, P.: Minimum complexity echo state network. IEEE Transactions on Neural Networks 22(1), 131–144 (2011), doi:10.1109/TNN.2010.2089641
Roeschies, B., Igel, C.: Structure optimization of reservoir networks. Logic Journal of IGPL Advance Access (published September 3, 2009)
Xue, Y., Yang, L., Haykin, S.: Decoupled echo state networks with lateral inhibition. Neural Networks, 365–376 (2007)
Holzmann, G., Hauser, H.: Echo State Networks with Filter Neurons and a Delay & Sum Readout. Neural Networks 23(2), 244–256 (2010)
Koryakin, D., Lohmann, J., Butz, M.V.: Balanced Echo State Networks. Submitted for Publication in Neural Networks (2012)
Jaeger, H.: A tutorial on training recurrent neural networks, covering BPPT, RTRL, EKF and the “echo state network” approach, GMD Report 159, German National Research Center for Information Technology, 48 p. (2002)
Ozturk, M.C., Xu, D., Principe, J.C.: Analysis and design of echo state networks. Neural Computation 19(1), 111–138 (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Koryakin, D., Butz, M.V. (2012). Reservoir Sizes and Feedback Weights Interact Non-linearly in Echo State Networks. In: Villa, A.E.P., Duch, W., Érdi, P., Masulli, F., Palm, G. (eds) Artificial Neural Networks and Machine Learning – ICANN 2012. ICANN 2012. Lecture Notes in Computer Science, vol 7552. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33269-2_63
Download citation
DOI: https://doi.org/10.1007/978-3-642-33269-2_63
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-33268-5
Online ISBN: 978-3-642-33269-2
eBook Packages: Computer ScienceComputer Science (R0)