Abstract
Echo State Networks (ESNs) represent a successful methodology for efficient modeling of Recurrent Neural Networks. Untrained recurrent dynamics in ESNs apparently need to comply a trade-off between the two desirable features of implementing a long memory over past inputs and the ability of modeling non-linear dynamics. In this paper, we analyze such memory/non-linearity trade-off from the perspective of recurrent model design. In particular, we propose two variants to the standard ESN model, aiming at combining linear and non-linear dynamics both in the architectural setup of the recurrent system, and at the level of recurrent units activation functions. The proposed models are experimentally assessed on ad-hoc defined tasks as well as on standard benchmarks in the area of Reservoir Computing. Results show that the introduced ESN variants can grasp the proper trade-off between memory and non-linearity requirements, at the same time allowing to improve the performance of standard ESNs. Moreover, the analysis of the employed degree of non-linearity in the reservoir system can provide useful insights on the characterization of the learning task at hand.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
The spectral radius of a matrix is the maximum among its eigenvalues in modulus.
- 2.
\(y^{tg}_{N}(t) = 0.3y^{tg}_{N}(t-1)+0.5y^{tg}_{N}(t-1)(\sum _{i=1}^{10} y^{tg}_{N}(t-i))+1.5 u(t-10)u(t-1)+0.1)\).
- 3.
We used a value of \(\tau = 17\) as control parameter for the Mackey-Glass equation, as common in the RC literature.
References
Bacciu, D., Barsocchi, P., Chessa, S., Gallicchio, C., Micheli, A.: An experimental characterization of reservoir computing in ambient assisted living applications. Neural Comput. Appl. 24(6), 1451–1464 (2014)
Crisostomi, E., Gallicchio, C., Micheli, A., Raugi, M., Tucci, M.: Prediction of the italian electricity price for smart grid applications. Neurocomputing 170, 286–295 (2015)
Dragone, M., Gallicchio, C., Guzman, R., Micheli, A.: RSS-based robot localization in critical environments using reservoir computing. In: Proceedings of the 24th European Symposium on Artificial Neural Networks (ESANN), pp. 71–76 (2016)
Gallicchio, C., Martin-Guerrero, J., Micheli, A., Soria-Olivas, E.: Randomized machine learning approaches: recent developments and challenges. In: 25th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2017), pp. 77–86 (2017). i6doc.com
Gallicchio, C., Micheli, A.: Architectural and markovian factors of echo state networks. Neural Netw. 24(5), 440–456 (2011)
Gallicchio, C., Micheli, A., Pedrelli, L.: Deep reservoir computing: a critical experimental analysis. Neurocomputing 268, 87–99 (2017)
Gallicchio, C., Micheli, A., Tiňo, P.: Randomized recurrent neural networks. In: 26th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2018), pp. 415–424 (2018). i6doc.com
Gallicchio, C., Micheli, A.: Deep Echo State Network (DeepESN): A brief survey. arXiv preprint arXiv:1712.04323 (2017)
Inubushi, M., Yoshimura, K.: Reservoir computing beyond memory-nonlinearity trade-off. Sci. Rep. 7(1), 10199 (2017)
Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks - with an erratum note. Technical report, GMD - German National Research Institute for Computer Science (2001)
Jaeger, H.: Short term memory in echo state networks. Technical report, German National Research Center for Information Technology (2001)
Jaeger, H., Haas, H.: Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304(5667), 78–80 (2004)
Lukoševičius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3(3), 127–149 (2009)
Palumbo, F., Gallicchio, C., Pucci, R., Micheli, A.: Human activity recognition using multisensor data fusion based on reservoir computing. J. Ambient. Intell. Smart Environ. 8(2), 87–107 (2016)
Triefenbach, F., Jalalvand, A., Demuynck, K., Martens, J.P.: Acoustic modeling with hierarchical reservoirs. IEEE Trans. Audio Speech Lang. Process. 21(11), 2439–2450 (2013)
Verstraeten, D., Schrauwen, B., d’Haene, M., Stroobandt, D.: An experimental unification of reservoir computing methods. Neural Netw. 20(3), 391–403 (2007)
Verstraeten, D., Dambre, J., Dutoit, X., Schrauwen, B.: Memory versus non-linearity in reservoirs. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Di Gregorio, E., Gallicchio, C., Micheli, A. (2018). Combining Memory and Non-linearity in Echo State Networks. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds) Artificial Neural Networks and Machine Learning – ICANN 2018. ICANN 2018. Lecture Notes in Computer Science(), vol 11140. Springer, Cham. https://doi.org/10.1007/978-3-030-01421-6_53
Download citation
DOI: https://doi.org/10.1007/978-3-030-01421-6_53
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-01420-9
Online ISBN: 978-3-030-01421-6
eBook Packages: Computer ScienceComputer Science (R0)