Abstract
In Reservoir Computing, signals or sequences are fed into a set of interconnected non-linear units (neurons) with capabilities for storing information (reservoir). The reservoir generates an expanded representation of the input, which is subsequently mapped onto the desired output using a trained output layer (readout). However, despite their success in various experimental tasks, the dynamics of the reservoir are not yet well understood. In this paper we introduce a new technique, based on the well known Singular Value Decomposition (SVD), to obtain the main dynamic modes of the reservoir when excited with an input signal. We conduct experiments using Echo State Networks (ESN) to demonstrate the technique’s potential and its ability to decompose input signals into Principal Component Modes as expanded by the reservoir. We expect that this approach will open new possibilities in its application to the field of visual analytics in process state visualisation, determination of attribute vectors, and detection of novelties. Furthermore, this technique could serve as a foundation for a better understanding of the reservoir’s dynamic state that could help in other areas of research, such as domain shift or continual learning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The original version is given here. Numerous variants exist in the literature incorporating, for example, a bias term added to \(\textbf{u}(k)\), direct effect of \(\textbf{u}(k)\) on the output, feedback term (inclusion of \(\textbf{y}(k)\)) in the equation of state, application of a low-pass filter to the states, use of non-linear regression models to obtain \(\textbf{y}(k)\) from \(\textbf{x}(k)\), etc. A detailed description of many of these variants can be found in [12].
- 2.
Other parameters are: \(\rho = 0.95\), sparsity = 0.01, leaking rate = 0.025, input scaling = 10 and warm up = 20. A bias is added to the inputs.
References
Bollt, E.: On explaining the surprising success of reservoir computing forecaster of chaos? The universal machine learning dynamical system with contrast to VAR and DMD. Chaos: Interdiscip. J. Nonlinear Sci. 31, 013108 (2021). https://doi.org/10.1063/5.0024890. http://aip.scitation.org/doi/10.1063/5.0024890
Buehner, M., Young, P.: A tighter bound for the echo state property. IEEE Trans. Neural Netw. 17, 820–824 (2006). https://doi.org/10.1109/TNN.2006.872357
Dylewsky, D., Barajas-Solano, D., Ma, T., Tartakovsky, A.M., Kutz, J.N.: Stochastically forced ensemble dynamic mode decomposition for forecasting and analysis of near-periodic systems. IEEE Access 10, 33440–33448 (2022). https://doi.org/10.1109/ACCESS.2022.3161438
Dylewsky, D., Kaiser, E., Brunton, S.L., Kutz, J.N.: Principal component trajectories for modeling spectrally continuous dynamics as forced linear systems. Phys. Rev. E 105, 015312 (2022). https://doi.org/10.1103/PHYSREVE.105.015312/FIGURES/10/MEDIUM. https://journals.aps.org/pre/abstract/10.1103/PhysRevE.105.015312
Díaz Blanco, I., Cuadrado Vega, A.A., Muñiz, A.G., García Pérez, D.: Dataicann: datos de vibración y corriente de un motor de inducción. https://digibuo.uniovi.es/dspace/handle/10651/53461 (2019)
Gallicchio, C., Micheli, A.: A Markovian characterization of redundancy in echo state networks by PCA. In: Proceedings of the 18th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN-2010), pp. 321–326 (2010)
Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks. GMD Report 148, GMD - German National Research Institute for Computer Science (2001). http://www.faculty.jacobs-university.de/hjaeger/pubs/EchoStatesTechRep.pdf
Jaeger, H.: Echo state network. Scholarpedia 2(9), 2330 (2007)
Jaeger, H., Haas, H.: Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication. Science 304, 78–80 2004). https://doi.org/10.1126/science.1091277. https://www.science.org/doi/10.1126/science.1091277
Khoshrou, A., Pauwels, E.J.: Data-driven pattern identification and outlier detection in time series. Adv. Intell. Syst. Comput. 858, 471–484 (2018). https://doi.org/10.1007/978-3-030-01174-1-35. http://arxiv.org/abs/1807.03386
Li, F., Wang, X., Li, Y.: Effects of singular value spectrum on the performance of echo state network. Neurocomputing 358, 414–423 (2019). https://doi.org/10.1016/j.neucom.2019.05.068. https://linkinghub.elsevier.com/retrieve/pii/S092523121930774X
Lukoševičius, M.: A practical guide to applying echo state networks. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 659–686. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_36
Lukoševičius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Comput. Sci. Rev. 3(3), 127–149 (2009)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323, 533–536 (1986). https://doi.org/10.1038/323533a0. https://www.nature.com/articles/323533a0
Tanaka, G., et al.: Recent advances in physical reservoir computing: a review. Neural Netw. 115, 100–123 (2019). https://doi.org/10.1016/J.NEUNET.2019.03.005
Verstraeten, D., Schrauwen, B.: On the quantification of dynamics in reservoir computing. In: Alippi, C., Polycarpou, M., Panayiotou, C., Ellinas, G. (eds.) ICANN 2009. LNCS, vol. 5768, pp. 985–994. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04274-4_101
Acknowledgements
This work was supported by the Ministerio de Ciencia e Innovación / Agencia Estatal de Investigación (MCIN/AEI/ 10.13039/ 501100011033) grant [PID2020-115401GB-I00].
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 IFIP International Federation for Information Processing
About this paper
Cite this paper
Enguita, J.M., Díaz, I., García, D., Cuadrado, A.A., Rodríguez, J.R. (2023). Principal Component Modes of Reservoir Dynamics in Reservoir Computing. In: Maglogiannis, I., Iliadis, L., Papaleonidas, A., Chochliouros, I. (eds) Artificial Intelligence Applications and Innovations. AIAI 2023 IFIP WG 12.5 International Workshops. AIAI 2023. IFIP Advances in Information and Communication Technology, vol 677. Springer, Cham. https://doi.org/10.1007/978-3-031-34171-7_35
Download citation
DOI: https://doi.org/10.1007/978-3-031-34171-7_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-34170-0
Online ISBN: 978-3-031-34171-7
eBook Packages: Computer ScienceComputer Science (R0)
