Skip to main content
SpringerLink
Log in

On the Interpretation and Characterization of Echo State Networks Dynamics: A Complex Systems Perspective

  • Chapter
  • First Online:
Book cover Advances in Data Analysis with Computational Intelligence Methods

Part of the book series: Studies in Computational Intelligence ((SCI,volume 738))

Abstract

In this chapter, we discuss recently developed methods for characterizing the dynamics of recurrent neural networks. Such methods rely on theory and concepts coming from the field of complex systems. We focus on a class of recurrent networks called echo state networks. First, we present a method to analyze and characterize the evolution of its internal state. This allows to provide a qualitative interpretation of the network dynamics. In addition, it allows to assess the stability of the system, a necessary requirement in many practical applications. Successively, we focus on the identification of the onset of criticality in such networks. We discuss an unsupervised method based on Fisher information, which can be used to tune the network hyperparameters. With respect to standard supervised techniques, we show that the proposed approach offers several advantages and is effective on a number of tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.reservoir-computing.org/node/129.

  2. 2.

    http://www.recurrence-plot.tk/.

References

  1. Aljadeff, J., Stern, M., Sharpee, T.: Transition to chaos in random networks with cell-type-specific connectivity. Phys. Rev. Lett. 114, 088101 (2015). doi:10.1103/PhysRevLett.114.088101

    Article  Google Scholar 

  2. Barzel, B., Barabási, A.-L.: Universality in network dynamics. Nat. Phys. 9(10), 673–681 (2013). doi:10.1038/nphys2741

    Article  Google Scholar 

  3. Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994). ISSN 1045-9227. doi:10.1109/72.279181

  4. Berisha, V., Hero, A.Q. III.: Empirical non-parametric estimation of the Fisher information. IEEE Signal Process. Lett. 22(7), 988–992 (2015). ISSN 1070-9908. doi:10.1109/LSP.2014.2378514

  5. Bertschinger, N., Natschläger, T.: Real-time computation at the edge of chaos in recurrent neural networks. Neural Comput. 16(7), 1413–1436 (2004). doi:10.1162/089976604323057443

    Article  MATH  Google Scholar 

  6. Bianchi, F.M., Livi, L., Alippi, C.: Investigating echo state networks dynamics by means of recurrence analysis. IEEE Trans. Neural Netw. Learn. Syst. 1–13 (2016). doi:10.1109/TNNLS.2016.2630802

  7. Boedecker, J., Obst, O., Lizier, J.T., Mayer, N.M., Asada, M.: Information processing in echo state networks at the edge of chaos. Theory Biosci. 131(3), 205–213 (2012). doi:10.1007/s12064-011-0146-8

    Article  Google Scholar 

  8. Charles, A., Yin, D., Rozell, C.: Distributed sequence memory of multidimensional inputs in recurrent networks. arXiv:1605.08346 (2016)

  9. De Arcangelis, L., Lombardi, F., Herrmann, H.J.: Criticality in the brain. J. Stat. Mech. Theory Exp. 2014(3), P03026 (2014). doi:10.1088/1742-5468/2014/03/P03026

    Article  MathSciNet  Google Scholar 

  10. Enel, P., Procyk, E., Quilodran, R., Dominey, P.F.: Reservoir computing properties of neural dynamics in prefrontal cortex. PLoS Comput. Biol. 12(6), e1004967 (2016). doi:10.1371/journal.pcbi.1004967

    Article  Google Scholar 

  11. Eroglu, D., Peron, T.K.D.M., Marwan, N., Rodrigues, F.A., da Costa, L.F., Sebek, M., Kiss, I.Z., Kurths, J.: Entropy of weighted recurrence plots. Phys. Rev. E 90(4), 042919 (2014). doi:10.1103/PhysRevE.90.042919

  12. Elman, J.L.: Finding structure in time. Cogn. Sci. 14(2), 179–211 (1990). ISSN 0364-0213. doi:10.1016/0364-0213(90)90002-E

  13. Friedman, J.H., Rafsky, L.C.: Multivariate generalizations of the Wald-Wolfowitz and Smirnov two-sample tests. Ann. Stat. 7(4), 697–717 (1979)

    Article  MathSciNet  MATH  Google Scholar 

  14. Grigolini, P.: Emergence of biological complexity: criticality, renewal and memory. Chaos, Solitons Fractals (2015). doi:10.1016/j.chaos.2015.07.025

  15. Hammer, B., Micheli, A., Sperduti, A., Strickert, M.: Recursive self-organizing network models. Neural Netw. 17(8–9), 1061–1085 (2004). ISSN 0893-6080. doi:10.1016/j.neunet.2004.06.009

  16. Hidalgo, J., Grilli, J., Suweis, S., Muñoz, M.A., Banavar, J.R., Maritan, A.: Information-based fitness and the emergence of criticality in living systems. Proc. Natl. Acad. Sci. 111(28), 10095–10100 (2014). doi:10.1073/pnas.1319166111

    Article  Google Scholar 

  17. Hidalgo, J., Grilli, J., Suweis, S., Maritan, A., Muñoz, M.A.: Cooperation, competition and the emergence of criticality in communities of adaptive systems. J. Stat. Mech. Theory Exp. 2016(3), 033203 (2016). doi:10.1088/1742-5468/2016/03/033203

    Article  MathSciNet  Google Scholar 

  18. Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, vol. 148, p. 34 (2001)

    Google Scholar 

  19. Lacasa, L., Nicosia, V., Latora, V.: Network structure of multivariate time series. Sci. Rep. 5, (2015). doi:10.1038/srep15508

  20. Langton, C.G.: Computation at the edge of chaos: Phase transitions and emergent computation. Phys. D Nonlinear Phenom. 42(1), 12–37 (1990). doi:10.1016/0167-2789(90)90064-V

    Article  MathSciNet  Google Scholar 

  21. Legenstein, R., Maass, W.: Edge of chaos and prediction of computational performance for neural circuit models. Neural Netw. 20(3), 323–334 (2007). doi:10.1016/j.neunet.2007.04.017

    Article  MATH  Google Scholar 

  22. Livi, L., Bianchi, F.M., Alippi, C.: Determination of the edge of criticality in echo state networks through Fisher information maximization. IEEE Trans. Neural Netw. Learn. Syst. 1–12 (2017). doi:10.1109/TNNLS.2016.2644268

  23. Maass, W., Joshi, P., Sontag, E.D.: Computational aspects of feedback in neural circuits. PLoS Comput. Biol. 3(1), e165 (2007). doi:10.1371/journal.pcbi.0020165.eor

    Article  MathSciNet  Google Scholar 

  24. Magnus, J.R., Neudecker, H.: Matrix Differential Calculus with Applications in Statistics and Econometrics. Wiley, New York (1995)

    MATH  Google Scholar 

  25. Manjunath, G., Jaeger, H.: Echo state property linked to an input: Exploring a fundamental characteristic of recurrent neural networks. Neural Comput. 25(3), 671–696 (2013). doi:10.1162/NECO_a_00411

    Article  MathSciNet  MATH  Google Scholar 

  26. Marichal, R.L., Piñeiro, J.D.: Analysis of multiple quasi-periodic orbits in recurrent neural networks. Neurocomputing 162, 85–95 (2015). doi:10.1016/j.neucom.2015.04.001

    Article  Google Scholar 

  27. Marwan, N.: How to avoid potential pitfalls in recurrence plot based data analysis. Int. J. Bifurcat. Chaos 21(04), 1003–1017 (2011). doi:10.1142/S0218127411029008

    Article  MathSciNet  MATH  Google Scholar 

  28. Marwan, N., Kurths, J.: Line structures in recurrence plots. Phys. Lett. A 336(4), 349–357 (2005). doi:10.1016/j.physleta.2004.12.056

    Article  MATH  Google Scholar 

  29. Marwan, N., Carmen, M., Thiel, R.M., Kurths, J.: Recurrence plots for the analysis of complex systems. Phys. Rep. 438(5), 237–329 (2007). doi:10.1016/j.physrep.2006.11.001

  30. Marwan, N., Schinkel, S., Kurths, J.: Recurrence plots 25 years later-Gaining confidence in dynamical transitions. EPL (Europhys. Lett.) 101(2), 20007 (2013). doi:10.1209/0295-5075/101/20007

    Article  Google Scholar 

  31. Massar, M., Massar, S.: Mean-field theory of echo state networks. Phys. Rev. E 87(4), 042809 (2013). doi:10.1103/PhysRevE.87.042809

    Article  MATH  Google Scholar 

  32. Massobrio, P., de Arcangelis, L., Pasquale, V., Jensen, H.J., Plenz, D.: Criticality as a signature of healthy neural systems. Front. Syst. Neurosci. 9, 22 (2015). doi:10.3389/fnsys.2015.00022

    Article  Google Scholar 

  33. Mastromatteo, I., Marsili, M.: On the criticality of inferred models. J. Stat. Mech. Theory Exp. 2011(10), P10012 (2011). doi:10.1088/1742-5468/2011/10/P10012

    Article  Google Scholar 

  34. Mora, T., Bialek, W.: Are biological systems poised at criticality? J. Stat. Phys. 144(2), 268–302 (2011). doi:10.1007/s10955-011-0229-4

    Article  MathSciNet  MATH  Google Scholar 

  35. Mora, T., Deny, S., Marre, O.: Dynamical criticality in the collective activity of a population of retinal neurons. Phys. Rev. Lett. 114(7), 078105 (2015). doi:10.1103/PhysRevLett.114.078105

    Article  Google Scholar 

  36. Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. arXiv:1211.5063 (2012)

  37. Peng, Y., Lei, M., Li, J.-B., Peng, X.-Y.: A novel hybridization of echo state networks and multiplicative seasonal ARIMA model for mobile communication traffic series forecasting. Neural Comput. Appl. 24(3–4), 883–890 (2014)

    Article  Google Scholar 

  38. Prokopenko, M., Lizier, J.T., Obst, O., Wang, X.R.: Relating Fisher information to order parameters. Phys. Rev. E 84(4), 041116 (2011). doi:10.1103/PhysRevE.84.041116

    Article  Google Scholar 

  39. Rajan, K., Abbott, L.F., Sompolinsky, H.: Stimulus-dependent suppression of chaos in recurrent neural networks. Phys. Rev. E 82(1), 011903 (2010). doi:10.1103/PhysRevE.82.011903

    Article  Google Scholar 

  40. Reinhart, R.F., Steil, J.J.: Regularization and stability in reservoir networks with output feedback. Neurocomputing 90, 96–105 (2012). doi:10.1016/j.neucom.2012.01.032

    Article  Google Scholar 

  41. Roli, A., Villani, M., Filisetti, A., Serra, R.: Dynamical criticality: overview and open questions. arXiv:1512.05259 (2015)

  42. Rumelhart, D.E., Smolensky, P., McClelland, J.L., Hinton, G.: Sequential thought processes in pdp models. V 2, 3–57 (1986)

    Google Scholar 

  43. Scheffer, M., Bascompte, J., Brock, W.A., Brovkin, V., Carpenter, S.R., Dakos, V., Held, H., Van Nes, E.H., Rietkerk, M., Sugihara, G.: Early-warning signals for critical transitions. Nature 461(7260), 53–59 (2009). doi:10.1038/nature08227

    Article  Google Scholar 

  44. Scheffer, M., Carpenter, S.R., Lenton, T.M., Bascompte, J., Brock, W., Dakos, V., van De Koppel, J., van De Leemput, I.A., Levin, S.A., van Nes, E.H., Pascual, M., Vandermeer, J.: Anticipating critical transitions. Science 338(6105), 344–348 (2012). doi:10.1126/science.1225244

    Article  Google Scholar 

  45. Schiller, U.D., Steil, J.J.: Analyzing the weight dynamics of recurrent learning algorithms. Neurocomputing 63, 5–23 (2005). doi:10.1016/j.neucom.2004.04.006

    Article  Google Scholar 

  46. Shen, Y., Wang, J.: An improved algebraic criterion for global exponential stability of recurrent neural networks with time-varying delays. IEEE Trans. Neural Netw. 19(3), 528–531 (2008). ISSN 1045-9227. doi:10.1109/TNN.2007.911751

  47. Steil, J.J.: Memory in backpropagation-decorrelation o(n) efficient online recurrent learning. In: Duch, W., Kacprzyk, J., Oja, E., Zadrożny, S. (eds.) Artificial Neural Networks: Formal Models and Their Applications-ICANN 2005, pp. 649–654. Springer, Berlin, Heidelberg (2005)

    Google Scholar 

  48. Sussillo, D.: Neural circuits as computational dynamical systems. Curr. Opin. Neurobiol. 25, 156–163 (2014). doi:10.1016/j.conb.2014.01.008

    Article  Google Scholar 

  49. Sussillo, D., Barak, O.: Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural Comput. 25(3), 626–649 (2013). doi:10.1162/NECO_a_00409

    Article  MathSciNet  MATH  Google Scholar 

  50. Tiňo, P., Rodan, A.: Short term memory in input-driven linear dynamical systems. Neurocomputing 112, 58–63 (2013). doi:10.1016/j.neucom.2012.12.041

    Article  Google Scholar 

  51. Tkačik, G., Bialek, W.: Information processing in living systems. Ann. Rev. Condens. Matter Phys. 7(1), 89–117 (2016). doi:10.1146/annurev-conmatphys-031214-014803

    Article  Google Scholar 

  52. Tkačik, G., Mora, T., Marre, O., Amodei, D., Palmer, S.E., Berry, M.J., Bialek, W.: Thermodynamics and signatures of criticality in a network of neurons. Proc. Natl. Acad. Sci. 112(37), 11508–11513 (2015). doi:10.1073/pnas.1514188112

    Article  Google Scholar 

  53. Torres, J.J., Marro, J.: Brain performance versus phase transitions. Sci. Rep. 5 (2015). doi:10.1038/srep12216

  54. Toyoizumi, T., Abbott, L.F.: Beyond the edge of chaos: amplification and temporal integration by recurrent networks in the chaotic regime. Phys. Rev. E 84(5), 051908 (2011). doi:10.1103/PhysRevE.84.051908

    Article  Google Scholar 

  55. Toyoizumi, T., Aihara, K., Amari, S.-I.: Fisher information for spike-based population decoding. Phys. Rev. Lett. 97(9), 098102 (2006). doi:10.1103/PhysRevLett.97.098102

    Article  Google Scholar 

  56. Verstraeten, D., Schrauwen, B.: On the quantification of dynamics in reservoir computing. In: Artificial Neural Networks–ICANN 2009, pp. 985–994. Springer, Berlin (2009). doi:10.1007/978-3-642-04274-4_101

  57. Verstraeten, D., Schrauwen, B., D’Haene, M., Stroobandt, D.: An experimental unification of reservoir computing methods. Neural Netw. 20(3), 391–403 (2007). ISSN 0893-6080. doi:10.1016/j.neunet.2007.04.003. Echo State Networks and Liquid State Machines

  58. Wainrib, G., Touboul, J.: Topological and dynamical complexity of random neural networks. Phys. Rev. Lett. 110, 118101 (2013). doi:10.1103/PhysRevLett.110.118101

    Article  Google Scholar 

  59. Wang, X., Lizier, J., Prokopenko, M.: Fisher information at the edge of chaos in random boolean networks. Artif. Life 17(4), 315–329 (2011). ISSN 1064-5462. doi:10.1162/artl_a_00041

  60. Werbos, P.J.: Backpropagation: past and future. Proc. IEEE Int. Conf. Neural Netw. 1, 343–353 (1988). doi:10.1109/ICNN.1988.23866

    Article  Google Scholar 

  61. Yildiz, I.B., Jaeger, H., Kiebel, S.J.: Re-visiting the echo state property. Neural Netw. 35, 1–9 (2012). doi:10.1016/j.neunet.2012.07.005

    Article  MATH  Google Scholar 

  62. Zegers, P.: Fisher information properties. Entropy 17(7), 4918–4939 (2015). doi:10.3390/e17074918

    Article  MathSciNet  Google Scholar 

  63. Zhang, B., Miller, D.J., Wang, Y.: Nonlinear system modeling with random matrices: echo state networks revisited. IEEE Trans. Neural Netw. Learn. Syst. 23(1), 175–182 (2012). ISSN 2162-237X. doi:10.1109/TNNLS.2011.2178562

  64. Zhang, Y., Wang, J.: Global exponential stability of recurrent neural networks for synthesizing linear feedback control systems via pole assignment. IEEE Trans. Neural Netw. 13(3), 633–644 (2002). ISSN 1045-9227. doi:10.1109/TNN.2002.1000129

Download references

Author information

Authors and Affiliations

  1. Machine Learning Group, Department of Physics and Technology, UiT the Arctic University of Norway, Tromsø, Norway

    Filippo Maria Bianchi

  2. Department of Electronics, Information, and Bioengineering, Politecnico di Milano, Milan, Italy

    Lorenzo Livi & Cesare Alippi

  3. Faculty of Informatics, Università della Svizzera Italiana, Lugano, Switzerland

    Lorenzo Livi & Cesare Alippi

Authors
  1. Filippo Maria Bianchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lorenzo Livi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cesare Alippi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cesare Alippi .

Editor information

Editors and Affiliations

  1. Department of Medicine, Division of Nephrology and Hypertension, University of Louisville, Louisville, Kentucky, USA

    Adam E Gawęda

  2. Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland

    Janusz Kacprzyk

  3. Institute of Computational Intelligence, Department of Mechanical Engineering and Computer Science, Częstochowa University of Technology, Częstochowa, Poland

    Leszek Rutkowski

  4. School of Electrical and Computer Engineering, Oklahoma State University, Stillwater, Oklahoma, USA

    Gary G. Yen

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this chapter

Cite this chapter

Bianchi, F.M., Livi, L., Alippi, C. (2018). On the Interpretation and Characterization of Echo State Networks Dynamics: A Complex Systems Perspective. In: Gawęda, A., Kacprzyk, J., Rutkowski, L., Yen, G. (eds) Advances in Data Analysis with Computational Intelligence Methods. Studies in Computational Intelligence, vol 738. Springer, Cham. https://doi.org/10.1007/978-3-319-67946-4_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-67946-4_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-67945-7

  • Online ISBN: 978-3-319-67946-4

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation