Skip to main content
SpringerLink
Log in

Self-organized Reservoirs and Their Hierarchies

  • Conference paper
Artificial Neural Networks and Machine Learning – ICANN 2012 (ICANN 2012)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 7552))

Included in the following conference series:

Abstract

We investigate how unsupervised training of recurrent neural networks (RNNs) and their deep hierarchies can benefit a supervised task like temporal pattern detection. The RNNs are fully and fast trained by unsupervised algorithms and only supervised feed-forward readouts are used. The unsupervised RNNs are shown to perform better in a rigorous comparison against state-of-art random reservoir networks. Unsupervised greedy bottom-up trained hierarchies of such RNNs are shown being capable of big performance improvements over single layer setups.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Chappell, G.J., Taylor, J.G.: The temporal Kohonen map. Neural Networks 6(3), 441–445 (1993)

    Article  Google Scholar 

  2. Erhan, D., Bengio, Y., Courville, A.C., Manzagol, P.-A., Vincent, P., Bengio, S.: Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research 11, 625–660 (2010)

    MathSciNet  MATH  Google Scholar 

  3. Farkaš, I., Crocker, M.W.: Systematicity in sentence processing with a recursive self-organizing neural network. In: Proceedings of ESANN 2007, pp. 49–54 (2007)

    Google Scholar 

  4. Hammer, B., Micheli, A., Sperduti, A., Strickert, M.: Recursive self-organizing network models. Neural Networks 17(8-9), 1061–1085 (2004)

    Article  MATH  Google Scholar 

  5. Jaeger, H.: Echo state network. Scholarpedia 2(9), 2330 (2007)

    Article  Google Scholar 

  6. Jaeger, H., Lukoševičius, M., Popovici, D., Siewert, U.: Optimization and applications of echo state networks with leaky-integrator neurons. Neural Networks 20(3), 335–352 (2007)

    Article  MATH  Google Scholar 

  7. Jarrett, K., Kavukcuoglu, K., Ranzato, M., LeCun, Y.: What is the best multi-stage architecture for object recognition? In: Proceedings of ICCV 2009. IEEE (2009)

    Google Scholar 

  8. Kohonen, T., Honkela, T.: Kohonen network. Scholarpedia 2(1), 1568 (2007)

    Article  Google Scholar 

  9. Koutník, J.: Inductive modelling of temporal sequences by means of self-organization. In: Proceeding of IWIM 2007, pp. 269–277. CTU in Prague (2007)

    Google Scholar 

  10. Lazar, A., Pipa, G., Triesch, J.: SORN: a self-organizing recurrent neural network. Frontiers in Computational Neuroscience 4, 12 (2010)

    Google Scholar 

  11. LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.: Backpropagation applied to handwritten zip code recognition. Neural Computation 1(4), 541–551 (1989)

    Article  Google Scholar 

  12. Lukoševičius, M.: Reservoir Computing and Self-Organized Neural Hierarchies. PhD thesis, Jacobs University Bremen, Bremen, Germany (2011)

    Google Scholar 

  13. Lukoševičius, M., Jaeger, H.: Reservoir computing approaches to recurrent neural network training. Computer Science Review 3(3), 127–149 (2009)

    Article  Google Scholar 

  14. Mak, M.W.: A learning algorithm for recurrent radial basis function networks. Neural Processing Letters 2(1), 27–31 (1995)

    Article  MathSciNet  Google Scholar 

  15. Martinetz, T., Schulten, K.: A “neural-gas” network learns topologies. Artificial Neural Networks 1, 397–402 (1991)

    Google Scholar 

  16. Rahimi, A., Recht, B.: Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In: Proceedings of NIPS 2008, pp. 1313–1320. MIT Press, Cambridge (2009)

    Google Scholar 

  17. Saxe, A., Koh, P., Chen, Z., Bhand, M., Suresh, B., Ng, A.Y.: On random weights and unsupervised feature learning. In: Proceedings of ICML 2011 (2011)

    Google Scholar 

  18. Tiňo, P., Farkaš, I., van Mourik, J.: Dynamics and topographic organization of recursive self-organizing maps. Neural Computation 18(10), 2529–2567 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  19. Varsta, M., Millán, J.D.R., Heikkonen, J.: A Recurrent Self-Organizing Map for Temporal Sequence Processing. In: Gerstner, W., Hasler, M., Germond, A., Nicoud, J.-D. (eds.) ICANN 1997. LNCS, vol. 1327, pp. 421–426. Springer, Heidelberg (1997)

    Google Scholar 

  20. Voegtlin, T.: Recursive self-organizing maps. Neural Networks 15(8-9), 979–991 (2002)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Jacobs University Bremen, Campus Ring 1, 28759, Bremen, Germany

    Mantas Lukoševičius

Authors
  1. Mantas Lukoševičius
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Neuro Heuristic Research Group, University of Lausanne, 1015, Lausanne, Switzerland

    Alessandro E. P. Villa

  2. Department of Informatics, Nicolaus Copernicus University, 87-100, Toruń, Poland

    Włodzisław Duch

  3. Center for Complex Systems Studies, Kalamazoo College, 49006, Kalamazoo, MI, USA

    Péter Érdi

  4. Dipartimento di Informatica e Scienze dell’Informazione, Università di Genova, 16146, Genoa, Italy

    Francesco Masulli

  5. Institut für Neuroinformatik, Universität Ulm, 89069, Ulm, Germany

    Günther Palm

Rights and permissions

Reprints and permissions

Copyright information

© 2012 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Lukoševičius, M. (2012). Self-organized Reservoirs and Their Hierarchies. In: Villa, A.E.P., Duch, W., Érdi, P., Masulli, F., Palm, G. (eds) Artificial Neural Networks and Machine Learning – ICANN 2012. ICANN 2012. Lecture Notes in Computer Science, vol 7552. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33269-2_74

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-33269-2_74

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-33268-5

  • Online ISBN: 978-3-642-33269-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation