Skip to main content
SpringerLink
Log in

Incremental learning with a homeostatic self-organizing neural model

  • WSOM 2017
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

We present a new self-organized neural model that we term resilient self-organizing tissue (ReST), which can be run as a convolutional neural network, possesses a \(c^\infty \) energy function as well as a probabilistic interpretation of neural activities. The latter arises from the constraint of lognormal activity distribution over time that is enforced during ReST learning. The principal message of this article is that self-organized models in general are, due to their localized learning rule that updates only those units close to the best-matching unit, ideal representation learners for incremental learning architectures. We present such an architecture that uses ReST layers as a building block, benchmark its performance w.r.t. incremental learning in three real-world visual classification problems, and justify the mechanisms implemented in the architecture by dedicated experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. Available under www.gepperth.net/alexander/data.

References

  1. Gepperth A, Hammer B (2016) Incremental learning algorithms and applications. In: European symposium on artificial neural networks (ESANN), (April), pp 357–368

  2. Vijayakumar Sethu, D’souza Aaron, Schaal Stefan (2005) Incremental online learning in high dimensions. Neural Comput 17(12):2602–2634

    Article  MathSciNet  Google Scholar 

  3. May RJ, Maier HR, Dandy GC (2010) Data splitting for artificial neural networks using som-based stratified sampling. Neural Netw 23(2):283–294

    Article  Google Scholar 

  4. McCloskey M, Cohen N (1989) Catastrophic interference in connectionist networks: the sequential learning problem. In: Bower GH (ed) The psychology of learning and motivation, vol 24. Academic Press, New York

    Google Scholar 

  5. Ratcliff R (1990) Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychol Rev 97:285

    Article  Google Scholar 

  6. French RM (1992) Semi-distributed representations and catastrophic forgetting in connectionist networks. Connect Sci 4:365–377

    Article  Google Scholar 

  7. French RM (1990) Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychol Rev 97(2):285

    Article  Google Scholar 

  8. McCloskey M, Cohen NJ (1989) Catastrophic interference in connectionist networks: the sequential learning problem. Psychol Learn Motiv 24:109–165

    Article  Google Scholar 

  9. Kohonen T (1982) Self-organized formation of topologically correct feature maps. Biol Cybernet 43:59–69

    Article  MathSciNet  Google Scholar 

  10. Gepperth A, Karaoguz C (2015) A bio-inspired incremental learning architecture for applied perceptual problems. Cognit Comput 8:924–934

    Article  Google Scholar 

  11. Gepperth A, Lefort M (2015) Biologically inspired incremental learning for high-dimensional spaces. In: IEEE international conference on development and learning (ICDL)

  12. Kulkarni P, Ade R (2014) Incremental learning from unbalanced data with concept class, concept drift and missing features: a review. Int J Data Min Knowl Manag Process 4(6):15

    Article  Google Scholar 

  13. Tsymbal Alexey (2004) The problem of concept drift: definitions and related work. Technical report. Computer Science Department, Trinity College Dublin

  14. Wen YM, Lu BL (2007) Incremental learning of support vector machines by classifier combining. In: Proceedings of 11th Pacific-Asia conference on knowledge discovery and data mining (PAKDD 2007), vol 4426 of LNCS

  15. Polikar Robi, Upda Lalita, Upda Satish S, Honavar Vasant (2001) Learn++: an incremental learning algorithm for supervised neural networks. IEEE Trans Syst Man Cybern Part C Appl Rev 31(4):497–508

    Article  Google Scholar 

  16. Sharkey N, Sharkey A (1995) An analysis of catastrophic interference. Connect Sci 7(3–4):301–329

    Article  Google Scholar 

  17. French RM (1994) Dynamically constraining connectionist networks to produce distributed, orthogonal representations to reduce catastrophic interference. In: Proceedings of the sixteenth annual conference of the cognitive science society

  18. Murre J (1992) The effects of pattern presentation on interference in backpropagation networks. In: Proceedings of the 14th annual conference of the cognitive science society

  19. Kortge C (1990) Episodic memory in connectionist networks. In: Proceedings of the 12th annual conference of the cognitive science society

  20. Goodfellow IJ, Mirza M, Xiao D, Courville A, Bengio Y (2013) An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211

  21. Krushke J (1992) ALCOVE: An exemplar-based model of category learning. Psychol Rev 99:22

    Article  Google Scholar 

  22. Sloman S, Rumelhart D (1992) Reducing interference in distributed memories through episodic gating. In: Healy A, Kosslynand S, Shiffrin R (eds) Essays in honor of. W K. Estes. Lawrence Erlbaum Associates, Potoma

    Google Scholar 

  23. Parisi GI, Kemker R, Part JL, Kanan C, Wermter S (2018) Continual lifelong learning with neural networks: a review. arXiv preprint arXiv:1802.07569

  24. Serra J, Suris D, Miron Ma, Karatzoglou A (2018) Overcoming catastrophic forgetting with hard attention to the task. In: Proceedings of the 35th international conference on machine learning. PMLR, pp 4548–4557

  25. Kemker R, McClure M, Abitino A, Hayes TL, Kanan C (2018) Measuring catastrophic forgetting in neural networks. In: Thirty-second AAAI conference on artificial intelligence

  26. Pfülb B, Gepperth A (2019) A comprehensive, application-oriented study of catastrophic forgetting in DNNS. In: International conference on learning representations (ICLR) (accepted)

  27. Ren Boya, Wang Hongzhi, Li Jianzhong, Gao Hong (2017) Life-long learning based on dynamic combination model. Appl Soft Comput J 56:398–404

    Article  Google Scholar 

  28. Fernando C, Banarse D, Blundell C, Zwols Y, Ha D, Rusu AA, Pritzel A, Wierstra D (2017) Pathnet: evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734

  29. Shin H, Lee JK, Kim J, Kim J (2017) Continual learning with deep generative replay. In: Advances in neural information processing systems, pp 2990–2999

  30. Kemker R, Kanan C (2017) Fearnet: Brain-inspired model for incremental learning. arXiv preprint arXiv:1711.10563

  31. Rebuffi S-A, Kolesnikov A, Sperl G, Lampert CH (2017) iCARL: incremental classifier and representation learning. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 5533–5542

  32. Aljundi R, Rohrbach M, Tuytelaars T (2018) Selfless sequential learning. arXiv preprint arXiv:1806:05421

  33. Srivastava RK, Masci J, Kazerounian S, Gomez F, Schmidhuber J (2013) Compete to compute. In: Advances in neural information processing systems, pp 2310–2318

  34. Kirkpatrick J, Pascanu R, Rabinowitz N, Veness J, Desjardins G, Rusu AA, Milan K, Quan J, Ramalho Ti, Grabska-Barwinska A et al (2017) Overcoming catastrophic forgetting in neural networks. In: Proceedings of the national academy of sciences, pp 201611835

  35. Lee SW, Kim JH, Jun J, Ha JW, Zhang BT (2017) Overcoming catastrophic forgetting by incremental moment matching. In: Advances in neural information processing systems, pp 4652–4662

  36. Rosenfeld A, Tsotsos JK (2017) Incremental learning through deep adaptation. arXiv preprint arXiv:1705.04228

  37. Shmelkov K, Schmid C, Alahari K (2017) Incremental learning of object detectors without catastrophic forgetting. arXiv preprint arXiv:1708.06977

  38. Rebuffi S-A, Kolesnikov A, Lampert CH (2017) iCaRL: incremental classifier and representation learning. In: Proceedings of CVPR

  39. Kim H-E, Kim S, Lee J (2018) Keep and learn: Continual learning by constraining the latent space for knowledge preservation in neural networks. arXiv preprint arXiv:1805.10784

  40. Vijayakumar S, Schaal S (2000) Locally weighted projection regression: an o(n) algorithm for incremental real time learning in high-dimensional spaces. In: International conference on machine learning

  41. Nguyen-Tuong D, Peters J (2008) Local gaussian processes regression for real-time model-based robot control. In: IEEE/RSJ international conference on intelligent robot systems

  42. Sigaud O, Sagaün C, Padois V (2011) On-line regression algorithms for learning mechanical models of robots: a survey. Robot Auton Syst 59:1115–1129

    Article  Google Scholar 

  43. Butz MV, Goldberg DE, Lanzi PL (2005) Computational complexity of the XCS classifier system. In: Foundations of learning classifier systems. Springer, Berlin, Heidelberg, pp 91–125

  44. Cederborg T, Li M, Baranes A, Oudeyer PY (2010) Incremental local online gaussian mixture regression for imitation learning of multiple tasks. In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, pp 267–274

  45. Turrigiano Gina G, Nelson Sacha B (2004) Homeostatic plasticity in the developing nervous system. Nat Rev Neurosci 5(2):97

    Article  Google Scholar 

  46. Butko Nicholas J, Triesch Jochen (2007) Learning sensory representations with intrinsic plasticity. Neurocomputing 70(7):1130–1138 Advances in Computational Intelligence and Learning

    Article  Google Scholar 

  47. Buzsáki György, Mizuseki Kenji (2014) The log-dynamic brain: how skewed distributions affect network operations. Nat Rev Neurosci 15(4):264

    Article  Google Scholar 

  48. Ioffe Sergey, Szegedy Christian (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning, pp 448–456

  49. Heskes TM, Kappen B (1993) Error potentials for self-organization. In: IEEE International conference on neural networks, 1993. IEEE, pp 1219–1223

  50. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  51. Enzweiler M, Gavrila DM (2009) Monocular pedestrian detection: survey and experiments. IEEE Trans Pattern Anal Mach Intell 31(12):2179–2195

    Article  Google Scholar 

  52. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE computer society conference on computer vision and pattern recognition, 2005. CVPR 2005, vol 1. IEEE, pp 886–893

  53. Lefort M, Hecht T, Gepperth A (2015) Using self-organizing maps for regression: the importance of the output function. In: European symposium on artificial neural networks (ESANN)

  54. Abadi Martín, Barham Paul, Chen Jianmin, Chen Zhifeng, Davis Andy, Dean Jeffrey, Devin Matthieu, Ghemawat Sanjay, Irving Geoffrey, Isard Michael et al (2016) Tensorflow: a system for large-scale machine learning. OSDI 16:265–283

    Google Scholar 

  55. Polani D (2002) Measures for the organization of self-organizing maps. In: Seiffert U, Jain L (eds) Self-organizing neural networks. Physica-Verlag, Heidelberg, Germany, pp 13–44

    Chapter  Google Scholar 

  56. Gepperth A (2018) Catastrophic forgetting: still a problem for deep neural networks. In: IEEE international joint conference on neural networks (IJCNN)

  57. Pfülb B, Gepperth A, Abdullah S, Krawczyk A (2018) Catastrophic forgetting: still a problem for DNNS. In: International conference on artificial neural networks (ICANN)

Download references

Author information

Authors and Affiliations

  1. University of Applied Sciences Fulda, 36037, Fulda, Germany

    Alexander Gepperth

Authors
  1. Alexander Gepperth
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexander Gepperth.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Figures 11, 12 and 13 give a detailed comparison of all neural activities in a ReST layer with and without self-adaptation. This is a complement to the experiments in Sect. 3.2 which was shifted to the appendix for reasons of readability.

Fig. 11
figure 11

Effects of self-adaptation on hidden layer ReST activities for the pose classification task (“poses”). Self-adaptation is enabled in the upper and disabled in the lower diagram. Each diagram includes an activity histogram for every ReST neuron, giving \(10\times 10=100\) histograms. The interpretation of individual is identical to the one in Fig. 5. In particular, the targeted lognormal distribution is superimposed as a solid green line (color figure online)

Full size image
Fig. 12
figure 12

Effects of self-adaptation on hidden layer ReST activities for the pedestrian detection task (“peddet”). Self-adaptation is enabled in the upper and disabled in the lower diagram. Each diagram includes an activity histogram for every ReST neuron, giving \(10\times 10=100\) histograms. The interpretation of individual is identical to the one in Fig. 5. In particular, the targeted lognormal distribution is superimposed as a solid green line (color figure online)

Full size image
Fig. 13
figure 13

Effects of self-adaptation on hidden layer ReST activities for MNIST. Self-adaptation is enabled in the upper and disabled in the lower diagram. Each diagram includes an activity histogram for every ReST neuron, giving \(10\times 10=100\) histograms. The interpretation of individual is identical to the one in Fig. 5. In particular, the targeted lognormal distribution is superimposed as a solid green line (color figure online)

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gepperth, A. Incremental learning with a homeostatic self-organizing neural model. Neural Comput & Applic 32, 18101–18121 (2020). https://doi.org/10.1007/s00521-019-04112-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04112-0

Keywords

Search

Navigation