Skip to main content

Advertisement

Springer Nature Link
Log in

Incremental learning without looking back: a neural connection relocation approach

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Nowadays, artificial intelligence methods need to face more and more open application scenarios. They need to have the ability to continuously develop new skills and knowledge to respond to changes over time. However, how the learning system learns new tasks without affecting performance on old tasks remains a big challenge. In this work, we develop a learning system based on convolutional neural network (CNN) to implement the incremental learning mode for image classification tasks. Inspired by the way human learns, which includes abstracting learning experiences, keeping only key information in mind and forgetting trivial details, our proposed method contains a neural connection relocation mechanism to remove unimportant information from learned memory. And a mechanism composed of knowledge distillation and fine-tuning is also included to consolidate the learned knowledge using associations with the new task. To demonstrate the performance of our method, two pairs of image classification tasks are conducted with different CNN architectures. The experimental results show that our method performs better than the state of the art incremental learning methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The data that support the findings of this study are derived from the following public domain resources. 1. CIFAR100 can be downloaded from (http://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz). 2. CIFAR100 can be downloaded from (http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz). 3. MNIST can be downloaded from (http://yann.lecun.com/exdb/mnist). 4. Fashion MNIST can be downloaded from (https://github.com/zalandoresearch/fashion-mnist).

References

  1. McCloskey M, Cohen NJ (1989) Catastrophic interference in connectionist networks: the sequential learning problem. Psychol Learn Motiv 24:109–165

    Article  Google Scholar 

  2. Chen P, Wei W, Hsieh C, Dai B (2021) Overcoming catastrophic forgetting by Bayesian generative regularization. In: proceedings of the international conference on machine learning pp 1760– 1770

  3. Szadkowski R, Drchal J, Faigl J (2022) Continually trained life-long classification. Neural Comput Appl 34(1):135–152

    Article  Google Scholar 

  4. Chklovskii D, Mel B, Svoboda K (2004) Cortical rewiring and information storage. Nature 431(7010):782–788

    Article  Google Scholar 

  5. Rewiring the connectome (2018) Bennett, S.H., Kirby, A.J., Finnerty, G.T. Evidence and effects. Neuroscience &Biobehavioral Reviews 88:51–62

    Google Scholar 

  6. Smyth B, Keane MT (1995) Remembering to forget. In: proceedings of the international joint conference on artificial intelligence pp 377– 382

  7. Li Z, Hoiem D (2017) Learning without forgetting. IEEE Trans Pattern Anal Mach Intell 40(12):2935–2947

    Article  Google Scholar 

  8. Zhang Y, Ying S, Wen Z (2022) Multitask transfer learning with kernel representation. Neural Comput Appl 34(15):12709–12721

    Article  Google Scholar 

  9. Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359

    Article  Google Scholar 

  10. Zhou K, Yang Y, Hospedales T, Xiang T (2020) Deep domain-adversarial image generation for domain generalisation. In: proceedings of the AAAI conference on artificial intelligence, pp 13025– 13032

  11. Jiang Z, Liu C, Lee YM, Hegde C, Sarkar S, Jiang D (2022) The stochastic augmented lagrangian method for domain adaptation. Knowl-Based Syst 235:107593

    Article  Google Scholar 

  12. Hsu H, Yao C, Tsai YH, Hung WC, Tseng HY, Singh M, Yang M (2020) Progressive domain adaptation for object detection. In: proceedings of the IEEE winter conference on applications of computer vision pp 749– 757

  13. Chen Y, Lin Y, Yang M, Huang J (2019) Crdoco: Pixel-level domain transfer with cross-domain consistency. In: proceedings of the IEEE Conference on computer vision and pattern recognition pp 1791– 1800

  14. Liang J, Hu D, Feng J (2021) Domain adaptation with auxiliary target domain-oriented classifier. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 16632– 16642

  15. Gepperth, A (2022) Incremental learning with a homeostatic self-organizing neural model. Neural Comput Appl 18101–18121

  16. Belouadah E, Popescu A, Kanellos I (2021) A comprehensive study of class incremental learning algorithms for visual tasks. Neural Netw 135:38–54

    Article  Google Scholar 

  17. Rannen A, Aljundi R, Blaschko MB, Tuytelaars T (2017) Encoder based lifelong learning. In: proceedings of the IEEE international conference on computer vision pp 1320– 1328

  18. Kirkpatrick J, Pascanu R, Rabinowitz N, Veness J, Desjardins G, Rusu AA, Milan K, Quan J, Ramalho T, Grabska Barwinska A et al (2017) Overcoming catastrophic forgetting in neural networks. Proc Natl Acad Sci 114(13):3521–3526

    Article  MathSciNet  MATH  Google Scholar 

  19. Dhar P, Singh RV, Peng K, Wu Z, Chellappa R (2019) Learning without memorizing. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 5138– 5146

  20. Shi F, Wang P, Shi Z, Rui Y (2021). Selecting useful knowledge from previous tasks for future learning in a single network. In: international conference on pattern recognition pp 9727– 9732 . IEEE

  21. Mallya A, Lazebnik S (2018) Packnet: Adding multiple tasks to a single network by iterative pruning. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 7765– 7773

  22. Hou S, Pan X, Loy CC, Wang Z, Lin D (2019) Learning a unified classifier incrementally via rebalancing. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 831– 839

  23. Zhang J, Zhang J, Ghosh S, Li D, Tasci S, Heck L, Zhang H, Kuo C-CJ (2020) Class-incremental learning via deep model consolidation. In: proceedings of the IEEE winter conference on applications of computer vision pp 1131– 1140

  24. Rebuffi SA, Kolesnikov A, Sperl G, Lampert CH (2017) icarl: Incremental classifier and representation learning. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 2001– 2010

  25. Rolnick D, Ahuja A, Schwarz J, Lillicrap T, Wayne G (2019) Experience replay for continual learning. Adv Neural Inform Process Syst 32:1–11

    Google Scholar 

  26. Castro FM, Marín Jiménez MJ, Guil N, Schmid C, Alahari K (2018) End-to-end incremental learning. In: proceedings of the European conference on computer vision pp 233– 248

  27. Wu Y, Chen Y, Wang L, Ye Y, Liu Z, Guo Y, Fu Y (2019) Large scale incremental learning. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 374– 382

  28. Xiang Y, Fu Y, Ji P, Huang H (2019) Incremental learning using conditional adversarial networks. In: proceedings of the IEEE international conference on computer vision pp 6619– 6628

  29. Li H, Kadav A, Durdanovic I, Samet H, Graf HP (2017) Pruning filters for efficient convnets. In: proceedings of the international conference on learning representations pp 1– 13

  30. Molchanov P, Tyree S, Karras T, Aila T, Kautz J (2016) Pruning convolutional neural networks for resource efficient inference. In: proceedings of the international conference on learning representations pp 1– 17

  31. Figurnov M, Ibraimova A, Vetrov DP, Kohli P (2016) Perforatedcnns: Acceleration through elimination of redundant convolutions. In: advances in neural information processing systems pp 947– 955

  32. Lee N, Ajanthan T, Torr P (2019) Snip: Single-shot network pruning based on connection sensitivity. In: proceedings of the international conference on learning representations pp 1– 15

  33. Chang J, Lu Y, Xue P, Xu Y, Wei Z (2022) Global balanced iterative pruning for efficient convolutional neural networks. Neural Comput Appl 34(23):1–20

    Article  Google Scholar 

  34. Chen, H., Wang, Y., Xu, C., Yang, Z., Liu, C., Shi, B., Xu, C., Xu, C., Tian, Q.: Data-free learning of student networks. In: proceedings of the IEEE international conference on computer vision pp 3514– 3522 (2019)

  35. Xiao H, Rasul K, Vollgraf R (2017) Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747

  36. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  37. Krizhevsky, A (2009) Learning multiple layers of features from tiny images. Technical report, University of Toronto

  38. Krizhevsky A, Sutskever I, Hinton GE (2012): Imagenet classification with deep convolutional neural networks. In: advances in neural information processing systems pp 1097– 1105

  39. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. In: proceedings of the international conference on learning representations pp 1– 14

  40. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M et al (2015) Imagenet large scale visual recognition challenge 115:211–252

    Google Scholar 

  41. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 2921– 2929

  42. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: proceedings of the IEEE conference on computer vision and pattern recognition pp 2818– 2826

Download references

Funding

The work was partially supported by the National Natural Science Foundation of China [No. 61702153, 6210392] and Postdoctoral Research Foundation of China [No. 2021M691597]. We would like to express our sincere gratitude to those who have provided us with assistance.

Author information

Authors and Affiliations

  1. School of Automation, Nanjing University of Science and Technology, Nanjing, 210094, Jiangsu, China

    Yi Liu, Xiang Wu & Yuming Bo

  2. Department of Computer Science and Engineering, Michigan State University, East Lansing, MI, 48824, USA

    Zejia Zheng

  3. School of Automobile and Traffic Engineering, Jiangsu University of Technology, Changzhou, 213001, Jiangsu, China

    Mingfeng Yin

Authors
  1. Yi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuming Bo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zejia Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mingfeng Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zejia Zheng.

Ethics declarations

Conflict of interest

As far as the publication of this paper is concerned, the authors declare that there are no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, Y., Wu, X., Bo, Y. et al. Incremental learning without looking back: a neural connection relocation approach. Neural Comput & Applic 35, 14093–14107 (2023). https://doi.org/10.1007/s00521-023-08448-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08448-6

Keywords