Skip to main content
SpringerLink
Log in

Prototype Representation Expansion in Incremental Learning

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Deep neural networks have made outstanding achievements in many static tasks, however, when faced with incremental scenario, they suffer from catastrophic forgetting since the previous data is usually inaccessible. Stored data and generative models are commonly used for maintaining model performance, but there exist problems of memory utilization and privacy safety. In this paper, a novel non-exemplar based incremental learning model, Prototype Representation Expansion (PRE), which provides a great degree to retain the feature space of old tasks, is proposed. Firstly, prototypes are generated to meet the stability and robustness. The mean value of feature embedding for each class is used as prototype to maintain the model stability. Meanwhile, PRE also selects prototypes according to their responses of classifier by feature disturbance noise injection, and the decision boundary can better be maintained. Secondly, prototypes of various classes are linearly combined to construct the hybrid prototype with mixed labels. Along with prototype augmentation, they are used for incremental training phase. We conduct extensive experiments on two benchmark datasets, CIFAR-100 and ImageNet-Subset. It shows that PRE can be combined with some non-exemplar based methods to significantly improve their ability and achieve comparable performance to exemplar based methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data Availibility

Data openly available in a public repository. The data that support the findings of this study are openly available at: https://image-net.org/ and https://www.cs.toronto.edu/~kriz/cifar.html

References

  1. Aljundi R, Babiloni F, Elhoseiny M, Rohrbach M, Tuytelaars T (2018) Memory aware synapses: Learning what (not) to forget. ECCV, Lect Notes Comput Sci 11207:144–161. https://doi.org/10.1007/978-3-030-01219-9_9

    Article  Google Scholar 

  2. Aljundi, R, Chakravarty P, Tuytelaars T (2017) Expert gate: lifelong learning with a network of experts. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 7120–7129. IEEE Computer Society . https://doi.org/10.1109/CVPR.2017.753

  3. Bang J, Kim H, Yoo Y, Ha JW, Choi J (2021) Rainbow memory: continual learning with a memory of diverse samples. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 8218–8227

  4. Berthelot D, Carlini N, Goodfellow IJ, Papernot N, Oliver A, Raffel C (2019) Mixmatch: a holistic approach to semi-supervised learning. In: Advances in Neural Information Processing Systems, pp 5050–5060 .https://proceedings.neurips.cc/paper/2019/hash/1cd138d0499a68f4bb72bee04bbec2d7-Abstract.html

  5. Castro FM, Marín-Jiménez MJ, Guil N, Schmid C, Alahari K (2018) End-to-end incremental learning. ECCV 11216:241–257. https://doi.org/10.1007/978-3-030-01258-8_15

    Article  Google Scholar 

  6. Chaudhry A, Dokania PK, Ajanthan T, Torr PHS (2018) Riemannian walk for incremental learning: Understanding forgetting and intransigence. In: V. Ferrari, M. Hebert, C. Sminchisescu, Y. Weiss (eds.) ECCV, Lecture Notes in Computer Science, vol. 11215, pp. 556–572. Springer . https://doi.org/10.1007/978-3-030-01252-6_33

  7. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 770–778

  8. Hinton GE, Vinyals O, Dean J (2015) Distilling the knowledge in a neural network. ArXiv abs/1503.02531

  9. Hou S, Pan X, Loy CC, Wang Z, Lin D (2018) Lifelong learning via progressive distillation and retrospection. In: Proceedings of the European Conference on Computer Vision (ECCV)

  10. Hou S, Pan X, Loy CC, Wang Z, Lin D (2019) Learning a unified classifier incrementally via rebalancing. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 831–839

  11. Kemker R, McClure M, Abitino A, Hayes TL, Kanan C (2018) Measuring catastrophic forgetting in neural networks. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pp 3390–3398. AAAI Press . https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16410

  12. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: Y. Bengio, Y. LeCun (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. http://arxiv.org/abs/1412.6980

  13. Kirkpatrick J, Pascanu R, Rabinowitz NC, Veness J, Desjardins G, Rusu AA, Milan K, Quan J, Ramalho T, Grabska-Barwinska A, Hassabis D, Clopath C, Kumaran D, Hadsell R (2017) Overcoming catastrophic forgetting in neural networks. Proc Natl Acad Sci 114:3521–3526

    Article  MathSciNet  MATH  Google Scholar 

  14. Krizhevsky A (2009) Learning multiple layers of features from tiny images

  15. Lee K, Lee K, Shin J, Lee H (2019) Overcoming catastrophic forgetting with unlabeled data in the wild. In: IEEE/CVF International Conference on Computer Vision (ICCV), pp 312–321. IEEE . https://doi.org/10.1109/ICCV.2019.00040

  16. Li Z, Hoiem D (2018) Learning without forgetting. IEEE Trans Pattern Anal Mach Intell 40:2935–2947

    Article  Google Scholar 

  17. Liu Y, Liu A, Su Y, Schiele B, Sun Q (2020) Mnemonics training: multi-class incremental learning without forgetting. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 12242–12251

  18. Liu Y, Parisot S, Slabaugh GG, Jia X, Leonardis A, Tuytelaars T ()2020 More classifiers, less forgetting: a generic multi-classifier paradigm for incremental learning. In: A. Vedaldi, H. Bischof, T. Brox, J. Frahm (eds.) ECCV, Lecture Notes in Computer Science, vol. 12371, pp. 699–716. Springer . https://doi.org/10.1007/978-3-030-58574-7_42

  19. Lomonaco V, Maltoni D (2017) Core50: a new dataset and benchmark for continuous object recognition. In: CoRL, Proceedings of Machine Learning Research, vol. 78, pp. 17–26. PMLR. http://proceedings.mlr.press/v78/lomonaco17a.html

  20. McCloskey M, Cohen N (1989) Catastrophic interference in connectionist networks: the sequential learning problem. Psychol Learn Motiv- Adv Res Theory 24(C):109–165. https://doi.org/10.1016/S0079-7421(08)60536-8

    Article  Google Scholar 

  21. McCloskey M, Cohen N (1989) Catastrophic interference in connectionist networks: the sequential learning problem. Psychol Learn Motiv 24:109–165

    Article  Google Scholar 

  22. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller MA, Fidjeland A, Ostrovski G, Petersen S, Beattie C, Sadik A, Antonoglou I, King H, Kumaran D, Wierstra D, Legg S, Hassabis D (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533. https://doi.org/10.1038/nature14236

    Article  Google Scholar 

  23. Park DS, Chan W, Zhang Y, Chiu C, Zoph B, Cubuk ED, Le QV (2019) Specaugment: a simple data augmentation method for automatic speech recognition. In: Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pp 2613–2617. ISCA . https://doi.org/10.21437/Interspeech.2019-2680

  24. Ratcliff R (1990) Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychol Rev 97(2):285–308

    Article  Google Scholar 

  25. Rebuffi SA, Kolesnikov A, Sperl G, Lampert CH (2017) ICARL: incremental classifier and representation learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 5533–5542

  26. Robins AV (1993) Catastrophic forgetting in neural networks: the role of rehearsal mechanisms. In: Proceedings 1993 The First New Zealand International Two-Stream Conference on Artificial Neural Networks and Expert Systems pp 65–68

  27. Robins AV (1995) Catastrophic forgetting, rehearsal and pseudorehearsal. Connect Sci 7:123–146

    Article  Google Scholar 

  28. Roy D, Panda P, Roy K (2020) Tree-CNN: a hierarchical deep convolutional neural network for incremental learning. Neural Netw 121:148–160. https://doi.org/10.1016/j.neunet.2019.09.010

    Article  Google Scholar 

  29. Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein MS, Berg A, Fei-Fei L (2015) Imagenet large scale visual recognition challenge. Int J Comput Vision 115:211–252

    Article  MathSciNet  Google Scholar 

  30. Rusu AA, Rabinowitz NC, Desjardins G, Soyer H, Kirkpatrick J, Kavukcuoglu K, Pascanu R, Hadsell R (2016) Progressive neural networks. ArXiv abs/1606.04671

  31. Silver D, Huang A, Maddison CJ, Guez A, Sifre L, van den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, Dieleman S, Grewe D, Nham J, Kalchbrenner N, Sutskever I, Lillicrap TP, Leach M, Kavukcuoglu K, Graepel T, Hassabis D (2016) Mastering the game of go with deep neural networks and tree search. Nature 529(7587):484–489. https://doi.org/10.1038/nature16961

    Article  Google Scholar 

  32. van de Ven GM, Tolias A (2019) Three scenarios for continual learning. ArXiv abs/1904.07734

  33. Wei JW, Zou K (2019) Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In: EMNLP-IJCNLP, pp 6381–6387. Association for Computational Linguistics . https://doi.org/10.18653/v1/D19-1670

  34. Wu Y, Chen Y, Wang L, Ye Y, Liu Z, Guo Y, Fu Y (2019) Large scale incremental learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 374–382. Computer Vision Foundation / IEEE . https://doi.org/10.1109/CVPR.2019.00046. http://openaccess.thecvf.com/content_CVPR_2019/html/Wu_Large_Scale_Incremental_Learning_CVPR_2019_paper.html

  35. Yan S, Xie J, He X (2021) Der: Dynamically expandable representation for class incremental learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 3014–3023.

  36. Yoon J, Yang E, Lee J, Hwang SJ (2018) Lifelong learning with dynamically expandable networks. In: International Conference on Learning Representationss. OpenReview.net . https://openreview.net/forum?id=Sk7KsfW0-

  37. Yu L, Twardowski B, Liu X, Herranz L, Wang K, mei Cheng Y, Jui S, van de Weijer J (2020) Semantic drift compensation for class-incremental learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 6980–6989

  38. Zeng G, Chen Y, Cui B, Yu S (2019) Continual learning of context-dependent processing in neural networks. Nat Mach Intell 1(8):364–372. https://doi.org/10.1038/s42256-019-0080-x

    Article  Google Scholar 

  39. Zenke F, Poole B, Ganguli S (2017) Continual learning through synaptic intelligence. Proc Mach Learn Res 70:3987–3995

    Google Scholar 

  40. Zhang H, Cissé M, Dauphin YN, Lopez-Paz D (2018) MIXUP: beyond empirical risk minimization. In: International Conference on Learning Representations. OpenReview.net . https://openreview.net/forum?id=r1Ddp1-Rb

  41. Zhao B, Xiao X, Gan G, Zhang B, Xia S (2020) Maintaining disclrimination and fairness in class incremental learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 13205–13214

  42. Zhu F, Zhang X, Wang C, Yin F, Liu C (2021) Prototype augmentation and self-supervision for incremental learning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 5871–5880. Computer Vision Foundation / IEEE. https://openaccess.thecvf.com/content/CVPR2021/html/Zhu_Prototype_Augmentation_and_Self-Supervision_for_Incremental_Learning_CVPR_2021_paper.html

Download references

Funding

This research is supported by Natural Science Foundation of Liaoning Province, China (No.2022-MS-112).

Author information

Authors and Affiliations

  1. Software College, Northeastern University, Shenyang, China

    Keming Mao, Yong Luo, Yitao Ren & Ruixiang Wang

Authors
  1. Keming Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yong Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yitao Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ruixiang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yitao Ren.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mao, K., Luo, Y., Ren, Y. et al. Prototype Representation Expansion in Incremental Learning. Neural Process Lett 55, 8401–8417 (2023). https://doi.org/10.1007/s11063-023-11317-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11317-x

Keywords

Search

Navigation