Skip to main content
Springer Nature Link
Log in

SRIL: Selective Regularization for Class-Incremental Learning

  • Conference paper
  • First Online:
Computer Vision – ACCV 2024 (ACCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15479))

Included in the following conference series:

Abstract

Human intelligence gradually accepts new information and accumulates knowledge throughout the lifespan. However, deep learning models suffer from a catastrophic forgetting phenomenon, where they forget previous knowledge when acquiring new information. Class-Incremental Learning aims to create an integrated model that balances plasticity and stability to overcome this challenge. In this paper, we propose a selective regularization method that accepts new knowledge while maintaining previous knowledge. We first introduce an asymmetric feature distillation method for old and new classes inspired by cognitive science, using the gradient of classification and knowledge distillation losses to determine whether to perform pattern completion or pattern separation. We also propose a method to selectively interpolate the weight of the previous model for a balance between stability and plasticity, and we adjust whether to transfer through model confidence to ensure the performance of the previous class and enable exploratory learning. We validate the effectiveness of the proposed method, which surpasses the performance of existing methods through extensive experimental protocols using CIFAR-100, ImageNet-Subset, and ImageNet-Full.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: Learning what (not) to forget. In: Proceedings of the European conference on computer vision (ECCV). pp. 139–154 (2018)

    Google Scholar 

  2. Ashok, A., Joseph, K., Balasubramanian, V.N.: Class-incremental learning with cross-space clustering and controlled transfer. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVII. pp. 105–122. Springer (2022)

    Google Scholar 

  3. Castro, F.M., Marin-Jimenez, M.J., Guil, N., Schmid, C., Alahari, K.: End-to-end incremental learning. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 233–248 (2018), https://arxiv.org/abs/1807.09536

  4. Cha, S., Hsu, H., Hwang, T., Calmon, F.P., Moon, T.: Cpr: classifier-projection regularization for continual learning (2021)

    Google Scholar 

  5. Chaudhry, A., Dokania, P.K., Ajanthan, T., Torr, P.H.: Riemannian walk for incremental learning: Understanding forgetting and intransigence. In: ECCV (2018), https://arxiv.org/abs/1801.10112

  6. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)

    Google Scholar 

  7. Douillard, A., Cord, M., Ollion, C., Robert, T., Valle, E.: Podnet: Pooled outputs distillation for small-tasks incremental learning. In: Proceedings of the IEEE European Conference on Computer Vision (ECCV) (2020), https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123650086.pdf

  8. Du, Y., Czarnecki, W.M., Jayakumar, S.M., Farajtabar, M., Pascanu, R., Lakshminarayanan, B.: Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv:1812.02224 (2018)

  9. Eeckt, S.V., et al.: Weight averaging: A simple yet effective method to overcome catastrophic forgetting in automatic speech recognition. ICASSP (2023)

    Google Scholar 

  10. Goodfellow, I.J., Mirza, M., Xiao, D., Courville, A., Bengio, Y.: An empirical investigation of catastrophic forgetting in gradient-based neural networks. ArXiv e-prints (dec 2013), https://arxiv.org/abs/1312.6211

  11. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)

    Google Scholar 

  12. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

  13. Hou, S., Pan, X., Loy, C.C., Wang, Z., Lin, D.: Learning a unified classifier incrementally via rebalancing. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2019)

    Google Scholar 

  14. Hu, X., Tang, K., Miao, C., Hua, X.S., Zhang, H.: Distilling causal effect of data in class-incremental learning. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. pp. 3957–3966 (2021)

    Google Scholar 

  15. Hung, C.Y., Tu, C.H., Wu, C.E., Chen, C.H., Chan, Y.M., Chen, C.S.: Compacting, picking and growing for unforgetting continual learning. In: Advances in Neural Information Processing Systems. pp. 13647–13657 (2019), https://arxiv.org/abs/1910.06562

  16. Izmailov, P., Wilson, A., Podoprikhin, D., Vetrov, D., Garipov, T.: Averaging weights leads to wider optima and better generalization. In: 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018. pp. 876–885 (2018)

    Google Scholar 

  17. Kang, M., Park, J., Han, B.: Class-incremental learning by knowledge distillation with adaptive feature consolidation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16071–16080 (2022)

    Google Scholar 

  18. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Overcoming catastrophic forgetting in neural networks. Proc. of the national academy of sciences (2017)

    Google Scholar 

  19. Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)

    Google Scholar 

  20. Li, Z., Hoiem, D.: Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence (2017), https://arxiv.org/abs/1606.09282

  21. Liu, Y., Schiele, B., Sun, Q.: Adaptive aggregation networks for class-incremental learning. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. pp. 2544–2553 (2021)

    Google Scholar 

  22. Liu, Y., Schiele, B., Sun, Q.: Rmm: Reinforced memory management for class-incremental learning. Adv. Neural. Inf. Process. Syst. 34, 3478–3490 (2021)

    Google Scholar 

  23. Liu, Y., Su, Y., Liu, A.A., Schiele, B., Sun, Q.: Mnemonics training: Multi-class incremental learning without forgetting. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. pp. 12245–12254 (2020)

    Google Scholar 

  24. Van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9(11) (2008)

    Google Scholar 

  25. McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. In: Psychology of learning and motivation, vol. 24, pp. 109–165. Elsevier (1989)

    Google Scholar 

  26. Mermillod, M., Bugaiska, A., Bonin, P.: The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects (2013)

    Google Scholar 

  27. Movshovitz-Attias, Y., Toshev, A., Leung, T.K., Ioffe, S., Singh, S.: No fuss distance metric learning using proxies. In: Proceedings of the IEEE international conference on computer vision. pp. 360–368 (2017)

    Google Scholar 

  28. O’Reilly, R.C., McClelland, J.L.: Hippocampal conjunctive encoding, storage, and recall: Avoiding a trade-off. Hippocampus 4(6), 661–682 (1994)

    Article  Google Scholar 

  29. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)

    Google Scholar 

  30. Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: Incremental classifier and representation learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2001–2010 (2017), https://arxiv.org/abs/1611.07725

  31. Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive neural networks. ArXiv e-prints (jun 2016), https://arxiv.org/abs/1606.04671

  32. Santoro, A.: Reassessing pattern separation in the dentate gyrus (2013)

    Google Scholar 

  33. Shaheen, K., Hanif, M.A., Hasan, O., Shafique, M.: Continual learning for real-world autonomous systems: Algorithms, challenges and frameworks. Journal of Intelligent & Robotic Systems 105(1), 9 (2022)

    Article  Google Scholar 

  34. Simon, C., Koniusz, P., Harandi, M.: On learning the geodesic path for incremental learning. In: Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. pp. 1591–1600 (2021)

    Google Scholar 

  35. Stojanovski, Z., Roth, K., Akata, Z.: Momentum-based weight interpolation of strong zero-shot models for continual learning. In: NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications (2022)

    Google Scholar 

  36. Verbeke, P., Verguts, T.: Learning to synchronize: How biological agents can couple neural task modules for dealing with the stability-plasticity dilemma. PLoS Comput. Biol. 15(8), e1006604 (2019)

    Article  Google Scholar 

  37. Wortsman, M., Ilharco, G., Gadre, S.Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A.S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al.: Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In: International Conference on Machine Learning. pp. 23965–23998. PMLR (2022)

    Google Scholar 

  38. Wortsman, M., Ilharco, G., Kim, J.W., Li, M., Kornblith, S., Roelofs, R., Lopes, R.G., Hajishirzi, H., Farhadi, A., Namkoong, H., et al.: Robust fine-tuning of zero-shot models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7959–7971 (2022)

    Google Scholar 

  39. Wu, Y., Chen, Y., Wang, L., Ye, Y., Liu, Z., Guo, Y., Fu, Y.: Large scale incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 374–382 (2019), https://arxiv.org/abs/1905.13260

  40. Yoon, J., Yang, E., Lee, J., Hwang, S.: Lifelong learning with dynamically expandable networks. In: International Conference on Learning Representations, ICLR (2018)

    Google Scholar 

  41. Yu, Y.C., Huang, C.P., Chen, J.J., Chang, K.P., Lai, Y.H., Yang, F.E., Wang, Y.C.F.: Select and distill: Selective dual-teacher knowledge transfer for continual learning on vision-language models. arXiv preprint arXiv:2403.09296 (2024)

  42. Zhu, Y., Wang, Y.: Student customized knowledge distillation: Bridging the gap between student and teacher. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5057–5066 (2021)

    Google Scholar 

Download references

Acknowledgments

This work was supported by IITP grant funded by the Korea government (RS-2023-00236245/Dev. of Perception/Planning AI SW for Seamless Autonomous Driving in Adverse Weather/Unstructured Env., IITP-2024-No.RS-2023-00255968/AI Convergence Innovation Human Resources Development, RS-2021-II212068/AI Innovation Hub), and Korea NRF grant (NRF-2022R1A2C1091402). W. Hwang is the corresponding author.

Author information

Authors and Affiliations

  1. Department of Artificial Intelligence, Ajou University, Suwon, Korea

    Jisu Han & Wonjun Hwang

  2. Tech. Innovation Group, KT, Seoul, Korea

    Jaemin Na

Authors
  1. Jisu Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jaemin Na
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wonjun Hwang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wonjun Hwang .

Editor information

Editors and Affiliations

  1. Pohang University of Science and Technology (POSTECH), Pohang, Korea (Republic of)

    Minsu Cho

  2. Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates

    Ivan Laptev

  3. Google, Mountain View, CA, USA

    Du Tran

  4. National University of Singapore, Singapore, Singapore

    Angela Yao

  5. Peking University, Beijing, China

    Hongbin Zha

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Han, J., Na, J., Hwang, W. (2025). SRIL: Selective Regularization for Class-Incremental Learning. In: Cho, M., Laptev, I., Tran, D., Yao, A., Zha, H. (eds) Computer Vision – ACCV 2024. ACCV 2024. Lecture Notes in Computer Science, vol 15479. Springer, Singapore. https://doi.org/10.1007/978-981-96-0966-6_21

Download citation

  • DOI: https://doi.org/10.1007/978-981-96-0966-6_21

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-96-0965-9

  • Online ISBN: 978-981-96-0966-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics