Skip to main content

Leveraging Low Rank Filters for Efficient and Knowledge-Preserving Lifelong Learning

  • Conference paper
  • First Online:
Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2023)

Abstract

We propose a low rank filter approximation based continual learning approach which decomposes convolution filters into compact basis filters and remixing coefficients. For lifelong learning, we keep the same basis filters to allow knowledge sharing, but add separate coefficients for each new task. Task specific feature maps are computed by a sequence of convolutions, first with shared basis filters and followed by the task specific coefficients. This method enables the model to preserve the previously learned knowledge, thus avoiding the problem of catastrophic forgetting. Additionally, choosing compact basis lets us get away with using a small number of basis filters which enables reduction in FLOPs and number of parameters in the model. To demonstrate efficiency of the proposed approach, we evaluate our model on a variety of datasets and network architectures. With Resnet18 based architecture, we report performance improvement on CIFAR100 with significantly low FLOPs and parameters as compared to other methods. For ImageNet our method achieves comparable performance to other recent methods with reduced FLOPs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society (2016)

    Google Scholar 

  2. Redmon, J., Farhadi, A.: Yolov3: An incremental improvement (2018). https://arxiv.org/abs/1804.02767

  3. Rusu, A.A., et al.: Progressive neural networks, arXiv preprint arXiv:1606.04671 (2016)

  4. Yoon, J., Yang, E., Lee, J., Hwang, S.-J.:Lifelong learning with dynamically expandable networks. In: International Conference on Learning Representations (2018)

    Google Scholar 

  5. Jerfel, G., Grant, E., Griffiths, T., Heller, K.A.: Reconciling meta-learning and continual learning with online mixtures of tasks. Adv. Neural Inform. Process. Syst. 32 (2019)

    Google Scholar 

  6. Verma, V.K., Liang, K.J., Mehta, N., Rai, P., Carin, L.: Efficient feature transformations for discriminative and generative continual learning. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 13860–13870 (2021)

    Google Scholar 

  7. Miao, Z., Wang, Z., Chen, W., Qiu, Q.: Continual learning with filter atom swapping. In: International Conference on Learning Representations (2022)

    Google Scholar 

  8. Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images, Tech. Rep. (2009)

    Google Scholar 

  9. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2009)

    Google Scholar 

  10. Delange, M., et al.: A continual learning survey: Defying forgetting in classification tasks IEEE Trans. Pattern Analy. Mach. Intell. (2021)

    Google Scholar 

  11. Mallya, A., Lazebnik, S.: Packnet: adding multiple tasks to a single network by iterative pruning. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7765–7773 (2018)

    Google Scholar 

  12. Wortsman, M., et al.: Supermasks in superposition. Adv. Neural. Inf. Process. Syst. 33, 15173–15184 (2020)

    Google Scholar 

  13. Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. Advances in Neural Information Processing Syst. 30 (2017)

    Google Scholar 

  14. Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: incremental classifier and representation learning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5533–5542 (2017)

    Google Scholar 

  15. Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T., Wayne, G.: Experience replay for continual learning. Adv. Neural Inform. Process. Syst. 32 (2019)

    Google Scholar 

  16. Yan, S., Xie, J., He, X.: Der: dynamically expandable representation for class incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3014–3023 (2021)

    Google Scholar 

  17. Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 40, 2935–2947 (2018)

    Article  PubMed  Google Scholar 

  18. Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Nat. Acad. Sci. 114(13), 3521–3526 (2017)

    Article  ADS  MathSciNet  PubMed  PubMed Central  MATH  Google Scholar 

  19. Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: learning what (not) to forget. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 144–161. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_9

    Chapter  Google Scholar 

  20. Titsias, M.K., Schwarz, J., de G. Matthews, A.G., Pascanu, R., Teh, Y.W.: Functional regularisation for continual learning with gaussian processes. In: International Conference on Learning Representations (2020)

    Google Scholar 

  21. Jaderberg, M., Vedaldi, A., Zisserman, A.: Speeding up convolutional neural networks with low rank expansions. In: BMVC 2014 - Proceedings of the British Machine Vision Conference 2014. British Machine Vision Association, BMVA (2014)

    Google Scholar 

  22. Denton, E., Zaremba, W., Bruna, J., LeCun, Y., Fergus, R.: Exploiting linear structure within convolutional networks for efficient evaluation. Adv. Neural Inform. Process. Syst. (2014)

    Google Scholar 

  23. Zhang, X., Zou, J., He, K., Sun, J.: Accelerating very deep convolutional networks for classification and detection. IEEE Trans. Pattern Analy. Mach. Intell. (2016)

    Google Scholar 

  24. Qiu, Q., Cheng, X., Calderbank, R., Sapiro, G.: Dcfnet: deep neural network with decomposed convolutional filters. In: 35th International Conference on Machine Learning, ICML 2018 (2018)

    Google Scholar 

  25. Zhang, J., et al.: Class-incremental learning via deep model consolidation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1131–1140 (2020)

    Google Scholar 

  26. Chaudhry, A., Dokania, P.K., Ajanthan, T., Torr, P.H.S.: Riemannian walk for incremental learning: understanding forgetting and intransigence. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 556–572. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_33

    Chapter  Google Scholar 

  27. Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: International Conference on Machine Learning. PMLR, pp. 3987–3995 (2017)

    Google Scholar 

  28. Yu, L., et alSemantic drift compensation for class-incremental learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6980–6989 (2020)

    Google Scholar 

  29. Lee, S.-W., Kim, J.-H., Jun, J., Ha, J.-W., Zhang, B.-T.: Overcoming catastrophic forgetting by incremental moment matching. Adv. Neural Inform. Process. Syst. 30 (2017)

    Google Scholar 

  30. Jung, H., Ju, J., Jung, M., Kim, J.: Less-forgetting learning in deep neural networks, ArXiv, vol. abs/ arXiv: 1607.00122 (2016)

  31. Serra, J., Suris, D., Miron, M., Karatzoglou, A.: Overcoming catastrophic forgetting with hard attention to the task. In: International Conference on Machine Learning. PMLR, pp. 4548–4557 (2018)

    Google Scholar 

  32. Masana, M., Tuytelaars, T., van de Weijer, J.: Ternary feature masks: continual learning without any forgetting, vol. 4(5), p. 6, arXiv preprint arXiv:2001.08714 (2020)

  33. Wu, Y., et al.: Large scale incremental learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 374–382 (2019)

    Google Scholar 

  34. Hou, S., Pan, X., Loy, C.C., Wang, Z., Lin, D.: Learning a unified classifier incrementally via rebalancing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 831–839 (2019)

    Google Scholar 

  35. Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M.: Efficient lifelong learning with a-gem. In: International Conference on Learning Representations (2019)

    Google Scholar 

  36. Chaudhry, A., et al.: On tiny episodic memories in continual learning, arXiv preprint arXiv:1902.10486 (2019)

  37. Chaudhry, A., Khan, N., Dokania, P., Torr, P.: Continual learning in low-rank orthogonal subspaces. Adv. Neural. Inf. Process. Syst. 33, 9900–9911 (2020)

    Google Scholar 

  38. Wang, S., Li, X., Sun, J., Xu, Z.: Training networks in null space of feature covariance for continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 184–193 (2021)

    Google Scholar 

  39. Mehta, N., Liang, K., Verma, V.K., Carin, L.: Continual learning using a bayesian nonparametric dictionary of weight factors. In: International Conference on Artificial Intelligence and Statistics. PMLR, pp. 100–108 (2021)

    Google Scholar 

  40. Hyder, R., Shao, K., Hou, B., Markopoulos, P., Prater-Bennette, A., Asif, M.S.: Incremental task learning with incremental rank updates. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds. Computer Vision – ECCV 2022. ECCV 2022. LNCS, vol. 13683. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20050-2_33

  41. Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. Adv. Neural Inform. Process. syst. 30 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Muhammad Tayyab .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Tayyab, M., Mahalanobis, A. (2025). Leveraging Low Rank Filters for Efficient and Knowledge-Preserving Lifelong Learning. In: Meo, R., Silvestri, F. (eds) Machine Learning and Principles and Practice of Knowledge Discovery in Databases. ECML PKDD 2023. Communications in Computer and Information Science, vol 2136. Springer, Cham. https://doi.org/10.1007/978-3-031-74640-6_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-74640-6_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-74639-0

  • Online ISBN: 978-3-031-74640-6

  • eBook Packages: Artificial Intelligence (R0)

Publish with us

Policies and ethics