Abstract
We propose a low rank filter approximation based continual learning approach which decomposes convolution filters into compact basis filters and remixing coefficients. For lifelong learning, we keep the same basis filters to allow knowledge sharing, but add separate coefficients for each new task. Task specific feature maps are computed by a sequence of convolutions, first with shared basis filters and followed by the task specific coefficients. This method enables the model to preserve the previously learned knowledge, thus avoiding the problem of catastrophic forgetting. Additionally, choosing compact basis lets us get away with using a small number of basis filters which enables reduction in FLOPs and number of parameters in the model. To demonstrate efficiency of the proposed approach, we evaluate our model on a variety of datasets and network architectures. With Resnet18 based architecture, we report performance improvement on CIFAR100 with significantly low FLOPs and parameters as compared to other methods. For ImageNet our method achieves comparable performance to other recent methods with reduced FLOPs.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE Computer Society (2016)
Redmon, J., Farhadi, A.: Yolov3: An incremental improvement (2018). https://arxiv.org/abs/1804.02767
Rusu, A.A., et al.: Progressive neural networks, arXiv preprint arXiv:1606.04671 (2016)
Yoon, J., Yang, E., Lee, J., Hwang, S.-J.:Lifelong learning with dynamically expandable networks. In: International Conference on Learning Representations (2018)
Jerfel, G., Grant, E., Griffiths, T., Heller, K.A.: Reconciling meta-learning and continual learning with online mixtures of tasks. Adv. Neural Inform. Process. Syst. 32 (2019)
Verma, V.K., Liang, K.J., Mehta, N., Rai, P., Carin, L.: Efficient feature transformations for discriminative and generative continual learning. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 13860–13870 (2021)
Miao, Z., Wang, Z., Chen, W., Qiu, Q.: Continual learning with filter atom swapping. In: International Conference on Learning Representations (2022)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images, Tech. Rep. (2009)
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2009)
Delange, M., et al.: A continual learning survey: Defying forgetting in classification tasks IEEE Trans. Pattern Analy. Mach. Intell. (2021)
Mallya, A., Lazebnik, S.: Packnet: adding multiple tasks to a single network by iterative pruning. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7765–7773 (2018)
Wortsman, M., et al.: Supermasks in superposition. Adv. Neural. Inf. Process. Syst. 33, 15173–15184 (2020)
Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. Advances in Neural Information Processing Syst. 30 (2017)
Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: incremental classifier and representation learning. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5533–5542 (2017)
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T., Wayne, G.: Experience replay for continual learning. Adv. Neural Inform. Process. Syst. 32 (2019)
Yan, S., Xie, J., He, X.: Der: dynamically expandable representation for class incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3014–3023 (2021)
Li, Z., Hoiem, D.: Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 40, 2935–2947 (2018)
Kirkpatrick, J., et al.: Overcoming catastrophic forgetting in neural networks. Proc. Nat. Acad. Sci. 114(13), 3521–3526 (2017)
Aljundi, R., Babiloni, F., Elhoseiny, M., Rohrbach, M., Tuytelaars, T.: Memory aware synapses: learning what (not) to forget. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 144–161. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_9
Titsias, M.K., Schwarz, J., de G. Matthews, A.G., Pascanu, R., Teh, Y.W.: Functional regularisation for continual learning with gaussian processes. In: International Conference on Learning Representations (2020)
Jaderberg, M., Vedaldi, A., Zisserman, A.: Speeding up convolutional neural networks with low rank expansions. In: BMVC 2014 - Proceedings of the British Machine Vision Conference 2014. British Machine Vision Association, BMVA (2014)
Denton, E., Zaremba, W., Bruna, J., LeCun, Y., Fergus, R.: Exploiting linear structure within convolutional networks for efficient evaluation. Adv. Neural Inform. Process. Syst. (2014)
Zhang, X., Zou, J., He, K., Sun, J.: Accelerating very deep convolutional networks for classification and detection. IEEE Trans. Pattern Analy. Mach. Intell. (2016)
Qiu, Q., Cheng, X., Calderbank, R., Sapiro, G.: Dcfnet: deep neural network with decomposed convolutional filters. In: 35th International Conference on Machine Learning, ICML 2018 (2018)
Zhang, J., et al.: Class-incremental learning via deep model consolidation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1131–1140 (2020)
Chaudhry, A., Dokania, P.K., Ajanthan, T., Torr, P.H.S.: Riemannian walk for incremental learning: understanding forgetting and intransigence. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11215, pp. 556–572. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_33
Zenke, F., Poole, B., Ganguli, S.: Continual learning through synaptic intelligence. In: International Conference on Machine Learning. PMLR, pp. 3987–3995 (2017)
Yu, L., et alSemantic drift compensation for class-incremental learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6980–6989 (2020)
Lee, S.-W., Kim, J.-H., Jun, J., Ha, J.-W., Zhang, B.-T.: Overcoming catastrophic forgetting by incremental moment matching. Adv. Neural Inform. Process. Syst. 30 (2017)
Jung, H., Ju, J., Jung, M., Kim, J.: Less-forgetting learning in deep neural networks, ArXiv, vol. abs/ arXiv: 1607.00122 (2016)
Serra, J., Suris, D., Miron, M., Karatzoglou, A.: Overcoming catastrophic forgetting with hard attention to the task. In: International Conference on Machine Learning. PMLR, pp. 4548–4557 (2018)
Masana, M., Tuytelaars, T., van de Weijer, J.: Ternary feature masks: continual learning without any forgetting, vol. 4(5), p. 6, arXiv preprint arXiv:2001.08714 (2020)
Wu, Y., et al.: Large scale incremental learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 374–382 (2019)
Hou, S., Pan, X., Loy, C.C., Wang, Z., Lin, D.: Learning a unified classifier incrementally via rebalancing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 831–839 (2019)
Chaudhry, A., Ranzato, M., Rohrbach, M., Elhoseiny, M.: Efficient lifelong learning with a-gem. In: International Conference on Learning Representations (2019)
Chaudhry, A., et al.: On tiny episodic memories in continual learning, arXiv preprint arXiv:1902.10486 (2019)
Chaudhry, A., Khan, N., Dokania, P., Torr, P.: Continual learning in low-rank orthogonal subspaces. Adv. Neural. Inf. Process. Syst. 33, 9900–9911 (2020)
Wang, S., Li, X., Sun, J., Xu, Z.: Training networks in null space of feature covariance for continual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 184–193 (2021)
Mehta, N., Liang, K., Verma, V.K., Carin, L.: Continual learning using a bayesian nonparametric dictionary of weight factors. In: International Conference on Artificial Intelligence and Statistics. PMLR, pp. 100–108 (2021)
Hyder, R., Shao, K., Hou, B., Markopoulos, P., Prater-Bennette, A., Asif, M.S.: Incremental task learning with incremental rank updates. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds. Computer Vision – ECCV 2022. ECCV 2022. LNCS, vol. 13683. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20050-2_33
Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. Adv. Neural Inform. Process. syst. 30 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Tayyab, M., Mahalanobis, A. (2025). Leveraging Low Rank Filters for Efficient and Knowledge-Preserving Lifelong Learning. In: Meo, R., Silvestri, F. (eds) Machine Learning and Principles and Practice of Knowledge Discovery in Databases. ECML PKDD 2023. Communications in Computer and Information Science, vol 2136. Springer, Cham. https://doi.org/10.1007/978-3-031-74640-6_17
Download citation
DOI: https://doi.org/10.1007/978-3-031-74640-6_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-74639-0
Online ISBN: 978-3-031-74640-6
eBook Packages: Artificial Intelligence (R0)