skip to main content
10.1145/3503161.3548160acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Semantics-Driven Generative Replay for Few-Shot Class Incremental Learning

Published: 10 October 2022 Publication History

Abstract

We deal with the problem of few-shot class incremental learning (FSCIL), which requires a model to continuously recognize new categories for which limited training data are available. Existing FSCIL methods depend on prior knowledge to regularize the model parameters for combating catastrophic forgetting. Devising an effective prior in a low-data regime, however, is not trivial. The memory-replay based approaches from the fully-supervised class incremental learning (CIL) literature cannot be used directly for FSCIL as the generative memory-replay modules of CIL are hard to train from few training samples. However, generative replay can tackle both the stability and plasticity of the models simultaneously by generating a large number of class-conditional samples. Convinced by this fact, we propose a generative modeling-based FSCIL framework using the paradigm of memory-replay in which a novel conditional few-shot generative adversarial network (GAN) is incrementally trained to produce visual features while ensuring the stability-plasticity trade-off through novel loss functions and combating the mode-collapse problem effectively. Furthermore, the class-specific synthesized visual features from the few-shot GAN are constrained to match the respective latent semantic prototypes obtained from a well-defined semantic space. We find that the advantages of this semantic restriction is two-fold, in dealing with forgetting, while making the features class-discernible. The model requires a single per-class prototype vector to be maintained in a dynamic memory buffer. Experimental results on the benchmark and large-scale CiFAR-100, CUB-200, and Mini-ImageNet confirm the superiority of our model over the current FSCIL state of the art.

Supplementary Material

MP4 File (MM_f1.mp4)
Presentation video

References

[1]
Adamu Ali-Gombe, Eyad Elyan, Yann Savoye, and Chrisina Jayne. 2018. Few-shot Classifier GAN. In 2018 International Joint Conference on Neural Networks (IJCNN). 1--8. https://doi.org/10.1109/IJCNN.2018.8489387
[2]
S Divakar Bhat, Biplab Banerjee, and Subhasis Chaudhuri. 2021. SemGIF: A Semantics Guided Incremental Few-shot Learning Framework with Generative Replay. (2021).
[3]
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics 5 (2017), 135--146.
[4]
Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. 2018. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV). 233--248.
[5]
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. 2018. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV). 532--547.
[6]
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. 2020. Podnet: Pooled outputs distillation for small-tasks incremental learning. In European Conference on Computer Vision. Springer, 86--102.
[7]
Harrison Edwards and Amos Storkey. 2016. Towards a neural statistician. arXiv preprint arXiv:1606.02185 (2016).
[8]
Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, and Frank Hutter. 2020. Meta-learning of neural architectures for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12365--12375.
[9]
Robert M French. 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences 3, 4 (1999), 128--135.
[10]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.
[11]
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. 2019. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 831--839.
[12]
Xinting Hu, Kaihua Tang, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. 2021. Distilling causal effect of data in class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3957--3966.
[13]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[14]
Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009).
[15]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012), 1097--1105.
[16]
Kai Li, Yulun Zhang, Kunpeng Li, and Yun Fu. 2020. Adversarial feature hallucination networks for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13470--13479.
[17]
Zhizhong Li and Derek Hoiem. 2017. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence 40, 12 (2017), 2935--2947.
[18]
Binghao Liu, Jianbin Jiao, and Qixiang Ye. 2021. Harmonic Feature Activation for Few-Shot Semantic Segmentation. IEEE Transactions on Image Processing 30 (2021), 3142--3153.
[19]
Xialei Liu, Chenshen Wu, Mikel Menta, Luis Herranz, Bogdan Raducanu, Andrew D Bagdanov, Shangling Jui, and Joost van de Weijer. 2020. Generative feature replay for class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 226--227.
[20]
Yaoyao Liu, Bernt Schiele, and Qianru Sun. 2021. Adaptive aggregation networks for class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2544--2553.
[21]
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. 2020. Mnemonics training: Multi-class incremental learning without forgetting. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. 12245--12254.
[22]
Arun Mallya and Svetlana Lazebnik. 2018. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 7765--7773.
[23]
Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. 2020. Class-incremental learning: survey and performance evaluation on image classification. arXiv preprint arXiv:2010.15277 (2020).
[24]
Pratik Mazumder, Pravendra Singh, and Piyush Rai. 2021. Few-Shot Lifelong Learning. arXiv preprint arXiv:2103.00991 (2021).
[25]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. 3111--3119.
[26]
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. 2017. Variational continual learning. arXiv preprint arXiv:1710.10628 (2017).
[27]
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1532--1543.
[28]
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2001--2010.
[29]
Scott Reed, Yutian Chen, Thomas Paine, Aäron van den Oord, SM Eslami, Danilo Rezende, Oriol Vinyals, and Nando de Freitas. 2017. Few-shot autoregressive density estimation: Towards learning to learn distributions. arXiv preprint arXiv:1710.10304 (2017).
[30]
Danilo Rezende, Ivo Danihelka, Karol Gregor, Daan Wierstra, et al. 2016. One-shot generalization in deep generative models. In International conference on machine learning. PMLR, 1521--1529.
[31]
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. 2018. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning. PMLR, 4548--4557.
[32]
Guangyuan Shi, Jiaxin Chen, Wenlong Zhang, Li-Ming Zhan, and Xiao-Ming Wu. 2021. Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minima. Advances in Neural Information Processing Systems 34 (2021).
[33]
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. 2017. Continual learning with deep generative replay. arXiv preprint arXiv:1705.08690 (2017).
[34]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[35]
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. 2019. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 403--412.
[36]
Xiaoyu Tao, Xinyuan Chang, Xiaopeng Hong, Xing Wei, and Yihong Gong. 2020. Topology-preserving class-incremental learning. In European Conference on Computer Vision. Springer, 254--270.
[37]
Xiaoyu Tao, Xiaopeng Hong, Xinyuan Chang, Songlin Dong, Xing Wei, and Yihong Gong. 2020. Few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12183--12192.
[38]
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008).
[39]
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016. Matching networks for one shot learning. Advances in neural information processing systems 29 (2016), 3630--3638.
[40]
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. 2011. The caltech-ucsd birds-200--2011 dataset. (2011).
[41]
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. 2019. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 374--382.
[42]
Boyu Yang, Mingbao Lin, Binghao Liu, Mengying Fu, Chang Liu, Rongrong Ji, and Qixiang Ye. 2021. Learnable Expansion-and-Compression Network for Few-shot Class-Incremental Learning. arXiv preprint arXiv:2104.02281 (2021).
[43]
Ruixiang ZHANG, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. 2018. MetaGAN: An Adversarial Approach to Few-Shot Learning. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2018/file/4e4e53aa080247bc31d0eb4e7aeb07a0-Paper.pdf

Cited By

View all
  • (2024)TMM-CLIP: Task-guided Multi-Modal Alignment for Rehearsal-Free Class Incremental LearningProceedings of the 6th ACM International Conference on Multimedia in Asia10.1145/3696409.3700182(1-7)Online publication date: 3-Dec-2024
  • (2024)Advancing Incremental Few-Shot Video Action Recognition with Cluster Compression and Generative SeparationInternational Journal of Pattern Recognition and Artificial Intelligence10.1142/S021800142455013938:14Online publication date: 28-Oct-2024
  • (2024)Few-shot Incremental Identification of Specific Emitter Based on Ancillary Prototypes2024 IEEE/CIC International Conference on Communications in China (ICCC)10.1109/ICCC62479.2024.10681932(1567-1572)Online publication date: 7-Aug-2024
  • Show More Cited By

Index Terms

  1. Semantics-Driven Generative Replay for Few-Shot Class Incremental Learning

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '22: Proceedings of the 30th ACM International Conference on Multimedia
    October 2022
    7537 pages
    ISBN:9781450392037
    DOI:10.1145/3503161
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 10 October 2022

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. few-shot learning
    2. incremental learning
    3. memory replay

    Qualifiers

    • Research-article

    Conference

    MM '22
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)147
    • Downloads (Last 6 weeks)13
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)TMM-CLIP: Task-guided Multi-Modal Alignment for Rehearsal-Free Class Incremental LearningProceedings of the 6th ACM International Conference on Multimedia in Asia10.1145/3696409.3700182(1-7)Online publication date: 3-Dec-2024
    • (2024)Advancing Incremental Few-Shot Video Action Recognition with Cluster Compression and Generative SeparationInternational Journal of Pattern Recognition and Artificial Intelligence10.1142/S021800142455013938:14Online publication date: 28-Oct-2024
    • (2024)Few-shot Incremental Identification of Specific Emitter Based on Ancillary Prototypes2024 IEEE/CIC International Conference on Communications in China (ICCC)10.1109/ICCC62479.2024.10681932(1567-1572)Online publication date: 7-Aug-2024
    • (2024)A survey on few-shot class-incremental learningNeural Networks10.1016/j.neunet.2023.10.039169:C(307-324)Online publication date: 4-Mar-2024
    • (2024)Learning optimal inter-class margin adaptively for few-shot class-incremental learning via neural collapse-based meta-learningInformation Processing & Management10.1016/j.ipm.2024.10366461:3(103664)Online publication date: May-2024
    • (2024)Rethinking few-shot class-incremental learning: A lazy learning baselineExpert Systems with Applications10.1016/j.eswa.2024.123848250(123848)Online publication date: Sep-2024
    • (2024)Few-Shot Class-Incremental Learning via Cross-Modal Alignment with Feature ReplayPattern Recognition and Computer Vision10.1007/978-981-97-8487-5_2(19-33)Online publication date: 4-Nov-2024
    • (2024)Rethinking Few-Shot Class-Incremental Learning: Learning from YourselfComputer Vision – ECCV 202410.1007/978-3-031-73030-6_7(108-128)Online publication date: 24-Nov-2024

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media