Skip to main content
Springer Nature Link
Log in

Joint Relation Modeling and Feature Learning for Class-Incremental Facial Expression Recognition

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14429))

Included in the following conference series:

  • 709 Accesses

Abstract

Due to the diversity of human emotions, it is often difficult to collect all the expression categories at once in many practical applications. In this paper, we investigate facial expression recognition (FER) under the class-incremental learning (CIL) paradigm, where we define easily-accessible basic expressions as an initial task and learn new compound expressions continuously. To this end, we propose a novel joint relation modeling and feature learning (JRF) method, which mainly consists of a local nets module (LNets), a dynamic relation modeling module (DRM), and an adaptive feature learning module (AFL) by taking advantage of the relationship between old and new expressions, effectively alleviating the stability-plasticity dilemma. Specifically, we develop LNets to capture subtle distinctions across expressions, where a novel diversity loss is designed to locate informative facial regions in each local net. Then, we introduce DRM to enhance feature representations based on two types of graph convolutional networks (GCNs) (including an image-shared GCN and two image-specific GCNs) from the perspectives of global-local graphs and old-new classes. Finally, we design AFL to explicitly fuse old and new class features via a weight selection mechanism. Extensive experiments on both in-the-lab and in-the-wild facial expression databases demonstrate the superiority of our method in comparison with several state-of-the-art methods for class-incremental FER.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bang, J., Kim, H., Yoo, Y., Ha, J.W., Choi, J.: Rainbow memory: continual learning with a memory of diverse samples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8218–8227 (2021)

    Google Scholar 

  2. Chang, D., et al.: The devil is in the channels: mutual-channel loss for fine-grained image classification. IEEE Trans. Image Process. 29, 4683–4695 (2020)

    Article  Google Scholar 

  3. Chen, A., Zhou, Y.: An attention enhanced graph convolutional network for semantic segmentation. In: Peng, Y., et al. (eds.) PRCV 2020. LNCS, vol. 12305, pp. 734–745. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60633-6_61

    Chapter  Google Scholar 

  4. Douillard, A., Cord, M., Ollion, C., Robert, T., Valle, E.: PODNet: pooled outputs distillation for small-tasks incremental learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12365, pp. 86–102. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58565-5_6

    Chapter  Google Scholar 

  5. Du, S., Tao, Y., Martinez, A.M.: Compound facial expressions of emotion. Proc. Natl. Acad. Sci. 111(15), E1454–E1462 (2014)

    Article  Google Scholar 

  6. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124–129 (1971)

    Article  Google Scholar 

  7. Fabian Benitez-Quiroz, C., Srinivasan, R., Martinez, A.M.: EmotioNet: an accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5562–5570 (2016)

    Google Scholar 

  8. Farzaneh, A.H., Qi, X.: Facial expression recognition in the wild via deep attentive center loss. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2402–2411 (2021)

    Google Scholar 

  9. Goodfellow, I., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.: Maxout networks. In: International Conference on Machine Learning, pp. 1319–1327 (2013)

    Google Scholar 

  10. Grossberg, S.: Adaptive resonance theory: how a brain learns to consciously attend, learn, and recognize a changing world. Neural Netw. 37, 1–47 (2013)

    Article  Google Scholar 

  11. Kang, M., Park, J., Han, B.: Class-incremental learning by knowledge distillation with adaptive feature consolidation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16071–16080 (2022)

    Google Scholar 

  12. Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)

  13. Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2852–2861 (2017)

    Google Scholar 

  14. Li, X., Wang, W., Hu, X., Yang, J.: Selective kernel networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 510–519 (2019)

    Google Scholar 

  15. Li, X., Deng, W., Li, S., Li, Y.: Compound expression recognition in-the-wild with au-assisted meta multi-task learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5734–5743 (2023)

    Google Scholar 

  16. Liu, Y., et al.: MAFW: a large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild. arXiv preprint arXiv:2208.00847 (2022)

  17. Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: iCaRL: incremental classifier and representation learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2001–2010 (2017)

    Google Scholar 

  18. Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 400–407 (1951)

    Google Scholar 

  19. Song, S., Huang, H., Wang, J., Zheng, A., He, R.: Prior-guided multi-scale fusion transformer for face attribute recognition. In: Yu, S., et al. (eds.) PRCV 2022. LNCS, vol. 13534, pp. 645–659. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-18907-4_50

    Chapter  Google Scholar 

  20. Wang, F.Y., Zhou, D.W., Ye, H.J., Zhan, D.C.: FOSTER: feature boosting and compression for class-incremental learning. arXiv preprint arXiv:2204.04662 (2022)

  21. Wang, K., Peng, X., Yang, J., Lu, S., Qiao, Y.: Suppressing uncertainties for large-scale facial expression recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6897–6906 (2020)

    Google Scholar 

  22. Wang, Q., Guo, G.: LS-CNN: characterizing local patches at multiple scales for face recognition. IEEE Trans. Inf. Forensics Secur. 15, 1640–1653 (2020)

    Article  Google Scholar 

  23. Yan, S., Xie, J., He, X.: DER: dynamically expandable representation for class incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3014–3023 (2021)

    Google Scholar 

  24. Ye, J., He, J., Peng, X., Wu, W., Qiao, Yu.: Attention-driven dynamic graph convolutional network for multi-label image recognition. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12366, pp. 649–665. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58589-1_39

    Chapter  Google Scholar 

  25. Zhang, Z., Yi, M., Xu, J., Zhang, R., Shen, J.: Two-stage recognition and beyond for compound facial emotion recognition. In: Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition, pp. 900–904 (2020)

    Google Scholar 

  26. Zhou, D.W., Wang, F.Y., Ye, H.J., Zhan, D.C.: PyCIL: a python toolbox for class-incremental learning. arXiv preprint arXiv:2112.12533 (2021)

  27. Zhou, D.W., Wang, Q.W., Qi, Z.H., Ye, H.J., Zhan, D.C., Liu, Z.: Deep class-incremental learning: a survey (2023)

    Google Scholar 

  28. Zhou, D.W., Wang, Q.W., Ye, H.J., Zhan, D.C.: A model or 603 exemplars: towards memory-efficient class-incremental learning. In: International Conference on Learning Representations (2023)

    Google Scholar 

  29. Zhou, D.W., Ye, H.J., Zhan, D.C.: Co-transport for class-incremental learning. In: Proceedings of the ACM International Conference on Multimedia, pp. 1645–1654 (2021)

    Google Scholar 

  30. Zhu, J., Luo, B., Zhao, S., Ying, S., Zhao, X., Gao, Y.: IExpressNet: facial expression recognition with incremental classes. In: Proceedings of the ACM International Conference on Multimedia, pp. 2899–2908 (2020)

    Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grants 62372388, 62071404, U21A20514, by the Natural Science Foundation of Fujian Province under Grant 2020J01001, and by the Fuxiaquan National Independent Innovation Demonstration Zone Collaborative Innovation Platform Project under Grant 3502ZCQXT2022008.

Author information

Authors and Affiliations

  1. Fujian Key Laboratory of Sensing and Computing for Smart City, School of Informatics, Xiamen University, Xiamen, China

    Yuanling Lv, Yan Yan & Hanzi Wang

  2. State Key Laboratory of Integrated Services Networks (Xidian University), Xi’an, China

    Yuanling Lv & Yan Yan

Authors
  1. Yuanling Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yan Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hanzi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yan Yan .

Editor information

Editors and Affiliations

  1. Nanjing University of Information Science and Technology, Nanjing, China

    Qingshan Liu

  2. Xiamen University, Xiamen, China

    Hanzi Wang

  3. Beijing University of Posts and Telecommunications, Beijing, China

    Zhanyu Ma

  4. Sun Yat-sen University, Guangzhou, China

    Weishi Zheng

  5. Peking University, Beijing, China

    Hongbin Zha

  6. Chinese Academy of Sciences, Beijing, China

    Xilin Chen

  7. Chinese Academy of Sciences, Beijing, China

    Liang Wang

  8. Xiamen University, Xiamen, China

    Rongrong Ji

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lv, Y., Yan, Y., Wang, H. (2024). Joint Relation Modeling and Feature Learning for Class-Incremental Facial Expression Recognition. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14429. Springer, Singapore. https://doi.org/10.1007/978-981-99-8469-5_11

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8469-5_11

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8468-8

  • Online ISBN: 978-981-99-8469-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics