Abstract
Due to the diversity of human emotions, it is often difficult to collect all the expression categories at once in many practical applications. In this paper, we investigate facial expression recognition (FER) under the class-incremental learning (CIL) paradigm, where we define easily-accessible basic expressions as an initial task and learn new compound expressions continuously. To this end, we propose a novel joint relation modeling and feature learning (JRF) method, which mainly consists of a local nets module (LNets), a dynamic relation modeling module (DRM), and an adaptive feature learning module (AFL) by taking advantage of the relationship between old and new expressions, effectively alleviating the stability-plasticity dilemma. Specifically, we develop LNets to capture subtle distinctions across expressions, where a novel diversity loss is designed to locate informative facial regions in each local net. Then, we introduce DRM to enhance feature representations based on two types of graph convolutional networks (GCNs) (including an image-shared GCN and two image-specific GCNs) from the perspectives of global-local graphs and old-new classes. Finally, we design AFL to explicitly fuse old and new class features via a weight selection mechanism. Extensive experiments on both in-the-lab and in-the-wild facial expression databases demonstrate the superiority of our method in comparison with several state-of-the-art methods for class-incremental FER.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bang, J., Kim, H., Yoo, Y., Ha, J.W., Choi, J.: Rainbow memory: continual learning with a memory of diverse samples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8218–8227 (2021)
Chang, D., et al.: The devil is in the channels: mutual-channel loss for fine-grained image classification. IEEE Trans. Image Process. 29, 4683–4695 (2020)
Chen, A., Zhou, Y.: An attention enhanced graph convolutional network for semantic segmentation. In: Peng, Y., et al. (eds.) PRCV 2020. LNCS, vol. 12305, pp. 734–745. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60633-6_61
Douillard, A., Cord, M., Ollion, C., Robert, T., Valle, E.: PODNet: pooled outputs distillation for small-tasks incremental learning. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12365, pp. 86–102. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58565-5_6
Du, S., Tao, Y., Martinez, A.M.: Compound facial expressions of emotion. Proc. Natl. Acad. Sci. 111(15), E1454–E1462 (2014)
Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124–129 (1971)
Fabian Benitez-Quiroz, C., Srinivasan, R., Martinez, A.M.: EmotioNet: an accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5562–5570 (2016)
Farzaneh, A.H., Qi, X.: Facial expression recognition in the wild via deep attentive center loss. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2402–2411 (2021)
Goodfellow, I., Warde-Farley, D., Mirza, M., Courville, A., Bengio, Y.: Maxout networks. In: International Conference on Machine Learning, pp. 1319–1327 (2013)
Grossberg, S.: Adaptive resonance theory: how a brain learns to consciously attend, learn, and recognize a changing world. Neural Netw. 37, 1–47 (2013)
Kang, M., Park, J., Han, B.: Class-incremental learning by knowledge distillation with adaptive feature consolidation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16071–16080 (2022)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2852–2861 (2017)
Li, X., Wang, W., Hu, X., Yang, J.: Selective kernel networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 510–519 (2019)
Li, X., Deng, W., Li, S., Li, Y.: Compound expression recognition in-the-wild with au-assisted meta multi-task learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5734–5743 (2023)
Liu, Y., et al.: MAFW: a large-scale, multi-modal, compound affective database for dynamic facial expression recognition in the wild. arXiv preprint arXiv:2208.00847 (2022)
Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: iCaRL: incremental classifier and representation learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2001–2010 (2017)
Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat. 400–407 (1951)
Song, S., Huang, H., Wang, J., Zheng, A., He, R.: Prior-guided multi-scale fusion transformer for face attribute recognition. In: Yu, S., et al. (eds.) PRCV 2022. LNCS, vol. 13534, pp. 645–659. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-18907-4_50
Wang, F.Y., Zhou, D.W., Ye, H.J., Zhan, D.C.: FOSTER: feature boosting and compression for class-incremental learning. arXiv preprint arXiv:2204.04662 (2022)
Wang, K., Peng, X., Yang, J., Lu, S., Qiao, Y.: Suppressing uncertainties for large-scale facial expression recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6897–6906 (2020)
Wang, Q., Guo, G.: LS-CNN: characterizing local patches at multiple scales for face recognition. IEEE Trans. Inf. Forensics Secur. 15, 1640–1653 (2020)
Yan, S., Xie, J., He, X.: DER: dynamically expandable representation for class incremental learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3014–3023 (2021)
Ye, J., He, J., Peng, X., Wu, W., Qiao, Yu.: Attention-driven dynamic graph convolutional network for multi-label image recognition. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12366, pp. 649–665. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58589-1_39
Zhang, Z., Yi, M., Xu, J., Zhang, R., Shen, J.: Two-stage recognition and beyond for compound facial emotion recognition. In: Proceedings of the IEEE International Conference on Automatic Face & Gesture Recognition, pp. 900–904 (2020)
Zhou, D.W., Wang, F.Y., Ye, H.J., Zhan, D.C.: PyCIL: a python toolbox for class-incremental learning. arXiv preprint arXiv:2112.12533 (2021)
Zhou, D.W., Wang, Q.W., Qi, Z.H., Ye, H.J., Zhan, D.C., Liu, Z.: Deep class-incremental learning: a survey (2023)
Zhou, D.W., Wang, Q.W., Ye, H.J., Zhan, D.C.: A model or 603 exemplars: towards memory-efficient class-incremental learning. In: International Conference on Learning Representations (2023)
Zhou, D.W., Ye, H.J., Zhan, D.C.: Co-transport for class-incremental learning. In: Proceedings of the ACM International Conference on Multimedia, pp. 1645–1654 (2021)
Zhu, J., Luo, B., Zhao, S., Ying, S., Zhao, X., Gao, Y.: IExpressNet: facial expression recognition with incremental classes. In: Proceedings of the ACM International Conference on Multimedia, pp. 2899–2908 (2020)
Acknowledgements
This work was supported by the National Natural Science Foundation of China under Grants 62372388, 62071404, U21A20514, by the Natural Science Foundation of Fujian Province under Grant 2020J01001, and by the Fuxiaquan National Independent Innovation Demonstration Zone Collaborative Innovation Platform Project under Grant 3502ZCQXT2022008.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Lv, Y., Yan, Y., Wang, H. (2024). Joint Relation Modeling and Feature Learning for Class-Incremental Facial Expression Recognition. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14429. Springer, Singapore. https://doi.org/10.1007/978-981-99-8469-5_11
Download citation
DOI: https://doi.org/10.1007/978-981-99-8469-5_11
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8468-8
Online ISBN: 978-981-99-8469-5
eBook Packages: Computer ScienceComputer Science (R0)