research-article

Lightweight Facial Expression Recognition Network with Dynamic Deep Mutual Learning

Authors:

Guangyuan LiuAuthors Info & Claims

BIC '23: Proceedings of the 2023 3rd International Conference on Bioinformatics and Intelligent Computing

Pages 222 - 226

https://doi.org/10.1145/3592686.3592726

Published: 31 May 2023 Publication History

Abstract

Existing CNN-based Facial Expression Recognition (FER) networks perform well in terms of performance, but these large networks often have a large number of parameters and high computational complexity, resulting in low efficiency in practical applications. To enable facial expression recognition models to run efficiently in low computing power platforms such as mobile devices. We build a lightweight Facial Expression Recognition (LWFER) network based on GhostNet and apply a lightweight attention module to the network, which makes the network focus more on important features while adding little computational cost, and finally the whole LWFER network has only about 0.8 million parameters and about 0.9 million multiply-add operations (MAdds). And a dynamic deep mutual learning approach is used to train our network to further improve the recognition accuracy. We conducted experiments on the RAF-DB, AffectNet-7 and AffectNet-8 datasets and achieved comparable results on the RAF-DB dataset and the best results on the AffectNet-7 and AffectNet-8 datasets compared to the state-of-the-art lightweight approach. The proposed network is deployed to Android devices with Snapdragon 865 chips and the latency of inferring an expression image of 224 × 224 size is 17 millisecond (ms), demonstrating that our approach strikes a good balance between performance and efficiency. It can be used in fields such as driver sentiment recognition, and in the future, in fields such as user preference prediction and advertising recommendations.

References

[1]

Kai Wang, Xiaojiang Peng, Jianfei Yang, Shijian Lu and Yu Qiao. 2020. Suppressing Uncertainties for Large-Scale Facial Expression Recognition. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20). IEEE, Los Alamitos, CA, 6896-6905. https://doi.org/10.1109/CVPR42600.2020.00693

[2]

Jiahui She, Yibo Hu, Hailin Shi, Jun Wang, Qiu Shen and Tao Mei. 2021. Dive into Ambiguity: Latent Distribution Mining and Pairwise Uncertainty Estimation for Facial Expression Recognition. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’21). IEEE, Los Alamitos, CA, 6244-6253. https://doi.org/

[3]

Charlie Hewitt and Hatice Gunes. 2018. CNN-based Facial Affect Analysis on Mobile Devices. https://doi.org/10.48550/arXiv.1807.08775

[4]

Xiuyi Jia, Xiang Zheng, Weiwei Li, Changqing Zhang and Zechao Li. 2019. Facial Emotion Distribution Learning by Exploiting Low-Rank Label Correlations Locally. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19), IEEE, Los Alamitos, CA, 9833-9842. https://doi.org/10.1109/CVPR.2019.01007

[5]

Shikai Chen, Jianfeng Wang, Yuedong Chen, Zhongchao Shi, Xin Geng and Yongi Rui. 2020. Label Distribution Learning on Auxiliary Label Space Graphs for Facial Expression Recognition. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20), IEEE, Los Alamitos, CA, 13981-13990. https://doi.org/10.1109/CVPR42600.2020.01400

[6]

Zengqun Zhao, Qingshan Liu, and Feng Zhou. 2021. Robust lightweight facial expression recognition network with label distribution training. In Proceedings of the AAAI conference on artificial intelligence (AAAI 2021). AAAI Press, Menlo Park, 3510-3519. https://doi.org/10.1609/aaai.v35i4.16465

[7]

Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu and Chang Xu. 2020. GhostNet: More Features From Cheap Operations. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20). IEEE, Los Alamitos, CA, 1577-1586. https://doi.org/10.1109/CVPR42600.2020.00165

[8]

Qibin Hou, Daquan Zhou and Jiashi Feng. 2021. Coordinate Attention for Efficient Mobile Network Design. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’21). IEEE, Los Alamitos, CA, 13708-13717. https://doi.org/10.1109/CVPR46437.2021.01350

[9]

Ying Zhang, Tao Xiang, Timothy M. Hospedales and Huchuan Lu. 2018. Deep Mutual Learning. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’18). IEEE, Los Alamitos, CA, 4320-4328. https://doi.org/10.1109/CVPR.2018.00454

[10]

Ningning Ma, Xiangyu Zhang, Haitao Zheng and Jian Sun. 2018. ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design. In Proceedings of the 15th European Conference on Computer Vision (ECCV 2018). Springer, Berlin, 122-138. https://doi.org/10.48550/arXiv.1807.11164

Digital Library

[11]

Shan Li and Weihong Deng. 2019. Reliable Crowdsourcing and Deep Locality-Preserving Learning for Unconstrained Facial Expression Recognition, IEEE Trans Image Process 28, 1 (January 2019), 356-370. https://doi.org/ 10.1109/TIP.2018.2868382

Digital Library

[12]

Ali Mollahosseini, Behzad Hasani and Mohammad H.Mahoor. 2019. AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild. IEEE Trans Affect Comput 10, 1 (Jan-March 2019), 18-31. https://doi.org/ 10.1109/TAFFC.2017.2740923

Digital Library

[13]

Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia and Stefanos Zafeiriou. 2020. RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20). IEEE, Los Alamitos, CA, 5202-5211. https://doi.org/ 10.1109/CVPR42600.2020.00525

[14]

Vladislav Sovrasov. 2019. ptflops: a flops counting tool for neural networks in pytorch framework. https://github.com/sovrasov/flops-counter.pytorch

[15]

Chengfei Lv, Chaoyue Niu, Renjie Gu, Xiaotang Jiang, Zhaode Wang, Bin Liu, Ziqi Wu, Qiulin Yao, Congyu Huang, Panos Huang, Tao Huang, Hui Shu, Jinde Song, Bin Zou, Peng Lan, Guohuan Xu, Fei Wu, Shaojie Tang, Fan Wu and Guihai Chen. 2020. Walle: An End-to-End, General-Purpose, and Large-Scale Production System for Device-Cloud Collaborative Machine Learning. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22). USENIX, Berkeley, USA, 249-265

[16]

Shikai Chen, Jianfeng Wang, Yuedong Chen, Zhongchao Shi, Xin Geng and Yong Rui. 2020. Label Distribution Learning on Auxiliary Label Space Graphs for Facial Expression Recognition. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20). IEEE, Los Alamitos, CA, 13981-13990. https://doi.org/10.1109/CVPR42600.2020.01400

[17]

Yahui Nan, Jianguo Ju, Qingyi Hua, Haoming Zhang and Bo Wang. 2022. A-MobileNet: An approach of facial expression recognition, Alexandria Eng. J 61, 6 (June 2022), 4435-4444. https://doi.org/10.1016/j.aej.2021.09.066

[18]

Mark Sandler, Andrew Howard, Menglong Zhu, Zhmoginov Andrey and Liang-Chieh Chen. 2018. MobileNetV2: Inverted Residuals and Linear Bottlenecks. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’18). IEEE, Los Alamitos, CA, 4510-4520. https://doi.org/10.1109/CVPR.2018.00474

[19]

Dimitrios Kollias, Shiyang Cheng, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. 2020. Deep Neural Network Augmentation: Generating Faces for Affect Analysis. Int. J. Comput. Vision 128, 5 (May 2020), 1455–1484. https://doi.org/10.1007/s11263-020-01304-3

Digital Library

[20]

Dimitrios Kollias, Shiyang Cheng, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. 2020. Deep Neural Network Augmentation: Generating Faces for Affect Analysis. Int. J. Comput. Vision 128, 5 (May 2020), 1455–1484. https://doi.org/10.1007/s11263-020-01304-3

Digital Library

Cited By

Zhang HZhuang XGao XMao RRen Q(2024)Lightweight Facial Expression Recognition Based on Hybrid Multiscale and Multi-Head Collaborative AttentionPattern Recognition and Computer Vision10.1007/978-981-97-8490-5_22(304-316)Online publication date: 7-Nov-2024
https://doi.org/10.1007/978-981-97-8490-5_22

Index Terms

Lightweight Facial Expression Recognition Network with Dynamic Deep Mutual Learning
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision

Recommendations

Expression-invariant face recognition by facial expression transformations

In this paper, we present a method of expression-invariant face recognition that transforms input face image with an arbitrary expression into its corresponding neutral facial expression image. When a new face image with an arbitrary expression is ...
Facial expression recognition via learning deep sparse autoencoders

Facial expression recognition is an important research issue in the pattern recognition field. In this paper, we intend to present a novel framework for facial expression recognition to automatically distinguish the expressions with high accuracy. ...
Pose-robust feature learning for facial expression recognition

Automatic facial expression recognition (FER) from non-frontal views is a challenging research topic which has recently started to attract the attention of the research community. Pose variations are difficult to tackle and many face analysis methods ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

BIC '23: Proceedings of the 2023 3rd International Conference on Bioinformatics and Intelligent Computing

February 2023

398 pages

ISBN:9798400700200

DOI:10.1145/3592686

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 May 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

BIC 2023

BIC 2023: 2023 3rd International Conference on Bioinformatics and Intelligent Computing

February 10 - 12, 2023

Sanya, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
47
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)4

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang HZhuang XGao XMao RRen Q(2024)Lightweight Facial Expression Recognition Based on Hybrid Multiscale and Multi-Head Collaborative AttentionPattern Recognition and Computer Vision10.1007/978-981-97-8490-5_22(304-316)Online publication date: 7-Nov-2024
https://doi.org/10.1007/978-981-97-8490-5_22

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten