Patch Attention Network for Video Facial Expression Recognition
Authors: 
Yingkai Hua, Xinmin XuAuthors Info & Claims
Yingkai Hua, Xinmin XuAuthors Info & ClaimsPages 689 - 694
Abstract
Facial expression recognition (FER) is a hot research topic in computer vision. In recent years, attention mechanism has been widely used in facial expression recognition tasks and achieved good results. However, most methods applying attention mechanism are aimed at static image-based FER. In this paper, we propose a novel convolution neutral network (CNN) with attention mechanism for video-based FER. Our network introduces Patch Diff Attention (PDA) module to focus on regions with large variation, and Patch Self Attention (PSA) module to focus on regions containing more expression information. With extensive experiments on CK+ and AFEW datasets, our proposed method shows superior or similar performance compared to the state-of-the-art approaches.
References
[1]
A. Mehrabian. 2017. Communication Without Words. communication theory.
[2]
Shan Li and Weihong Deng. 2020. Deep facial expression recognition: A survey. IEEE T. Affect. Comput. (2020).
[3]
Mehmet Korkmaz and Nihat Yilmaz. 2016. Face recognition by using back propagation artificial neural network and windowing method. Journal of Image and Graphics (2016), 15-19.
[4]
Ralf C. Staudemeyer and Eric Rothstein Morris. 2019. Understanding LSTM–a tutorial into long short-term memory recurrent neural networks. arXiv preprint arXiv:1909.09586 (2019).
[5]
D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. 2014. Learning Spatiotemporal Features with 3D Convolutional Networks (2014).
[6]
Luefeng Chen, Mengtian Zhou, Wanjuan Su, Min Wu, Jinhua She, and Kaoru Hirota. 2018. Softmax regression based deep sparse autoencoder network for facial emotion recognition in human-robot interaction. Inform. Sciences (2018), 49-61.
[7]
Humayra B. Ali and David MW Powers. 2014. Fusion based fastica method: Facial expression recognition. Journal of Image and Graphics (2014), 1-7.
[8]
Huibai Wang and Siyang Hou Facial expression recognition based on the fusion of CNN and SIFT features. IEEE, 2020.
[9]
Huiyuan Yang, Umur Ciftci, and Lijun Yin Facial expression recognition by de-expression residue learning., 2018.
[10]
Kai Wang, Xiaojiang Peng, Jianfei Yang, Shijian Lu, and Yu Qiao Suppressing uncertainties for large-scale facial expression recognition., 2020.
[11]
J. Chen, Z. Chen, Z. Chi, and H. Fu. 2018. Facial Expression Recognition in Video with Multiple Feature Fusion. IEEE T. Affect. Comput. (2018), 1.
[12]
Yuanyuan Ding, Qin Zhao, Baoqing Li, and Xiaobing Yuan. 2017. Facial expression recognition from image sequence based on LBP and Taylor expansion. IEEE Access (2017), 19409-19419.
[13]
Duo Feng and Fuji Ren Dynamic facial expression recognition based on two-stream-cnn with lbp-top. IEEE, 2018.
[14]
Zhenbo Yu, Guangcan Liu, Qingshan Liu, and Jiankang Deng. 2018. Spatio-temporal convolutional features with nested LSTM for facial expression recognition. Neurocomputing (2018), 50-57.
[15]
Jianfeng Zhao, Xia Mao, and Jian Zhang. 2018. Learning deep facial expression features from image and optical flow sequences using 3D CNN. The Visual Computer (2018), 1461-1475.
[16]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun Deep residual learning for image recognition., 2016.
[17]
Debin Meng, Xiaojiang Peng, Kai Wang, and Yu Qiao Frame attention networks for facial expression recognition in videos. IEEE, 2019.
[18]
Yu Wang, XinMin Xu, and Yao Zhuang Learning Dynamics for Video Facial Expression Recognition., 2021.
[19]
Yong Li, Jiabei Zeng, Shiguang Shan, and Xilin Chen Patch-gated CNN for occlusion-aware facial expression recognition. IEEE, 2018.
[20]
Yong Li, Jiabei Zeng, Shiguang Shan, and Xilin Chen. 2018. Occlusion aware facial expression recognition using CNN with attention mechanism. IEEE T. Image Process. (2018), 2439-2450.
[21]
Jing Li, Kan Jin, Dalin Zhou, Naoyuki Kubota, and Zhaojie Ju. 2020. Attention mechanism-based CNN for facial expression recognition. Neurocomputing (2020), 340-350.
[22]
Shervin Minaee, Mehdi Minaei, and Amirali Abdolrashidi. 2021. Deep-emotion: Facial expression recognition using attentional convolutional network. Sensors-Basel (2021), 3046.
[23]
Kai Wang, Xiaojiang Peng, Jianfei Yang, Debin Meng, and Yu Qiao. 2020. Region attention networks for pose and occlusion robust facial expression recognition. IEEE T. Image Process. (2020), 4057-4069.
[24]
K. Zhang, Z. Zhang, Z. Li, and Y. Qiao. 2016. Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks. IEEE Signal Proc. Let. (2016), 1499-1503. DOI 10.1109/LSP.2016.2603342
[25]
Brandon Amos, Bartosz Ludwiczuk, and Mahadev Satyanarayanan. 2016. Openface: A general-purpose face recognition library with mobile applications. CMU School of Computer Science (2016), 20.
[26]
Patrick Lucey, Jeffrey F. Cohn, Takeo Kanade, Jason Saragih, Zara Ambadar, and Iain Matthews The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. IEEE, 2010.
[27]
Abhinav Dhall, Amanjot Kaur, Roland Goecke, and Tom Gedeon Emotiw 2018: Audio-video, student engagement and group-level affect prediction., 2018.
[28]
Heechul Jung, Sihaeng Lee, Junho Yim, Sunjeong Park, and Junmo Kim Joint fine-tuning in deep neural networks for facial expression recognition., 2015.
[29]
Cheng Lu, Wenming Zheng, Chaolong Li, Chuangao Tang, Suyuan Liu, Simeng Yan, and Yuan Zong Multiple spatio-temporal feature learning for video-based emotion recognition in the wild., 2018.
Index Terms
- Patch Attention Network for Video Facial Expression Recognition
Index terms have been assigned to the content through auto-classification.
Recommendations
Expression-invariant face recognition by facial expression transformations
In this paper, we present a method of expression-invariant face recognition that transforms input face image with an arbitrary expression into its corresponding neutral facial expression image. When a new face image with an arbitrary expression is ...
STAN: spatiotemporal attention network for video-based facial expression recognition
AbstractVideo-based facial expression recognition is a very challenging task. The expression features portrayed by traditional ResNet18 are not rich enough, while the classical method LSTM to process expression videos may not extract effective temporal ...
Dynamic-Static Graph Convolutional Network for Video-Based Facial Expression Recognition
MultiMedia ModelingAbstractMost of the current methods for video-based facial expression recognition (FER) in the wild are based on deep neural networks with attention mechanism to capture the relationships between frames. However, these methods suffer from the large ...
Comments
Information & Contributors
Information
Published In
September 2022
1221 pages
ISBN:9781450396899
DOI:10.1145/3573942
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 16 May 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
AIPR 2022
AIPR 2022: 2022 5th International Conference on Artificial Intelligence and Pattern Recognition
September 23 - 25, 2022
Xiamen, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 36Total Downloads
- Downloads (Last 12 months)12
- Downloads (Last 6 weeks)2
Reflects downloads up to 01 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format