Skip to main content
SpringerLink
Log in

E-FCNN for tiny facial expression recognition

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

As a hot issue in recent years, facial expression recognition(FER) has been widely applied in many fields, but it still faces great challenges in tiny facial expression recognition. Currently, most of the FER networks only consider images of ideal sizes. Their recognition accuracy would significantly decrease as the image resolution decreases. However, images captured by surveillance cameras often contain tiny faces with low-resolution. This paper proposes an edge-aware feedback convolutional neural network(E-FCNN) for tiny FER, which associates image super-resolution and facial expression recognition together. To effectively leverage the texture information of faces, we design a novel three-stream super-resolution network, which is embedded with an edge-enhancement block as one branch. The other two branches are the up-sampling branch and SR(Super-Resolution) primary branch. Specifically, visual features are extracted from tiny images based on a hierarchical strategy, and then put into a feedback block with fused results of the three branches. Experiments are performed on down-sampled images in four facial expression datasets: CK+, FER2013, BU-3DFE, RAF-DB. The results demonstrate the favorable performance of our network.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Weiguo W, Qingmei M, Yu W (2004) Development of the humanoid head portrait robot system with flexible face and expression. In: Proceedings of the 2004 IEEE International Conference on Robotics and Biomimetics, pp 757–762. https://doi.org/10.1109/ROBIO.2004.1521877

  2. Patil S A, Deore P J (2016) Local binary pattern based face recognition system for automotive security. In: Proceedings of the International Conference on Signal Processing, Computing and Control, pp 13–17

  3. Su M H, Wu C H, Huang K Y, Hong Q B, Wang H M (2017) Exploring microscopic fluctuation of facial expression for mood disorder classification. In: Proceedings of the International Conference on Orange Technologies, pp 65–69

  4. Mollahosseini A, Chan D, Mahoor M H (2016) Going deeper in facial expression recognition using deep neural networks. In: Proceedings of the Applications of Computer Vision, pp 1–10

  5. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely Connected Convolutional Networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  6. Kim J, Lee JK, Lee KM (2016) Deeply-Recursive Convolutional Network for Image Super-Resolution. In: CVPR

  7. Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks TPAMI

  8. Lim B, Son S, Kim H, Nah S, Lee KM (2017) Enhanced deep residual networks for single image Super-Resolution. In: IEEE Computer vision and pattern recognition workshops (CVPRW)

  9. Li Z, Yang J, Liu Z, Yang X, Jeon G, Wu W (2019) Feedback network for image Super-Resolution. In: IEEE Conference on computer vision and pattern recognition (CVPR)

  10. Zhang T, Zheng W, Cui Z, Zong Y, Yan J, Yan K (2016) A deep neural network driven feature learning method for multi-view facial expression recognition. IEEE Trans Multimed 18(12):2528–2536

    Article  Google Scholar 

  11. Sharma S, Kiros R, Salakhutdinov R (2015) Action recognition using visual attention. In: NIPS workshop on Time Series, pp 1–11

  12. Shao J, Qian Y (2019) Three convolutional neural network models for facial expression recognition in the wild. In: NEUROCOMPUTING, pp 82–92. https://doi.org/10.1016/j.neucom.2019.05.005

  13. Cheng B, Wang Z, Zhang Z, Li Z, Liu D, Yang J, Huang S, Huang T (2017) Robust emotion recognition from low quality and low bit rate video: A deep learning approach. In: 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII)

  14. Yu C, Tai Y, Liu X, Shen C, Yang J (2018) FSRNet: End-to-end Learning Face Super-Resolution With Facial Priors. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 2492–2501

  15. Dogan B, Gu S, Timofte R (2019) Exemplar guided face image Super-Resolution without facial landmarks the. IEEE conference on computer vision and pattern recognition (CVPR) workshops

  16. Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y (2018) Residual dense network for image super-resolution. In: CVPR

  17. Yang W, Feng J, Yang J, Zhao F, Liu J, Guo Z, Yan S, et al. (2017) Deep edge guided recurrent residual learning for image super-Resolution. IEEE Trans Image Process 26(12):5895–5907. https://doi.org/10.1109/TIP.2017.2750403

    Article  MathSciNet  Google Scholar 

  18. Chen Y, Tai Y, Liu X, Shen C, Yang J (2018) Fsrnet: End-to-end learning face super-resolution with facial priors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2492–2501

  19. Liu B, Ait-Boudaoud D (2019) Effective image super resolution via hierarchical convolutional neural network. In: Neurocomputing. https://doi.org/10.1016/j.neucom.2019.09.035

  20. Xie Saining T u (2017) Zhuowen, Holistically-Nested Edge Detection. In: International Journal of Computer Vision, pp 3–18. https://doi.org/10.1007/s11263-017-1004-z

  21. Qin Z, Wu J, Liu Y, Gedeon T (2018) Visual saliency maps can apply to facial expression. Recognition, arXiv:1811.04544

  22. Lucey P, Cohn J F, Kanade T, Saragih J (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: Proceedings of the Computer Vision and Pattern Recognition Workshops, pp 94–101

  23. Fer K (2013) dataset. http://web.archive.org/web/20080207010024/http://web.archive.org/web/20080207010024/http://www.808multimedia.com/winnt/kernel.htmhttp://www.808multimedia.com/winnt/kernel.htm. Accessed: 2018- 11-10

  24. Goodfellow I J, Erhan D, Luc C P, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee D H (2015) Challenges in representation learning: a report on three machine learning contests. Neural Netw 64:59–63

    Article  Google Scholar 

  25. Yin L, Wei X, Sun Y, Wang J, Rosato M J (2006) A 3D facial expression database for facial behavior research. In: Proceedings of the International Conference on Automatic Face and Gesture Recognition, pp 211–216

  26. Li S, Deng W, Du J (2017) Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2852–2861

  27. Yu X, Fernando B, Ghanem B, Porikli F, Hartley R (2018) Face Super-resolution Guided by Facial Component Heatmaps. The European Conference on Computer Vision (ECCV), pp 217–233

  28. Yu X, Fernando B, Hartley R, Porikli F (2018) Super-resolving very Low-Resolution face images with supplementary attributes. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 908–917

  29. Kim D, Kim M, Kwon G, Kim D-S Progressive Face Super-Resolution via Attention to Facial Landmark. arXiv:1908.08239 [cs.CV]

  30. Liu P, Zhou J T, Tsang W H, Meng Z, Han S, Tong Y (2014) Feature disentangling machine-a novel approach of feature selection and disentangling in facial expression analysis. In: European conference on computer vision (ECCV), pp 151–166

  31. Pramerdorfer C, Martin Kampel E Facial Expression Recognition using Convolutional Neural Networks: State of the Art, arXiv:1612.02903 [cs.CV]

  32. Kuo C-M, Lai S-H, Sarkis M (2018) A compact deep learning model for robust facial expression recognition. 2018 IEEE conference on computer vision and pattern recognition

  33. Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2014) Going Deeper with Convolutions. Computer Science

  34. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  35. Jing W, Tian F, Zhang J, Chao K-M, Hong Z, Liu X Feature Super-Resolution Based Facial Expression Recognition for Multi-scale Low-Resolution Faces, arXiv:2004.02234 [cs.CV]

  36. Yan Y, Zhang Z, Chen S, Wang H (2020) Low-resolution facial expression recognition: A filter learning perspective. Signal Processing. https://doi.org/10.1016/j.sigpro.2019.107370

Download references

Author information

Authors and Affiliations

  1. Department of Electronic and Information Engineering, Shanghai University of Electric Power, Shanghai 200120, China

    Jie Shao & Qiyu Cheng

Authors
  1. Jie Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qiyu Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qiyu Cheng.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shao, J., Cheng, Q. E-FCNN for tiny facial expression recognition. Appl Intell 51, 549–559 (2021). https://doi.org/10.1007/s10489-020-01855-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01855-5

Keywords

Search

Navigation