Skip to main content
SpringerLink
Log in

Robust facial expression recognition with global-local joint representation learning

  • Special Issue Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

As an important part in computer vision, facial expression recognition (FER) has received extensive attention, but it still has lots of challenges in this area. One of the important difficulties is to remain the topological information in the feature extraction operation. In this paper, we propose a novel facial expression recognition method with lite dual channel neural network based on graph convolutional networks (DCNN-GCN). In the proposed method, (1) the topological structure information and texture feature of regions of interest (ROIs) are modeled as graphs and processed with graph convolutional network (GCN) to remain the topological features. (2) The local features of ROIs and global features are extracted with dual channel neural networks, which can improve the performance of features extraction and reduce the complexity of networks. The proposed method is evaluated on CK+, Oulu-CASIA and MMI data sets. Experiment results show that the proposed method can significantly improve the accuracy of facial expression recognition. In addition, the network is much lite and suitable for application.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Chen, L., et al.: Softmax regression based deep sparse autoencoder network for facial emotion recognition in human-robot interaction. Inf. Sci. 428, 49–61 (2018)

    Article  MathSciNet  Google Scholar 

  2. Jabon, M., Bailenson, J., Pontikakis, E., Takayama, L., Nass, C.: Facial expression analysis for predicting unsafe driving behavior. IEEE Pervasive Comput. 10(4), 84–95 (2010)

    Article  Google Scholar 

  3. Chen, J., Wang, G., Zhang, K., Wang, G., Liu, L.: A pilot study on evaluating children with autism spectrum disorder using computer games. Comput. Hum. Behav. 90, 204–214 (2019)

    Article  Google Scholar 

  4. Ekman, P., Friesen, W.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124–129 (1971)

    Article  Google Scholar 

  5. Wold, S., Esbensen, K., Geladi, P.: Principal component analysis[J]. Chemometr. Intell. Lab. Syst. 2(1–3), 37–52 (1987)

    Article  Google Scholar 

  6. Hyvarinen, A., Oja, E.: Independent component analysis: algorithms and applications. Neural Netw. 13(4), 411–430 (2000)

    Article  Google Scholar 

  7. Belhumeur, P.N., Hespanha, J.P., Kriegman, D.J.: Recognition using class specific linear projection. In: European Conference on Computer Vision, pp. 43–58 (1966)

  8. Ekman, P., Friesen, W.V.: Facial action coding system (facs): a technique for the measurement of facial actions. Riv. Psichiatr. 47(2), 126–38 (1978)

    Google Scholar 

  9. Bartlett, M. S., et al.: Recognizing facial expression: machine learning and application to spontaneous behavior. In: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2, 568–573 (2005)

  10. Shan, C., Gong, S., Mcowan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  11. Huang, M.-W., Wang, Z.-w., Ying, Z.-L.: A new method for facial expression recognition based on sparse representation plus lbp. In: 2010 3rd International Congress on Image and Signal Processing 4, 1750–1754 (2010)

  12. Liu, X., Yang, X., Wang, M., Hong, R.: Deep neighborhood component analysis for visual similarity modeling. ACM Trans. Intell. Syst. Technol. (TIST) 11(3), 1–15 (2020)

    Google Scholar 

  13. Yang, X., Zhou, P., Wang, M.: Person reidentification via structural deep metric learning. IEEE Trans. Neural Netw. Learn. Syst. 30(10), 2987–2998 (2018)

    Article  Google Scholar 

  14. Cheng, S., Zhou, G.: Facial expression recognition method based on improved vgg convolutional neural network. Int. J. Pattern Recognit. Artif Intell. 34(07), 2056003 (2020)

    Article  Google Scholar 

  15. Zhong, Y., Qiu, S., Luo, X., Meng, Z., Liu, J.: Facial expression recognition based on optimized resnet. In: 2020 2nd World Symposium on Artificial Intelligence, pp. 84–91 (2020)

  16. Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter Conference on Applications of Computer Vision, pp. 1–10 (2016)

  17. Hamester, D., Barros, P., Wermter, S.: Face expression recognition with a 2-channel convolutional neural network. In: 2015 International Joint Conference on Neural Networks, pp. 1–8 (2015)

  18. Zhao, X., et al.: Peak-piloted deep network for facial expression recognition. In: European Conference on Computer Vision, pp. 425–442 (2016)

  19. Xie, S., Hu, H.: Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks. IEEE Trans. Multimed. 21(1), 211–220 (2018)

    Article  Google Scholar 

  20. Wang, Z., Ying, Z.: Facial expression recognition based on local phase quantization and sparse representation. In: 2012 8th International Conference on Natural Computation, pp. 222–225 (2012)

  21. Cootes, T.F., Taylor, C.J., Cooper, D.H., Graham, J.: Active shape models-their training and application. Comput. Vis. Image Underst. 61(1), 38–59 (1995)

    Article  Google Scholar 

  22. Zhong, L., et al.: Learning active facial patches for expression analysis. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2562–2569 (2012)

  23. Bandrabur, A., Florea, L., Florea, C., Mancas, M.: Emotion identification by facial landmarks dynamics analysis. In: 2015 IEEE International Conference on Intelligent Computer Communication and Processing, pp. 379–382 (2015)

  24. Desrosiers, P.A., Daoudi, M., Devanne, M.: Novel generative model for facial expressions based on statistical shape analysis of landmarks trajectories. In: 2016 23rd International Conference on Pattern Recognition, pp. 961–966 (2016)

  25. Meng, L., et al.: Learning using privileged information for food recognition. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 557–565 (2019)

  26. Dong, J., et al.: Fine-grained fashion similarity prediction by attribute-specific embedding learning. IEEE Trans. Image Process. 30, 8410–8425 (2021)

    Article  Google Scholar 

  27. Simonyan, K., Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2015)

  28. He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: European Conference on Computer Vision, pp. 630–645 (2016)

  29. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)

  30. Liu, M., Li, S., Shan, S., Wang, R., Chen, X.: Deeply learning deformable facial action parts model for dynamic expression analysis. In: Asian Conference on Computer Vision, pp. 143–157 (2014)

  31. Gan, Y., Chen, J., Yang, Z., Xu, L.: Multiple attention network for facial expression recognition. IEEE Access 8, 7383–7393 (2020)

    Article  Google Scholar 

  32. Meng, Z., Liu, P., Cai, J., Han, S., Tong, Y.: Identity-aware convolutional neural network for facial expression recognition. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 558–565 (2017)

  33. Wang, K., Peng, X., Yang, J., Lu, S., Qiao, Y.: Suppressing uncertainties for large-scale facial expression recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6897–6906 (2020)

  34. Yang, H., Ciftci, U., Yin, L.: Facial expression recognition by de-expression residue learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2168–2177 (2018)

  35. Yang, H., Zhang, Z., Yin, L.: Identity-adaptive facial expression recognition through expression regeneration using conditional generative adversarial networks. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 294–301 (2018)

  36. Agrawal, A., Mittal, N.: Using cnn for facial expression recognition: a study of the effects of kernel size and number of filters on accuracy. Vis. Comput. 36, 405–412 (2019)

    Article  Google Scholar 

  37. Chen, Y., Hu, H.: Facial expression recognition by inter-class relational learning. IEEE Access PP(99), 1–1 (2019)

    Article  Google Scholar 

  38. Wang, F., Shen, L.: Expression recognition using region features and facial action units. In: 2019 15th International Conference on Intelligent Environments, pp. 9–15 (2019)

  39. Xu, Q., Zhao, N.: A facial expression recognition algorithm based on cnn and lbp feature. 2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference 1, 2304–2308 (2020)

  40. Nguyen, D.H., Kim, S., Lee, G.S., Yang, H.J., Na, I.S., Kim, S.H.: Facial expression recognition using a temporal ensemble of multi-level convolutional neural networks. IEEE Trans. Affect. Comput. 13(1), 226–237 (2022). https://doi.org/10.1109/TAFFC.2019.2946540

    Article  Google Scholar 

  41. Li, Y., Zeng, J., Shan, S., Chen, X.: Occlusion aware facial expression recognition using cnn with attention mechanism. IEEE Trans. Image Process. 28(5), 2439–2450 (2018)

    Article  MathSciNet  Google Scholar 

  42. Jung, H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2983–2991 (2015)

  43. Kipf, T. N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)

  44. Yang, X., Du, X., Wang, M.: Learning to match on graph for fashion compatibility modeling. In: Proceedings of the AAAI Conference on Artificial Intelligence 34(01), 287–294 (2020)

  45. Liu, D., Zhang, H., Zhou, P.: Video-based facial expression recognition using graph convolutional networks. In: 2020 25th International Conference on Pattern Recognition, pp. 607–614 (2021)

  46. Kumar, A.J.R., Bhanu, B.: Micro-expression classification based on landmark relations with graph attention convolutional network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1511–1520 (2021)

  47. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural. Inf. Process. Syst. 25, 1097–1105 (2012)

    Google Scholar 

  48. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456 (2015)

  49. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  50. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Matthews, I.: The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, pp. 94–101 (2010). https://doi.org/10.1109/CVPRW.2010.5543262

  51. Zhao, G., Huang, X., Taini, M., Li, S.Z., Pietikainen, M.: Facial expression recognition from near-infrared videos. Image Vis. Comput. 29, 607–619 (2011)

    Article  Google Scholar 

  52. Pantic, M., Valstar, M., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: 2005 IEEE international conference on multimedia and Expo, p. 5 (2005)

  53. King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)

    Google Scholar 

  54. Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874 (2014)

  55. Kim, Y., Yoo, B., Kwak, Y., Choi, C., Kim, J.: Deep generative-contrastive networks for facial expression recognition. arXiv preprint arXiv:1703.07140 (2017)

  56. Hui, D., Zhou, S.K., Chellappa, R.: Facenet2expnet: regularizing a deep face recognition net for expression recognition. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), IEEE, pp. 118–126 (2017)

  57. Liu, L., Jiang, R., Huo, J., Chen, J.: Self-difference convolutional neural network for facial expression recognition. Sensors 21(6), 2250 (2021)

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (No. 61802105, 61976078), the University Synergy Innovation Program of Anhui Province (No. GXXT-2021-005, GXXT-2020-014), and Natural Science Foundation of Anhui Province (No. 1908085QF265, 2108085MF203).

Author information

Authors and Affiliations

  1. Key Laboratory of Knowledge Engineering with Big Data, Ministry of Education, Hefei University of Technology, Hefei, 230009, Anhui, China

    Chunxiao Fan, Zhenxing Wang, Jia Li & Xiao Sun

  2. Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230088, Anhui, China

    Chunxiao Fan

  3. School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230009, Anhui, China

    Chunxiao Fan, Zhenxing Wang, Jia Li & Xiao Sun

  4. Institute of Physical Science and Information Technology, Anhui University, Hefei, 230039, Anhui, China

    Shanshan Wang

Authors
  1. Chunxiao Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhenxing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jia Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shanshan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiao Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiao Sun.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fan, C., Wang, Z., Li, J. et al. Robust facial expression recognition with global-local joint representation learning. Multimedia Systems 29, 3069–3079 (2023). https://doi.org/10.1007/s00530-022-00907-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-022-00907-9

Keywords

Search

Navigation