Skip to main content
SpringerLink
Log in

A Mix Fusion Spatial-Temporal Network for Facial Expression Recognition

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14429))

Included in the following conference series:

  • 338 Accesses

Abstract

Facial expression is a powerful, natural and universal signal for human beings to convey their emotional states and intentions. In this paper, we propose a new spatial-temporal facial expression recognition network which outperforms many state-of-the-art methods. Our model is composed by two networks, a temporal feature extraction network based on facial landmarks and a spatial feature extraction network based on densely connected network. Image preprocessing method is optimized according to the features of the expression image to reduce network’s overfitting on small datasets. In addition, we propose a mix fusion strategy to better combine spatial and temporal features. Finally, experiments on public datasets are conducted to verify the effectiveness of each module and the improvement of expression recognition accuracy of the spatial-temporal fusion network. The accuracies on OULU-CASIA and CK + datasets reach 90.21% and 99.82% respectively.

This work has been supported by the Fundamental Research Funds for the Central Universities.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affective Comput. 13, 1195-1215 (2022)

    Google Scholar 

  2. Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. In: Lee, M., Hirose, A., Hou, ZG., Kil, R.M. (eds.) Neural Information Processing. ICONIP 2013. LNCS, vol. 8228. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-42051-1_16

  3. Tang, Y.: Deep learning using linear support vector machines (2013). arXiv preprint arXiv:1306.0239

  4. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

  5. Zhou, J., Zhang, X., Liu, Y., Lan, X.: Facial expression recognition using spatial-temporal semantic graph network. In: 2020 IEEE International Conference on Image Processing (ICIP), pp. 1961–1965. IEEE (2020)

    Google Scholar 

  6. Zhang, K., Huang, Y., Du, Y., Wang, L.: Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans. Image Process. 26(9), 4193–4203 (2017)

    Article  MathSciNet  Google Scholar 

  7. Jung, H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2983–2991 (2015)

    Google Scholar 

  8. Liu, W. F., Wang, Z.F.: Facial expression recognition based on fusion of multiple Gabor features. In: 18th International Conference on Pattern Recognition. IEEE (2006)

    Google Scholar 

  9. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 94–101. IEEE (2010)

    Google Scholar 

  10. Taini, M., Zhao, G., Li, S. Z., Pietikainen, M.: Facial expression recognition from near-infrared video sequences. In: 2008 19th International Conference on Pattern Recognition, pp. 1–4. IEEE (2008)

    Google Scholar 

  11. Kuo, C.M., Lai, S.H., Sarkis, M.: A compact deep learning model for robust facial expression recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 2121–2129 (2018)

    Google Scholar 

  12. Zhu, X., He, Z., Zhao, L., Dai, Z., Yang, Q.: A cascade attention based facial expression recognition network by fusing multi-scale spatio-temporal features. Sensors 22(4), 1350 (2022)

    Article  Google Scholar 

  13. Ramachandram, D., Taylor, G.W.: Deep multimodal learning: a survey on recent advances and trends. IEEE Signal Process. Mag. 34(6), 96–108 (2017)

    Article  Google Scholar 

  14. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, vol. 1, pp. I-I. IEEE (2001)

    Google Scholar 

  15. Kowalski, M., Naruniec, J., Trzcinski, T.: Deep alignment network: a convolutional neural network for robust face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 88–97 (2017)

    Google Scholar 

  16. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Electronic Science and Technology of China, Chengdu, Sichuan, China

    Chang Shu & Feng Xue

Authors
  1. Chang Shu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Feng Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chang Shu .

Editor information

Editors and Affiliations

  1. Nanjing University of Information Science and Technology, Nanjing, China

    Qingshan Liu

  2. Xiamen University, Xiamen, China

    Hanzi Wang

  3. Beijing University of Posts and Telecommunications, Beijing, China

    Zhanyu Ma

  4. Sun Yat-sen University, Guangzhou, China

    Weishi Zheng

  5. Peking University, Beijing, China

    Hongbin Zha

  6. Chinese Academy of Sciences, Beijing, China

    Xilin Chen

  7. Chinese Academy of Sciences, Beijing, China

    Liang Wang

  8. Xiamen University, Xiamen, China

    Rongrong Ji

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shu, C., Xue, F. (2024). A Mix Fusion Spatial-Temporal Network for Facial Expression Recognition. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14429. Springer, Singapore. https://doi.org/10.1007/978-981-99-8469-5_25

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-8469-5_25

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-8468-8

  • Online ISBN: 978-981-99-8469-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation