Skip to main content
SpringerLink
Log in

Micro-expression recognition with attention mechanism and region enhancement

  • Special Issue Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Micro-expression recognition (MER) is an interdisciplinary research task that has attracted attention. This is because MER can be relevant to multiple fields, such as computer vision, psychology, human-computer interaction, and social security. Because the scarcity of databases and difficulty in video semantics understanding, end-to-end MER still faces many challenges. In this study, we propose an MER framework with attention mechanism and region enhancement (MER-AMRE). Attention mechanisms are introduced to enhance the representation performance of the model, which can improve the recognition accuracy. Additionally, we use Euler video magnification in data preprocessing to enhance facial variation areas. AffectNet is leveraged to pretrain a facial region of interest (RoI) feature extractor with attention regions. Finally, we combine the facial RoI features with global facial features to recognize micro-expressions. Extensive experiments on two well-known micro-expression datasets, CASME II and SAMM, verified the robustness and generalization of the proposed MER-AMRE framework.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Ekman, P., Friesen, W.V., Ellsworth, P.: Emotion in the human face: guidelines for research and an integration of findings. Pergamon Press (1972)

  2. Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., Gao, W.: Pre-trained image processing transformer. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 12299–12310 (2021)

  3. Guo, D., Tang, S., Wang, M.: Connectionist temporal modeling of video and language: a joint model for translation and sign labeling. In: International Joint Conference on Artificial Intelligence, pp. 751–757 (2019)

  4. Tang, S., Guo, D., Hong, R., Wang, M.: Graph-based multimodal sequential embedding for sign language translation. IEEE Trans Multimed (2021). https://doi.org/10.1109/TMM.2021.3117124

  5. Ekman, P., Friesen, W.V.: Measuring facial movement. Environ. Psychol. Nonverbal Behav. 1(1), 56–75 (1976)

    Article  Google Scholar 

  6. Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans Affect Comput (2020). https://doi.org/10.1109/TAFFC.2020.2981446

  7. Oh, Y.-H., See, J., Le Ngo, A.C., Phan, R.C.-W., Baskaran, V.M.: A survey of automatic facial micro-expression analysis: databases, methods, and challenges. Front. Psychol. 9, 1128 (2018)

    Article  Google Scholar 

  8. Marrero Fernandez, P.D., Guerrero Pena, F.A., Ren, T., Cunha, A.: Feratt: Facial expression recognition with attention net. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (2019)

  9. Ben, X., Ren, Y., Zhang, J., Wang, S.-J., Kpalma, K., Meng, W., Liu, Y.-J.: Video-based facial micro-expression analysis: a survey of datasets, features and algorithms. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)

  10. Mollahosseini, A., Hasani, B., Mahoor, M.H.: Affectnet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2017)

    Article  Google Scholar 

  11. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 94–101 (2010)

  12. Goodfellow, I.J., Erhan, D., Carrier, P.L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al.: Challenges in representation learning: a report on three machine learning contests. In: International Conference on Neural Information Processing, pp. 117–124 (2013)

  13. Yan, W.-J., Wu, Q., Liu, Y.-J., Wang, S.-J., Fu, X.: Casme database: A dataset of spontaneous micro-expressions collected from neutralized faces. In: IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp. 1–7 (2013)

  14. Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., Fu, X.: Casme ii: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1), 86041 (2014)

    Article  Google Scholar 

  15. Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: Samm: a spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(1), 116–129 (2016)

    Article  Google Scholar 

  16. Li, X., Pfister, T., Huang, X., Zhao, G., Pietikäinen, M.: A spontaneous micro-expression database: Inducement, collection and baseline. In: IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, pp. 1–6 (2013)

  17. Jin, H., Liu, Q., Lu, H., Tong, X.: Face detection using improved lbp under bayesian framework. In: International Conference on Image and Graphics, pp. 306–309 (2004)

  18. Mattivi, R., Shao, L.: Human action recognition using lbp-top as sparse spatio-temporal feature descriptor. In: International Conference on Computer Analysis of Images and Patterns, pp. 740–747 (2009)

  19. Ben, X., Zhang, P., Yan, R., Yang, M., Ge, G.: Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation. Neural Comput. Appl. 27(8), 2629–2646 (2016)

    Article  Google Scholar 

  20. Ben, X., Jia, X., Yan, R., Zhang, X., Meng, W.: Learning effective binary descriptors for micro-expression recognition transferred by macro-information. Pattern Recogn. Lett. 107, 50–58 (2018)

    Article  Google Scholar 

  21. Guo, Y., Li, B., Ben, X., Ren, Y., Zhang, J., Yan, R., Li, Y.: A magnitude and angle combined optical flow feature for micro-expression spotting. IEEE MultiMedia 28(2), 29–39 (2021)

    Article  Google Scholar 

  22. Thi Thu Nguyen, N., Thi Thu Nguyen, D., The Pham, B.: Micro-expression recognition based on the fusion between optical flow and dynamic image. In: 2021 The 5th International Conference on Machine Learning and Soft Computing, pp. 115–120 (2021)

  23. Shin, M., Kim, M., Kwon, D.-S.: Baseline cnn structure analysis for facial expression recognition. In: IEEE International Symposium on Robot and Human Interactive Communication, pp. 724–729 (2016)

  24. Yang, H., Ciftci, U., Yin, L.: Facial expression recognition by de-expression residue learning. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 2168–2177 (2018)

  25. Liong, S.-T., See, J., Wong, K., Phan, R.C.-W.: Less is more: micro-expression recognition from video using apex frame. Signal Process.: Image Commun. 62, 82–92 (2018)

    Google Scholar 

  26. Duthoit, C.J., Sztynda, T., Lal, S.K., Jap, B.T., Agbinya, J.I.: Optical flow image analysis of facial expressions of human emotion: forensic applications. In: International Conference on Forensic Applications and Techniques in Telecommunications, Information and Multimedia, pp. 1–6 (2008)

  27. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  28. Wang, C., Peng, M., Bi, T., Chen, T.: Micro-attention for micro-expression recognition. Neurocomputing 410, 354–362 (2020)

    Article  Google Scholar 

  29. Stenberg, C.R., Emde, C.: The facial expression of anger in seven-month-old infants. Child Dev. 54(1), 178 (1983)

    Google Scholar 

  30. Lo, L., Xie, H.-X., Shuai, H.-H., Cheng, W.-H.: Mer-gcn: Micro-expression recognition based on relation modeling with graph convolutional networks. In: IEEE Conference on Multimedia Information Processing and Retrieval, pp. 79–84 (2020)

  31. Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., Freeman, W.: Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. 31(4), 1–8 (2012)

    Article  Google Scholar 

  32. Wadhwa, N., Wu, H.-Y., Davis, A., Rubinstein, M., Shih, E., Mysore, G.J., Chen, J.G., Buyukozturk, O., Guttag, J.V., Freeman, W.T., et al.: Eulerian video magnification and analysis. Commun. ACM 60(1), 87–95 (2016)

    Article  Google Scholar 

  33. Köksoy, O.: Multiresponse robust design: man square error (mse) criterion. Appl. Math. Comput. 175(2), 1716–1729 (2006)

    MathSciNet  MATH  Google Scholar 

  34. De Boer, P.-T., Kroese, D.P., Mannor, S., Rubinstein, R.Y.: A tutorial on the cross-entropy method. Ann. Oper. Res. 134(1), 19–67 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  35. Wang, S.-J., Yan, W.-J., Li, X., Zhao, G., Fu, X.: Micro-expression recognition using dynamic textures on tensor independent color space. In: International Conference on Pattern Recognition, pp. 4678–4683 (2014)

  36. Kim, D.H., Baddar, W.J., Jang, J., Ro, Y.M.: Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans. Affect. Comput. 10(2), 223–236 (2017)

    Article  Google Scholar 

  37. Liu, J., Li, K., Song, B., Zhao, L.: A multi-stream convolutional neural network for micro-expression recognition using optical flow and evm. arXiv preprint arXiv:2011.03756 (2020)

  38. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

  39. Hong, X., Zhao, G., Zafeiriou, S., Pantic, M., Pietikäinen, M.: Capturing correlations of local features for image representation. Neurocomputing 184, 99–106 (2016)

    Article  Google Scholar 

  40. Li, X., Hong, X., Moilanen, A., Huang, X., Pfister, T., Zhao, G., Pietikäinen, M.: Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans. Affect. Comput. 9(4), 563–577 (2017)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, 230601, China

    Yi Wang, Shixin Zheng, Xiao Sun, Dan Guo & Junjie Lang

Authors
  1. Yi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shixin Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiao Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dan Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Junjie Lang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shixin Zheng.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., Zheng, S., Sun, X. et al. Micro-expression recognition with attention mechanism and region enhancement. Multimedia Systems 29, 3095–3103 (2023). https://doi.org/10.1007/s00530-022-00934-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-022-00934-6

Keywords

Search

Navigation