Skip to main content
SpringerLink
Log in

To balance: balanced micro-expression recognition

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Micro-expressions are subtle facial movements that expose a person’s hidden emotions. Recognizing the micro-expression has importance for example in criminal investigations and psychotherapy. Compared with the shallower-architecture model, image magnification of these movements, which is also crucial for accurate recognition, has received relatively less attention in the field of micro-expression recognition. In this work, we find that there are some limitations during the training process, in particular, an imbalance in the distribution of motion amplitudes of samples, optical flow features, and semantic features. To mitigate their adverse effects, we propose adaptive balanced magnification, the balance of optical flow features and the balance of enhanced semantic features, to reduce these imbalances. Experimental results from three benchmarks (CASMEII, SAMM, and SMIC) show that our proposed method has higher accuracy and better recognition success than other micro-expression recognition methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Ekman, P.: Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage, Revised WW Norton & Company, New York (2009)

    Google Scholar 

  2. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  3. Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to forget: Continual prediction with lstm. Neural comput 12(10), 2451–2471 (2000)

    Article  Google Scholar 

  4. Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., Freeman, W.: Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. (TOG) 31(4), 1–8 (2012)

    Article  Google Scholar 

  5. Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: European Conference on Computer Vision, pp. 818–833. Springer. (2014)

  6. Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)

  7. Pang, J., Chen, K., Shi, J., Feng, H., Ouyang, W., and Lin, D.: Libra r-cnn: towards balanced learning for object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 821–830 (2019)

  8. Liu, Y.-J., Zhang, J.-K., Yan, W.-J., Wang, S.-J., Zhao, G., Xiaolan, F.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. 7(4), 299–310 (2015)

    Article  Google Scholar 

  9. Karmakar, D., Sarkar, R., Datta, M.: Spoofed replay attack detection by multidimensional fourier transform on facial micro-expression regions. Signal Process. Image Commun. 93(7), 116164 (2021)

    Article  Google Scholar 

  10. Liu, K.H., Jin, Q.S., Xu, H.C., Gan, Y.S., Liong, S.T.: Micro-expression recognition using advanced genetic algorithm. Signal Process. Image Commun. 93(3), 116153 (2021)

    Article  Google Scholar 

  11. Ren, Z., Ning, H.: A review of micro-expression recognition research (in Chinese). Comput. Eng. Appl. 57(1), 38–47 (2021)

    Google Scholar 

  12. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

  13. Li, X., Yu, J., Zhan, S.: Spontaneous facial micro-expression detection based on deep learning. In: 2016 IEEE 13th International Conference on Signal Processing (ICSP), pp. 1130–1134. IEEE. (2016)

  14. Chaudhry, R., Ravichandran, A., Hager, G., Vidal, R.: Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1932–1939. IEEE. (2009)

  15. Liong, S.-T., See, J., Wong, K., Phan, R.C.-W.: Less is more: micro-expression recognition from video using apex frame. Signal Process. Image Commun. 62, 82–92 (2018)

    Article  Google Scholar 

  16. Chongyang, W., Min, P., Tao, B., Tong, C.: Micro-attention for micro-expression recognition. Neurocomputing 410, 354–362 (2020)

    Article  Google Scholar 

  17. Liu, Y., Du, H., Zheng, L., Gedeon, T.: A neural micro-expression recognizer. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–4. IEEE. (2019)

  18. Yan, W.-J., Li, X., Wang, S.-J., Zhao, G., Liu, Y.-J., Chen, Y.-H., Xiaolan, F.: Casme II: an improved spontaneous micro-expression database and the baseline evaluation. PloS One 9(1), e86041 (2014)

    Article  Google Scholar 

  19. Li, X., Pfister, T., Huang, X., Zhao, G., Pietikäinen, M.: A spontaneous micro-expression database: Inducement, collection and baseline. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (fg), pp. 1–6. IEEE. (2013)

  20. Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: Samm: a spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(1), 116–129 (2016)

    Article  Google Scholar 

  21. Davison, A.K., Merghani, W., Yap, M.H.: Objective classes for micro-facial expression recognition. J. Imaging. 4(10), 119 (2018)

    Article  Google Scholar 

  22. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., and Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, pp. 94–101. IEEE. (2010)

  23. Xia, Z., Hong, X., Gao, X., Feng, X., Zhao, G.: Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions. IEEE Trans. Multimed. 22(3), 626–640 (2019)

    Article  Google Scholar 

  24. Liu, C., et al.: Beyond pixels: exploring new representations and applications for motion analysis. PhD thesis, Massachusetts Institute of Technology (2009)

  25. King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10, 1755–1758 (2009)

    Google Scholar 

  26. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  27. Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)

  28. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. IEEE. (2009)

  29. Gan, Y.S., Liong, S.-T., Yau, W.-C., Huang, Y.-C., Tan, L.-K.: Off-apexnet on micro-expression recognition system. Signal Process. Image Commun. 74, 129–139 (2019)

    Article  Google Scholar 

  30. Van Quang, N., Chun, J., Tokuyama, T.: Capsulenet for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–7. IEEE. (2019)

  31. Liong, S.-T., Gan, Y.S., See, J., Khor, H.-Q., and Huang, Y.-C.: Shallow triple stream three-dimensional CNN (STSTNET) for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5. IEEE. (2019)

  32. Zhou, L., Mao, Q., Xue, L.: Dual-inception network for cross-database micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5. IEEE. (2019)

  33. Krizhevsky, Alex., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In NIPS, page 1097–1105, 2012

  34. Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: Squeezenet: Alexnet-level accuracy with \(50\times\) fewer parameters and \(<0.5\) mb model size. arXiv preprint arXiv:1602.07360 (2016)

  35. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

  36. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (61872042, 61972375), the Key Project of Beijing Municipal Commission of Education (KZ201911417048), the Major Project of Technological Innovation 2030—“New Generation Artificial Intelligence” (2018AAA0100800), Premium Funding Project for Academic Human Resources Development in Beijing Union University (BPHR2020AZ01, BPH2020EZ01), Scientific Research Project of Beijing Municipal Commission of Education (KM202111417009), innovation funding project for postgraduates of Beijing Union University (YZ2020K001).

Author information

Authors and Affiliations

  1. Beijing Union University, Beijing, 100101, China

    Ren Zhang, Ning He, Yuzhe He & Kang Yan

  2. Beijing Information Science and Technology University, Beijing, 100101, China

    Ying Wu

Authors
  1. Ren Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ning He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ying Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuzhe He
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kang Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ning He.

Additional information

Communicated by B. K Bao.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, R., He, N., Wu, Y. et al. To balance: balanced micro-expression recognition. Multimedia Systems 28, 335–345 (2022). https://doi.org/10.1007/s00530-021-00842-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-021-00842-1

Keywords

Search

Navigation