Skip to main content

Advertisement

SpringerLink
Log in

Micro-expression recognition based on SqueezeNet and C3D

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Micro-expression recognition has attracted extensive attention from psychological and computer vision communities due to its multiple real-life applications. Compared with macro-expression, the change of micro-expression is subtle and difficult for detection and capture. Hence, the recognition is challenging. This paper proposes a micro-expression recognition model SQU-C3D which combines SqueezeNet and C3D methods. The features of micro-expression are mainly reflected in the apex frame, therefore a lightweight network called SqueezeNet is adopted to implement a reliable apex frame spotting method for dataset without apex frame labels. The position of the apex frame is detected by comparing the feature difference between the current and onset frames. In addition, the Convolutional 3D (C3D) network is utilized for micro-expression recognition due to its strong capability of extracting spatial–temporal features. The apex frame determined by SqueezeNet is fed into the C3D network along with the onset and offset frames, and the micro-expression is recognized by learning the features of these three key frames. Extensive experiments are conducted on three spontaneous micro-expression databases, namely, CASME II, SAMM, and SMIC-HS, where CASME II and SAMM include apex frame labels, whereas SMIC-HS does not. SQU-C3D achieves accuracy of 80.29% with 7 classes, 81.33% with 5 classes and 79.12% with 3 classes on the micro-expression benchmark dataset of CASME II, SAMM and SMIC-HS, respectively. Experimental results reveal that the proposed framework performs better than the state-of-the-art methods in the comparison.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Ekman, P.: Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage (Revised edition). WW Norton & Company, New York (2009)

    Google Scholar 

  2. Yan, W.J., Wu, Q., Liang, J., Chen, Y.H., Fu, X.: How fast are the leaked facial expressions: The duration of micro-expressions. J. Nonverbal Behav. 37(4), 217–230 (2013)

    Article  Google Scholar 

  3. Ekman, P., Friesen, W.V.: Nonverbal leakage and clues to deception. Psychiatry 32(1), 88–106 (1969)

    Article  Google Scholar 

  4. Ekman, P.: Lie catching and microexpressions. In: The Philosophy of Deception, pp. 118–133. Oxford University Press, Oxford (2009)

    Chapter  Google Scholar 

  5. Liong, S.T., See, J., Wong, K.S., Phan, C.W.: Automatic Micro-expression Recognition from Long Video Using a Single Spotted Apex. Asian Conference on Computer Vision, pp. 345–360. Springer, Cham (2016)

    Google Scholar 

  6. Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size (2016). Preprint at arXiv:1602.07360

  7. Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. In: IEEE Int. Conf. Comput. Vis. 4489–4497 (2015)

  8. Yan, W.J., Li, X., Wang, S.J., Zhao, G., Liu, Y.J., Chen, Y.H., Fu, X.: CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1), e86041 (2014)

    Article  Google Scholar 

  9. Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: SAMM: A spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(99), 116–129 (2018)

    Article  Google Scholar 

  10. Li, X., Pfister, T., Huang, X., Zhao, G., Pietikainen, M.: A spontaneous micro-expression database: inducement, collection and baseline. In: 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) (2013)

  11. Shreve, M., Godavarthy, S., Goldgof, D., Sarkar, S.: Macro- and micro-expression spotting in long videos using spatio-temporal strain. IEEE Comput. Soc. 51–56 (2011)

  12. Davison, A., Merghani, W., Lansley, C., Ng, C.C., Yap, M.H.: Objective micro-facial movement detection using FACS-based regions and baseline evaluation. In: IEEE International Conference on Automatic Face & Gesture, pp. 642–649 (2018)

  13. Ekman, P., Friesen, W.V.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)

    Google Scholar 

  14. Duque, C.A., Alata, O., Emonet, R., Legrand, A.C., Konik, H.: Micro-expression spotting using the riesz pyramid. In: IEEE Winter Conference on Applications of Computer Vision. (2018)

  15. Liong, S.T., See, J., Wong, K.S., Ngo, A., Phan, R.: Automatic apex frame spotting in micro-expression database. In: IAPR Asian Conference on Pattern Recognition, pp. 665–669 (2015)

  16. Zhang, Z., Chen, T., Meng, H., Liu, G., Fu, X.: SMEConvNet: A convolutional neural network for spotting spontaneous facial micro-expression from long videos. IEEE Access 6, 71143–71151 (2018)

    Article  Google Scholar 

  17. Pfister, T., Li, X.B., Zhao, G., Pietikäinen, M.: Recognising spontaneous facial micro-expressions. In: 2011 International Conference on Computer Vision, pp. 1449–1456 (2011)

  18. Timothy, F.: Active shape models-their training and application. Comput. Vis. Underst. 61, 38–59 (1995)

    Article  Google Scholar 

  19. Zhou, Z., Zhao, G., Pietikäinen, M.: Towards a practical lipreading system. In: The 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, Colorado Springs, CO, USA, 20–25 June 2011. (2011)

  20. Polikovsky, S., Kameda, Y., Ohta, Y.: Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In: 3rd International Conference on Imaging for Crime Detection and Prevention (ICDP), pp. 1–6 (2009)

  21. Zong, Y., Huang, X., Zheng, W., Cui, Z., Zhao, G.: Learning from hierarchical spatiotemporal descriptors for micro-expression recognition. IEEE Trans. Multimedia 20(11), 3160–3172 (2018)

    Article  Google Scholar 

  22. Patel, D., Hong, X., Zhao, G.: Selective deep features for micro-expression recognition. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2258–2263 (2016)

  23. Khor, H., See, J., Phan, R.C.W., Lin, W.: Enriched long-term recurrent convolutional network for facial micro-expression recognition. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 667–674 (2018)

  24. Peng, M., Wu, Z., Zhang, Z., Chen, T.: From macro to micro expression recognition: deep learning on small datasets using transfer learning. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 657–661 (2018)

  25. Xia, Z., Hong, X., Gao, X., Feng, X., Zhao, G.: Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions. IEEE Trans. Multimedia 22(3), 626–640 (2020)

    Article  Google Scholar 

  26. Liong, S., Gan, Y.S., See, J., Khor, H., Huang, Y.: Shallow triple stream three-dimensional CNN (STSTNet) for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2019), pp. 1–5 (2019)

  27. Reddy, S.P.T., Karri, S.T., Dubey, S.R., Mukherjee, S.: Spontaneous facial micro-expression recognition using 3D spatiotemporal convolutional neural networks. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2019)

  28. Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)

    Article  Google Scholar 

  29. Krizhevsky, A., Sutskever, I., Hinton G.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Article  Google Scholar 

  30. Geoffrey, H., Nitish, S., Kevin, S.S.: Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. (2012)

  31. Huang, X., Wang, S., Liu, X., Zhao, G., Feng, X., Pietikäinen, M.: Spontaneous facial micro-expression recognition using discriminative spatiotemporal local binary pattern with an improved integral projection. In: International Conference on Computer Vision(ICCV) Workshop, pp. 1–9 (2015)

  32. Khor, H., See, J., Liong, S., Phan, R.C.W., Lin, W.: Dual-stream Shallow Networks for Facial Micro-expression Recognition. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 36–40 (2019)

  33. Zhi, R., Xu, H., Wan, M., Li, T.: Combining 3D convolutional neural networks with transfer learning by supervised pre-training for facial micro-expression recognition. IEICE Trans. Inf. Syst. 102, 1054–1064 (2019)

    Article  Google Scholar 

  34. Cen, S.X., Yu, Y., Yan, G., Yu, M., Yang, Q.: Sparse spatiotemporal descriptor for micro-expression recognition using enhanced local cube binary pattern. Sensors 20(16), 4437 (2020)

    Article  Google Scholar 

  35. Song, B., Li, K., Zong, Y., Zhu, J., Zheng, W., Shi, J., Zhao, L.: Recognizing spontaneous micro-expression using a three-stream convolutional neural network. IEEE Access 7, 184537–184551 (2019)

    Article  Google Scholar 

  36. Verma, M., Vipparthi, S.K., Singh, G., Murala, S.: LEARNet: Dynamic imaging network for micro expression recognition. IEEE Trans. Image Process. 29, 1618–1627 (2020)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work is supported partially by the project of Jilin Provincial Science and Technology Department under the Grant 20180201003GX and the project of Jilin province development and reform commission under the Grant 2019C053-4.The authors gratefully acknowledge the helpful comments and suggestions of the reviewers, which have improved the presentation.

Funding

This research was funded by the project of Jilin Provincial Science and Technology Department under the Grant 20180201003GX and the APC was funded by Grant 20180201003GX too.

Author information

Authors and Affiliations

  1. School of Information Science and Technology, Northeast Normal University, Changchun, China

    Shuhua Liu, Yushu Ren, Lanting Li & Xiaoyu Sun

  2. School of Computer Science, Changchun University of Science and Technology, Changchun, China

    Yu Song

  3. Center for Machine Vision and Security Research, Kennesaw State University, Marietta, USA

    Chih-Cheng Hung

Authors
  1. Shuhua Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yushu Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lanting Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoyu Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yu Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Chih-Cheng Hung
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

This study was completed by the co-authors. SL conceived the research and wrote the draft. The major experiments and analyses were undertaken by YR and YS. LL and XS were responsible for data processing and drawing figures. C-CH edited and reviewed the paper. All authors have read and approved the final manuscript.

Corresponding author

Correspondence to Chih-Cheng Hung.

Ethics declarations

Conflict of interest

Authors declare no conflicts of interest.

Additional information

Communicated by I. Bartolini.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, S., Ren, Y., Li, L. et al. Micro-expression recognition based on SqueezeNet and C3D. Multimedia Systems 28, 2227–2236 (2022). https://doi.org/10.1007/s00530-022-00949-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-022-00949-z

Keywords

Search

Navigation