Skip to main content
Springer Nature Link
Log in

3DCNN Backed Conv-LSTM Auto Encoder for Micro Facial Expression Video Recognition

  • Conference paper
  • First Online:
Machine Learning and Intelligent Communications (MLICOM 2021)

Abstract

Facial Micro-Expression recognition in the field of emotional information processing has become an inexorable necessity for its exotic attributes. It is a non-verbal, spontaneous, and involuntary leakage of true emotion in disguise of most expressive intentional prototypical facial expressions. However, it persists only for a split-second duration and possesses fainted facial muscle movements that make the recognition task more difficult with naked eyes. Besides, there are a limited number of video samples and wide-span domain shifting among datasets. Considering these challenges, several video-based works have been done to improve the classification accuracy but still lack high accuracy. This works addresses these issues and presents an approach with a deep 3D Convolutional Residual Neural Network as a backbone followed by a Long-Short-Term-Memory auto-encoder with 2D convolutions model for automatic Spatio-temporal feature extractions, fine-tuning, and classifications from videos. Also, we have done transfer learning on three standard macro-expression datasets to reduce over-fitting. Our work has shown a significant accuracy gain with extensive experiments on composite video samples from five publicly available micro-expression benchmark datasets, CASME, CASMEII, CAS(ME)2, SMIC, and SAMM. This outweighs the state-of-the-art accuracy. It is the first attempt to work with five datasets and rational implication of LSTM auto-encoder for micro-expression recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Zhang, M., Fu, Q., Chen, Y.H., Fu, X.: Emotional context influences micro-expression recognition. PLoS ONE 9(4), 95018 (2014)

    Article  Google Scholar 

  2. Yan, W.-J., Wu, Q., Liang, J., Chen, Y.-H., Fu, X.: How Fast are the leaked facial expressions: the duration of micro-expressions. J. Nonverbal Behav. 37(4), 217–230 (2013). https://doi.org/10.1007/s10919-013-0159-8

    Article  Google Scholar 

  3. Takalkar, M., Xu, M., Wu, Q., Chaczko, Z.: A survey: facial micro-expression recognition. Multim. Tools Appl. 77(15), 19301–19325 (2017). https://doi.org/10.1007/s11042-017-5317-2

    Article  Google Scholar 

  4. Ekman, P., Cohn, J.F., Ambadar, Z.: Observer-based measurement of facial expression with the facial action coding system. Handbook Emot. Elicit. Assess. 1(3), 203–221 (2007)

    Google Scholar 

  5. Goh, K.M., Ng, C.H., Lim, L.L., Sheikh, U.U.: Micro-expression recognition: an updated review of current trends, challenges and solutions. Vis. Comput. 36(3), 445–468 (2020). https://doi.org/10.1007/s00371-018-1607-6

    Article  Google Scholar 

  6. Pfister, T., Li, X., Zhao, G., Pietikäinen, M.: Recognising spontaneous facial micro-expressions. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1449–1456 (2011)

    Google Scholar 

  7. Wang, Y., See, J., Phan, R.-W., Oh, Y.-H.: LBP with six intersection points: reducing redundant information in LBP-TOP for micro-expression recognition. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9003, pp. 525–537. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16865-4_34

    Chapter  Google Scholar 

  8. Zhao, G., Pietikäinen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

  9. Pietikinen, G.Z.M., Huang, X., Wang, S.J.: Facial micro_expression recognition using spatiotemporal local binary pattern with integral projection. In: ICCV Workshop on Computer Vision for Affective Computing, pp. 1–9 (2015)

    Google Scholar 

  10. Huang, X., Zhao, G., Hong, X., Zheng, W., Pietikäinen, M.: Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns. Neurocomputing 175(PartA), 564–578 (2015)

    Google Scholar 

  11. Huang, X., Wang, S.J., Liu, X., Zhao, G., Feng, X., Pietikainen, M.: Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans. Affect. Comput. 10(1), 32–47 (2017)

    Article  Google Scholar 

  12. Zong, Y., Huang, X., Zheng, W., Cui, Z., Zhao, G.: Learning from hierarchical spatiotemporal descriptors for micro-expression recognition. IEEE Trans. Multimed. 20(11), 3160–3172 (2018)

    Article  Google Scholar 

  13. Chaudhry, R., Ravichandran, A., Hager, G., Vidal, R.: Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions, pp. 1932–1939 (2009)

    Google Scholar 

  14. Liu, Y.J., Zhang, J.K., Yan, W.J., Wang, S.J., Zhao, G., Fu, X.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. 7(4), 299–310 (2016)

    Article  Google Scholar 

  15. Xu, F., Zhang, J., Wang, J.Z.: Microexpression identification and categorization using a facial dynamics map. IEEE Trans. Affect. Comput. 8(2), 254–267 (2017)

    Article  Google Scholar 

  16. Liong, S.T., See, J., Wong, K.S., Phan, R.C.W.: Less is more: micro-expression recognition from video using apex frame. Signal Process. Image Commun. 62, 82–92 (2018)

    Article  Google Scholar 

  17. Happy, S.L., Routray, A.: Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Trans. Affect. Comput. 10(3), 394–406 (2019)

    Article  Google Scholar 

  18. Polikovsky, S., Kameda, Y., Ohta, Y.: Facial micro-expressions recognition using high speed camera and 3D-Gradient descriptor. In: IET Seminar Digest, vol. 2009, no. 2 (2009)

    Google Scholar 

  19. Li, X., et al.: Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans. Affect. Comput. 9(4), 563–577 (2017)

    Article  Google Scholar 

  20. Patel, D., Hong, X., Zhao, G.: Selective deep features for micro-expression recognition. In: Proceedings - International Conference on Pattern Recognition, vol. 0, pp. 2258–2263 (2016)

    Google Scholar 

  21. Takalkar, M.A., Xu, M.: Image based facial micro-expression recognition using deep learning on small datasets. In: DICTA 2017 - 2017 International Conference on Digital Image Computing: Techniques and Applications, vol. 2017, pp. 1–7 (2017)

    Google Scholar 

  22. Mayya, V., Pai, R.M., Pai, M.M.M.: Combining temporal interpolation and DCNN for faster recognition of micro-expressions in video sequences. In: 2016 International Conference on Advances in Computing, Communications and Informatics, ICACCI 2016, pp. 699–703 (2016)

    Google Scholar 

  23. Peng, M., Wang, C., Chen, T., Liu, G., Xiaolan, F.: Dual temporal scale convolutional neural network for micro-expression recognition. Front. Psychol. 8 (2017). https://doi.org/10.3389/fpsyg.2017.01745

  24. Kim, D.H., Baddar, W.J., Jang, J., Ro, Y.M.: Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans. Affect. Comput. 10(2), 223–236 (2017)

    Article  Google Scholar 

  25. Wang, S.J., et al.: Micro-expression recognition with small sample size by transferring long-term convolutional neural network. Neurocomputing 312, 251–262 (2018)

    Article  Google Scholar 

  26. Li, Y., Huang, X., Zhao, G.: can micro-expression be recognized based on single apex frame? In: Proceedings - International Conference on Image Processing, ICIP, pp. 3094–3098 (2018)

    Google Scholar 

  27. Gan, Y.S., Liong, S.T., Yau, W.C., Huang, Y.C., Tan, L.K.: OFF-ApexNet on micro-expression recognition system. Signal Process. Image Commun. 74, 129–139 (2019)

    Article  Google Scholar 

  28. Khor, H.Q., See, J., Liong, S.T., Phan, R.C.W., Lin, W.: Dual-stream shallow networks for facial micro-expression recognition. In: Proceedings - International Conference on Image Processing, ICIP, vol. 2019, pp. 36–40 (2019)

    Google Scholar 

  29. Xia, Z., Feng, X., Hong, X., Zhao, G.: Spontaneous facial micro-expression recognition via deep convolutional network. In: 2018 8th International Conference on Image Processing Theory, Tools and Applications, IPTA 2018 – Proceedings (2019)

    Google Scholar 

  30. Xia, Z., Peng, W., Khor, H.Q., Feng, X., Zhao, G.: Revealing the invisible with model and data shrinking for composite-database micro-expression recognition. IEEE Trans. Image Process. 29, 8590–8605 (2020)

    Article  MATH  Google Scholar 

  31. Yang, B., Cheng, J., Yang, Y., Zhang, B., Li, J.: MERTA: micro-expression recognition with ternary attentions. Multim. Tools Appl. 80(11), 1–16 (2019). https://doi.org/10.1007/s11042-019-07896-4

    Article  Google Scholar 

  32. Li, X., Pfister, T., Huang, X., Zhao, G., Pietikainen, M.: A Spontaneous Micro-expression Database: Inducement, collection and baseline. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, FG 2013 (2013)

    Google Scholar 

  33. Yan, W.J., Wu, Q., Liu, Y.J., Wang, S.J., Fu, X.: CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, FG 2013 (2013)

    Google Scholar 

  34. Yan, W.J., et al.: CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9(1), e86041 (2014)

    Google Scholar 

  35. Qu, F., Wang, S.J., Yan, W.J., Li, H., Wu, S., Fu, X.: CAS(ME)2): a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Trans. Affect. Comput. 9(4), 424–436 (2018)

    Article  Google Scholar 

  36. Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: SAMM: a spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(1), 116–129 (2018)

    Article  Google Scholar 

  37. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops, CVPRW 2010, pp. 94–101 (2010)

    Google Scholar 

  38. Papachristou, C., Aifanti, A.D.N.: The MUG facial expression database. In: 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10, pp. 1–4 (2010)

    Google Scholar 

  39. Zhao, G., Huang, X., Taini, M., Li, S.Z., Pietikäinen, M.: Facial expression recognition from near-infrared videos. Image Vis. Comput. 29(9), 607–619 (2011)

    Article  Google Scholar 

  40. Tran, D., Wang, H., Torresani, L., Ray, J., Lecun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 6450–6459 (2018)

    Google Scholar 

  41. Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W., Woo, W.: Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. Adv. Neural Inf. Process. Syst. 2015, 802–810 (2015)

    Google Scholar 

  42. Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2D & 3d face alignment problem? (and a Dataset of 230,000 3D Facial Landmarks). In: Proceedings of the IEEE International Conference on Computer Vision, vol. 2017, pp. 1021–1030 (2017)

    Google Scholar 

  43. Zhang, S., Zhu, X., Lei, Z., Shi, H., Wang, X., Li, S.Z.: S3FD: Single Shot Scale-Invariant Face Detector. In: Proceedings of the IEEE International Conference on Computer Vision, vol. 2017, pp. 192–201 (2017)

    Google Scholar 

  44. W. Kay et al., “The Kinetics Human Action Video Dataset,” May 2017

    Google Scholar 

  45. Davison, A.K., Merghani, W., Yap, M.H.: Objective classes for micro-facial expression recognition. J. Imaging 4(10), 119 (2018)

    Article  Google Scholar 

  46. Van Quang, N., Chun, J., Tokuyama, T.: CapsuleNet for micro-expression recognition. In: Proceedings - 14th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2019 (2019)

    Google Scholar 

  47. Xia, B., Wang, W., Wang, S., Chen, E.: Learning from Macro-expression: a Micro-expression Recognition Framework. In: Proceedings of the 28th ACM International Conference on Multimedia, pp. 2936–2944 (2020)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Computer Science, Sichuan University, Chengdu, 610065, China

    Md. Sajjatul Islam, Jiancheng Lv, Adam Ahmed Qaid Mohammed & Yongsheng Sang

  2. TAL Education Group, Beijing, 100080, China

    Yuan Gao & Zhilong Ji

Authors
  1. Md. Sajjatul Islam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuan Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhilong Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiancheng Lv
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Adam Ahmed Qaid Mohammed
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yongsheng Sang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conflict of Interests.

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Funding.

This work was supported by the National Key R&D Program of China (Grant No. 2020AAA0104500), and was partially supported by Sichuan Science and Technology Major Project (Grant No. 2019ZDZX0006).

Corresponding author

Correspondence to Yongsheng Sang .

Editor information

Editors and Affiliations

  1. Jinhua Advanced Research Institute, Jinhua, China

    Xiaolin Jiang

Rights and permissions

Reprints and permissions

Copyright information

© 2022 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Islam, M.S., Gao, Y., Ji, Z., Lv, J., Mohammed, A.A.Q., Sang, Y. (2022). 3DCNN Backed Conv-LSTM Auto Encoder for Micro Facial Expression Video Recognition. In: Jiang, X. (eds) Machine Learning and Intelligent Communications. MLICOM 2021. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 438. Springer, Cham. https://doi.org/10.1007/978-3-031-04409-0_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-04409-0_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-04408-3

  • Online ISBN: 978-3-031-04409-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics