Skip to main content
SpringerLink
Log in

Cross-database Micro Expression Recognition Based on Apex Frame Optical Flow and Multi-head Self-attention

  • Conference paper
  • First Online:
Parallel Architectures, Algorithms and Programming (PAAP 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1362))

Abstract

With the rise of deep learning in the field of compute vision, more and more researchers began to use CNNs for automatic recognition of micro-expressions, and achieved better performance than traditional methods. However, there is a overfitting problem with these methods. CNN focuses on local information and loses the global semantic information in the image. In this paper, we propose a novel micro-expression recognition method based on Apex frame and multi-head self-attention enhanced convolutional network. It consists of three parts: (1) The pre-processing part performs face detection, face alignment, cutting into uniform size and positioning Apex frames on micro-expressions; (2) Optical flow calculation of Apex frames, calculates TVL1 optical flow from Onset frames to Apex frames, and obtains horizontal and vertical optical flow component images; (3) A shallow network (AACNet) is designed to extract and classify the optical flow image components obtained in (2). The results has greatly improved over the benchmark method (LBP-TOP). The state-of-the-art results were achieved on the MEGC2019 database (UF1: 0.7572, UAR: 0.7564).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ekman, P., Friesen, W.V.: Nonverbal leakage and clues to deception. Psychiatry 32(1), 88–106 (1969)

    Article  Google Scholar 

  2. Ekman, P.: Darwin, deception, and facial expression. Ann. N. Y. Acad. Sci. 1000(1), 205–221 (2003)

    Article  Google Scholar 

  3. Gan, Y.S., Liong, S.T., Yau, W.C., et al.: Off-apexnet on micro-expression recognition system. Sig. Process. Image Commun. 74, 129–139 (2019)

    Article  Google Scholar 

  4. Pfister, T., Li, X., Zhao, G., et al.: Recognising spontaneous facial micro-expressions. In: 2011 International Conference on Computer Vision, pp. 1449–1456. IEEE (2011)

    Google Scholar 

  5. Liu, Y.J., Zhang, J.K., Yan, W.J., et al.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. 7(4), 299–310 (2015)

    Article  Google Scholar 

  6. Xu, F., Zhang, J., Wang, J.Z.: Microexpression identification and categorization using a facial dynamics map. IEEE Trans. Affect. Comput. 8(2), 254–267 (2017)

    Article  Google Scholar 

  7. Liong, S.T., See, J., Wong, K.S., et al.: Less is more: micro-expression recognition from video using apex frame. Sig. Process. Image Commun. 62, 82–92 (2018)

    Article  Google Scholar 

  8. Zhou, Z., Zhao, G., Pietikäinen, M.: Towards a practical lipreading system. In: CVPR 2011, pp. 137–144. IEEE (2011)

    Google Scholar 

  9. Peng, M., Wu, Z., Zhang, Z., et al.: From macro to micro expression recognition: deep learning on small datasets using transfer learning. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 657–661. IEEE (2018)

    Google Scholar 

  10. Wang, S.J., Li, B.J., Liu, Y.J., et al.: Micro-expression recognition with small sample size by transferring long-term convolutional neural network. Neurocomputing 312, 251–262 (2018)

    Article  Google Scholar 

  11. Zhi, R., Xu, H., Wan, M., et al.: Combining 3D convolutional neural networks with transfer learning by supervised pre-training for facial micro-expression recognition. IEICE Trans. Inf. Syst. 102(5), 1054–1064 (2019)

    Article  Google Scholar 

  12. Zhou, L., Mao, Q., Xue, L.: Cross-database micro-expression recognition: a style aggregated and attention transfer approach. In: 2019 IEEE International Conference on Multimedia & Expo Workshops (ICMEW), pp. 102–107. IEEE (2019)

    Google Scholar 

  13. Yap, M.H., See, J., Hong, X., et al.: Facial micro-expressions grand challenge 2018 summary. In: 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 675–678. IEEE (2018)

    Google Scholar 

  14. Yan, W.J., Li, X., Wang, S.J., et al.: CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1), e86041 (2014)

    Article  Google Scholar 

  15. Davison, A.K., Lansley, C., Costen, N., et al.: SAMM: a spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(1), 116–129 (2016)

    Article  Google Scholar 

  16. See, J., Yap, M.H., Li, J., et al.: MEGC 2019-the second facial micro-expressions grand challenge. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5. IEEE (2019)

    Google Scholar 

  17. Megvii, I.: Face++ research toolkit (2013)

    Google Scholar 

  18. King, D.E.: Dlib-ml: a machine learning toolkit. J. Mach. Learn. Res. 10(Jul), 1755–1758 (2009)

    Google Scholar 

  19. Liong, S.T., Gan, Y.S., Zheng, D., et al.: Evaluation of the spatio-temporal features and GAN for micro-expression recognition system. J. Signal Process. Syst. 1–21 (2020)

    Google Scholar 

  20. Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L\(^{1}\) optical flow. In: Hamprecht, F.A., Schnörr, C., Jähne, B. (eds.) DAGM 2007. LNCS, vol. 4713, pp. 214–223. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74936-3_22

    Chapter  Google Scholar 

  21. Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)

    Google Scholar 

  22. Bello, I., Zoph, B., Vaswani, A., et al.: Attention augmented convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3286–3295 (2019)

    Google Scholar 

  23. Peng, M., Wang, C., Bi, T., et al.: A novel apex-time network for cross-dataset micro-expression recognition. In: 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 1–6. IEEE (2019)

    Google Scholar 

  24. Van Quang, N., Chun, J., Tokuyama, T.: CapsuleNet for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–7. IEEE (2019)

    Google Scholar 

  25. Zhou, L., Mao, Q., Xue, L.: Dual-inception network for cross-database micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5. IEEE (2019)

    Google Scholar 

  26. Liong, S.T., Gan, Y.S., See, J., et al.: Shallow triple stream three-dimensional CNN (STSTNet) for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5. IEEE (2019)

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (No. U1603115),National key R&Dplan project (2017YF C0820702-3) and National Engineering Laboratory for Public Safety Risk Perception and Control by Big Data (XJ201810101).

Author information

Authors and Affiliations

  1. College of Information Science and Engineering, Xinjiang University, Urumqi, Xinjiang, China

    Jiebin Wen, Wenzhong Yang, LieJun Wang, Wenyu Wei, Sixiang Tan & Yongzhi Wu

  2. Key Laboratory of Multilingual Information Technology in Xinjiang Uygur Autonomous Region, Xinjiang University, Urumqi, Xinjiang, China

    Jiebin Wen, Wenzhong Yang, LieJun Wang, Wenyu Wei, Sixiang Tan & Yongzhi Wu

Authors
  1. Jiebin Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wenzhong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. LieJun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenyu Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sixiang Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yongzhi Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wenzhong Yang .

Editor information

Editors and Affiliations

  1. Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China

    Li Ning

  2. Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China

    Vincent Chau

  3. The University of Hong Kong, Hong Kong, China

    Francis Lau

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wen, J., Yang, W., Wang, L., Wei, W., Tan, S., Wu, Y. (2021). Cross-database Micro Expression Recognition Based on Apex Frame Optical Flow and Multi-head Self-attention. In: Ning, L., Chau, V., Lau, F. (eds) Parallel Architectures, Algorithms and Programming. PAAP 2020. Communications in Computer and Information Science, vol 1362. Springer, Singapore. https://doi.org/10.1007/978-981-16-0010-4_12

Download citation

  • DOI: https://doi.org/10.1007/978-981-16-0010-4_12

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-16-0009-8

  • Online ISBN: 978-981-16-0010-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation