Skip to main content
SpringerLink
Log in

Micro-expression Recognition Based on Dual-Branch Swin Transformer Network

  • Conference paper
  • First Online:
Advanced Intelligent Computing Technology and Applications (ICIC 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14087))

Included in the following conference series:

  • 962 Accesses

Abstract

Micro-expression (ME) refers to the facial expression that flashes instantly and can reveal the real feelings and emotions of people. Compared with ordinary facial expressions, ME is not easy to be identified due to its short duration and inconspicuous performance. This paper uses Swin Transformer as the main network and dual-branch as the main framework to extract the temporal and spatial features for micro-expression recognition (MER). The first branch uses optical flow operator to preprocess the ME sequences, and the resulting optical flow maps are fed into the first Swin Transformer to extract motion feature information. The second branch directly sends the apex frame in one ME clip to the second Swin Transformer to learn the spatial feature. Finally, the feature flows from the two branches are fused to implement the final MER task. Extensive experimental comparisons on three widely used public ME benchmarks show that the proposed method is superior to the-state-of-the-art MER approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Hong, T., Longjiao, Z., Sen, F.: Micro-expression recognition based on optical flow method and pseudo-3D residual network. J. Signal Process. 38(5), 13–21 (2022)

    Google Scholar 

  2. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

  3. Polikovsky, S., Kameda, Y., Ohta, Y.: Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In: International Conference on Crime Detection & Prevention. IET (2010)

    Google Scholar 

  4. Li, X., Hong, X., Moilanen, A., et al.: Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans. Affect. Comput. 9(4), 563–577 (2018)

    Article  Google Scholar 

  5. Jin, Y., Kai, J., et al.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. 7(4), 299–310 (2016)

    Article  Google Scholar 

  6. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  7. Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)

    Article  Google Scholar 

  8. Szegedy, C., Liu, W., Jia, Y.: Going deeper with convolutions. In; IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

    Google Scholar 

  9. He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In; IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  10. Liu, Z., Lin, Y., Cao, Y., et al.: Swin transformer: Hierarchical vision transformer using shifted windows. In; IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)

    Google Scholar 

  11. Li, X., Pfister, T., Huang, X., et al: A spontaneous micro-expression database: inducement, collection and baseline. In: 2013 10th IEEE International Conference and Workshops on Automatic face and gesture recognition (FG). IEEE, pp. 1–6 (2013)

    Google Scholar 

  12. WenJing, Y., Xiaobai, L., Su-Jing, W.: CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9(1) (2014)

    Google Scholar 

  13. Davison, A.K., Lansley, C., Costen, N., Tan, K.: SAMM: a spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(99), 116–129 (2018)

    Article  Google Scholar 

  14. Zhenyi, L., Renhe, C., Yurong, Q.: CNN real-time micro-expression recognition algorithm based on dilated convolution. Appl. Res. Comput. 37(12), 5–13 (2020)

    Google Scholar 

  15. Gajjala, V.R., et al.: MERANet: facial micro-expression recognition using 3D residual attention network. In: The Twelfth Indian Conference on Computer Vision, Graphics and Image Processing, pp. 1–10 (2021)

    Google Scholar 

  16. Xue, F., Wang, Q., Guo, G.: Transfer: Learning relation-aware facial expression representations with transformers. In: IEEE/CVF International Conference on Computer Vision, pp. 3601–3610 (2021)

    Google Scholar 

  17. Zhang, L., Hong, X., Arandjelović, O., et al.: Short and long range relation based spatio-temporal transformer for micro-expression recognition. IEEE Trans. Affect. Comput. 13(4), 1973–1985 (2022)

    Article  Google Scholar 

  18. Zhao, X., Lv, Y., Huang, Z.: multimodal fusion-based swin transformer for facial recognition micro-expression recognition. In: 2022 IEEE International Conference on Mechatronics and Automation (ICMA). IEEE, pp. 780–785 (2022)

    Google Scholar 

  19. Zhu, J., Zong, Y., Chang, H., et al.: A sparse-based transformer network with associated spatiotemporal feature for micro-expression recognition. IEEE Signal Process. Lett. 29, 2073–2077 (2022)

    Article  Google Scholar 

  20. Khor, H.Q., See, J., Liong, S.T., et al.: Dual-stream shallow networks for facial micro-expression recognition. In: 2019 IEEE International Conference on Image Processing (ICIP). IEEE, pp. 36–40 (2019)

    Google Scholar 

  21. Chongyang, W., Min, P., Tao, B., Tong, C.: Micro-attention for micro-expression recognition. Neurocomputing 410, 354–362 (2020)

    Article  Google Scholar 

  22. Bo, S., Siming, C., Dongliang, L., Jun, H., Lejun, Y.: Dynamic micro-expression recognition using knowledge distillation. IEEE Trans. Affect. Comput. 13, 1037–1043 (2020)

    Google Scholar 

  23. Nie, X., Takalkar, M.A., Duan, M., et al.: GEME: dual-stream multi-task GEnder-based micro-expression recognition. Neurocomputing 427, 13–28 (2021)

    Article  Google Scholar 

  24. Liu, Z., Ning, J., Cao, Y., et al.: Video swin transformer. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3202–3211 (2022)

    Google Scholar 

  25. Hong, J., Lee, C., Jung, H.: Late fusion-based video transformer for facial micro-expression recognition. Appl. Sci. 12(3) (2022)

    Google Scholar 

Download references

Acknowledgements

This paper is supported by the Natural Science Foundation of Jiangxi Province of China (No. 20224ACB202011), the National Nature Science Foundation of China (No. 61861020) and the Jiangxi Province Graduate Innovation Special Fund Project (No. YC2022-s790).

Author information

Authors and Affiliations

  1. Key Lab of Optic-Electronic and Communication, Jiangxi Science and Technology Normal University, 605 Fenglin Avenue, Nanchang, 330031, China

    Zhihua Xie & Chuwei Zhao

Authors
  1. Zhihua Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chuwei Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhihua Xie .

Editor information

Editors and Affiliations

  1. Department of Computer Science, Eastern Institute of Technology, Zhejiang, China

    De-Shuang Huang

  2. University of Wollongong, North Wollongong, NSW, Australia

    Prashan Premaratne

  3. Zhengzhou University of Light Industry, Zhengzhou, China

    Baohua Jin

  4. Zhong Yuan University of Technology, Zhengzhou, China

    Boyang Qu

  5. University of Ulsan, Ulsan, Korea (Republic of)

    Kang-Hyun Jo

  6. Department of Computer Science, Liverpool John Moores University, Liverpool, UK

    Abir Hussain

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xie, Z., Zhao, C. (2023). Micro-expression Recognition Based on Dual-Branch Swin Transformer Network. In: Huang, DS., Premaratne, P., Jin, B., Qu, B., Jo, KH., Hussain, A. (eds) Advanced Intelligent Computing Technology and Applications. ICIC 2023. Lecture Notes in Computer Science, vol 14087. Springer, Singapore. https://doi.org/10.1007/978-981-99-4742-3_45

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-4742-3_45

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-4741-6

  • Online ISBN: 978-981-99-4742-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation