Skip to main content
SpringerLink
Log in

Micro-expression spotting network based on attention and one-dimensional convolutional sliding window

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

In the field of computer vision, the research on micro-expression (ME) can be divided into two main tasks: ME spotting and ME recognition. ME spotting refers to finding the occurrence and interval of ME from video stream, which is an indispensable module for automatic ME analysis. In this paper, aiming at finding of the inaccurate location of MEs in long videos, we proposed a ME spotting network based on attention mechanism and one-dimensional (1D) convolution sliding window. In our proposed scheme, convolutional neural network (CNN), Bi-directional Long Short-Term Memory (BI-LSTM), and 1D convolution are used to extract features. The attention mechanism is used to highlight the key frames. 1D convolution with sliding window is applied to detect feature intervals, which are further combined with the intervals and judged as MEs to obtain the final ME spotting result. Simulation was done on CAS(ME)2 dataset. It is shown that the proposed algorithm outperforms other superior algorithms in terms of effectiveness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

The datasets analyzed during this study are available from the public data repositories at the website of http://casme.psych.ac.cn/casme/c3.

References

  1. Haggard, E.A., Isaacs, K.S.: Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. Meth Res Psychoth. (1966). https://doi.org/10.1007/978-1-4684-6045-2_14

    Article  Google Scholar 

  2. Ekman, P., Friesen, W.V.: Nonverbal leakage and clues to deception. Psychiatry 32(1), 88–106 (1969). https://doi.org/10.1080/00332747.1969.11023575

    Article  Google Scholar 

  3. Zhang, M., Zhao, K., Qu, F., Li, K., Fu, X.: brain activation in contrasts of micro-expression following emotional contexts. Front. Neurosci. 14, 329–329 (2020). https://doi.org/10.3389/fnins.2020.00329

    Article  Google Scholar 

  4. Curtis, D.A.: Deception detection and emotion recognition: Investigating FACE software. Psychother Res. 31(6), 802–816 (2021). https://doi.org/10.1080/10503307.2020.1836424

    Article  Google Scholar 

  5. Datz, F., Wong, G., Löffler-Stastka, H.: Interpretation and Working through Contemptuous Facial Micro-Expressions Benefits the Patient-Therapist Relationship. Int J Environ Res Public Health 16(24), 4901 (2019). https://doi.org/10.3390/ijerph16244901

    Article  Google Scholar 

  6. Jupe, L.M., Keatley, D.A.: Airport artificial intelligence can detect deception: or am i lying? Secur. J. 33(4), 622–635 (2019). https://doi.org/10.1057/s41284-019-00204-7

    Article  Google Scholar 

  7. Qu, F., Wang, S.-J., Yan, W.-J., Li, H., Wu, S., Fu, X.: CAS(ME)^2: A Database for Spontaneous Macro-expression and Micro-expression Spotting and Recognition. IEEE Trans. Affect. Comput. 9(4), 424–436 (2018). https://doi.org/10.1109/taffc.2017.2654440

    Article  Google Scholar 

  8. Tran, T., Vo, Q., Hong, X., Li, X., Zhao, G.: Micro-expression spotting: A new benchmark. Neurocomputing (Amsterdam) 443, 356–368 (2021). https://doi.org/10.1016/j.neucom.2021.02.022

    Article  Google Scholar 

  9. Moilanen, A., Zhao, G., Pietikainen, M.: Spotting rapid facial movements from videos using appearance-based feature difference analysis. Int Conf Pattern Recog (2014). https://doi.org/10.1109/ICPR.2014.303

    Article  Google Scholar 

  10. Davison, A.K., Yap, M.H., Lansley, C.: Micro-facial movement detection using individualised baselines and histogram-based descriptors. Int Conf Syst and Cybernet. (2015). https://doi.org/10.1109/smc.2015.326

    Article  Google Scholar 

  11. Li, X., Hong, X., Moilanen, A., Huang, X., Pfister, T., Zhao, G., Pietikainen, M.: Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. Trans Affect Comp. 9(4), 563–577 (2017). https://doi.org/10.1109/TAFFC.2017.2667642

    Article  Google Scholar 

  12. Wang, S.-J., Wu, S., Qian, X., Li, J., Fu, X.: A main directional maximal difference analysis for spotting facial movements from long-term videos. Neurocomputing 230, 382–389 (2017). https://doi.org/10.1016/j.neucom.2016.12.034

    Article  Google Scholar 

  13. Beh, K.X., Goh, K.M.: Micro-Expression spotting using facial landmarks. In CSPA. (2019). https://doi.org/10.1109/CSPA.2019.8696059

    Article  Google Scholar 

  14. He, Y., Xu, Z., Ma, L., Li, H.: Micro-expression spotting based on optical flow features. Patt recog lett 163, 57–63 (2022)

    Article  Google Scholar 

  15. Xia, Z., Feng, X., Peng, J., Peng, X., Zhao, G.: Spontaneous micro-expression spotting via geometric deformation modeling. Comput Vis Image Underst. 147, 87–94 (2016). https://doi.org/10.1016/j.cviu.2015.12.006

    Article  Google Scholar 

  16. Tran, T.K., Hong, X., Zhao, G.: Sliding Window Based Micro-expression Spotting: A Benchmark. Adv Conc Intell Vision Syst. (2017). https://doi.org/10.1007/978-3-319-70353-4_46

    Article  Google Scholar 

  17. Zhang, Z., Chen, T., Meng, H., Liu, G., Fu, X.: Smeconvnet a convolutional neural network for spotting spontaneous facial micro-expression from long videos. IEEE Access. 6, 7113 (2018)

    Google Scholar 

  18. Verburg, M., Menkovski, V.: Micro-expression detection in long videos using optical flow and recurrent neural networks. In Fg (2019). https://doi.org/10.1109/FG.2019.8756588

    Article  Google Scholar 

  19. Tran, T.-K., Vo, Q.-N., Hong, X., Zhao, G.: Dense prediction for micro-expression spotting based on deep sequence model. Electr Imag. 8, 401–411 (2019). https://doi.org/10.2352/issn.2470-1173.2019.8.imawm-401

    Article  Google Scholar 

  20. Chen, C.R., Fan, Q., Panda, R.: CrossViT: cross-attention multi-scale vision transformer for image classification. ICCV. (2021). https://doi.org/10.1109/ICCV48922.2021.00041

    Article  Google Scholar 

  21. Mao, X., Qi, G., Chen, Y., et al.: Towards robust vision transformer. EEE/CVF Conf Comp Vis Patt Recog. (2022). https://doi.org/10.1109/CVPR52688.2022.01173

    Article  Google Scholar 

  22. Zhou, Y., Song, Y., Chen, L., Chen, Y., Ben, X., Cao, Y.: A novel micro-expression detection algorithm based on BERT and 3DCNN. Image Vision Comp 119, 104378 (2022). https://doi.org/10.1016/j.imavis.2022.104378

    Article  Google Scholar 

  23. Li, Y., Yao, T., Pan, Y., Mei, T.: Contextual transformer networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 45(2), 1489–1500 (2023). https://doi.org/10.1109/TPAMI.2022.3164083

    Article  Google Scholar 

  24. Yao, T., Li, Y., Pan, Y., Wang, Y., Zhang, X., Mei, T.: Dual vision transformer. IEEE Trans. Pattern Anal. Mach. Intell. (2023). https://doi.org/10.1109/TPAMI.2023.3268446

    Article  Google Scholar 

  25. Krizhevsky, A., Sutskeve, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017). https://doi.org/10.1145/3065386

    Article  Google Scholar 

  26. Wang, C., Wang, Z.: Progressive Multi-Scale Vision Transformer for Facial Action Unit Detection. Front Neurorobot. 15, 824592 (2022). https://doi.org/10.3389/fnbot.2021.824592

    Article  Google Scholar 

  27. King, D.E.: Dlib-ml: a machine learning toolkit. J Mach Learn Res 10, 1755–1758 (2009)

    Google Scholar 

  28. Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. Lect Not Comp Sci (2003). https://doi.org/10.1007/3-540-45103-x_50

    Article  MATH  Google Scholar 

Download references

Acknowledgements

The authors would like to thank the anonymous reviewers for the constructive and valuable comments, which helped us improve this paper to its present form.

Funding

This work was supported in part by the Natural Science Foundation of Shandong Province under grant ZR2020MF004 and in part by the National Key R&D Program of China under grant2020YFC0833201.

Author information

Author notes

  1. Hongbo Xing and Guanqun Zhou have contributed equally to this work.

Authors and Affiliations

  1. School of Information Science and Engineering, Shandong University, Binhai Road, Qingdao, 266237, Shandong, China

    Hongbo Xing, Guanqun Zhou, Shusen Yuan, Youjun Jiang, Pinyong Geng, Yewen Cao, Yujun Li & Lei Chen

Authors
  1. Hongbo Xing
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guanqun Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shusen Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Youjun Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Pinyong Geng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yewen Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yujun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Lei Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by [Hongbo Xing] , [Guanqun Zhou] , [Shusen Yuan] , [Youjun Jiang] , [Pinyong Geng] , [Yewen Cao] , [Lei Chen] and [Yujun Li]. The first draft of the manuscript was written by [Hongbo Xing] and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding authors

Correspondence to Yewen Cao or Yujun Li.

Ethics declarations

Conflict of interest

Authors declare that there is no conflict of interest.

Additional information

Communicated by T. Yao.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xing, H., Zhou, G., Yuan, S. et al. Micro-expression spotting network based on attention and one-dimensional convolutional sliding window. Multimedia Systems 29, 2429–2437 (2023). https://doi.org/10.1007/s00530-023-01120-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-023-01120-y

Keywords

Search

Navigation