Skip to main content
SpringerLink
Log in
Book cover

International Conference on Neural Information Processing

ICONIP 2021: Neural Information Processing pp 76–87Cite as

DFFCN: Dual Flow Fusion Convolutional Network for Micro Expression Recognition

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 13110))

Abstract

Recently, micro-expression recognition (MER) has attracted much attention due to its wide application in various fields such as crime trials and psychotherapy. However, the short duration and subtle movement of facial muscles make it difficult to extract micro-expression features. In this article, we propose a Dual Flow Fusion Convolutional Network (DFFCN) that combines the learning flow and optical flow to capture spatiotemporal features. Specifically, we adopt a trainable Learning Flow Module to extract the frame-level motion characteristics, fused with the mask generated from hand-crafted optical flow, and finally predict the micro-expression. Additionally, to overcome the shortcomings of limited and imbalanced training samples, we propose a data augmentation strategy based on Generative Adversarial Network (GAN). Comprehensive experiments are conducted on three public micro-expression datasets: CASME II, SAMM and SMIC with Leave-One-Subject-Out (LOSO) cross-validation. The results demonstrated that our method achieves competitive performance when compared with the existing approaches, with the best UF1 (0.8452) and UAR (0.8465).

This research was supported by the National Natural Science Foundation of China under Project (Grant No. 61977045).

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Ekman, P., Friesen, W.V.: Nonverbal leakage and clues to deception. Psychiatry 32(1), 88–106 (1969)

    Article  Google Scholar 

  2. Haggard E.A., Isaacs K.S.: Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of Research in Psychotherapy. The Century Psychology Series, pp. 154–165. Springer, Boston (1966). https://doi.org/10.1007/978-1-4684-6045-2_14

  3. Ekman, R.: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, USA (1997)

    Google Scholar 

  4. Pumarola, A., Agudo, A., Martinez, A.M., Sanfeliu, A., Moreno-Noguer, F.: GANimation: anatomically-aware facial animation from a single image. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11214, pp. 835–851. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01249-6_50

    Chapter  Google Scholar 

  5. Kwon, H., Kim, M., Kwak, S., Cho, M.: MotionSqueeze: neural motion feature learning for video understanding. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12361, pp. 345–362. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58517-4_21

    Chapter  Google Scholar 

  6. Yan, W.-J., et al.: CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLOS ONE 9(1), e86041 (2014)

    Article  Google Scholar 

  7. Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: SAMM: a spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(1), 116–129 (2016)

    Article  Google Scholar 

  8. Li, X., Pfister, T., Huang, X., Zhao, G., Pietikäinen, M.: A spontaneous micro-expression database: inducement, collection and baseline. In: 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, FG 2013, pp. 1–6. IEEE (2013)

    Google Scholar 

  9. Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)

    Article  Google Scholar 

  10. Liong, S.-T., See, J., Wong, K., Phan, R.C.-W.: Less is more: micro-expression recognition from video using apex frame. Sig. Process. Image Commun. 62, 82–92 (2018)

    Article  Google Scholar 

  11. Khor, H.-Q., See, J., Phan, R.C.W., Lin, W.: Enriched long-term recurrent convolutional network for facial micro-expression recognition. In: FG 2018, pp. 667–674. IEEE (2018)

    Google Scholar 

  12. Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., Freeman, W.: Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. (TOG) 31(4), 1–8 (2012)

    Article  Google Scholar 

  13. Liu, Y., Du, H., Zheng, L., Gedeon, T.: A neural micro-expression recognizer. In: FG 2019, pp. 1–4. IEEE (2019)

    Google Scholar 

  14. Peng, M., Wang, C., Chen, T., Liu, G., Fu, X.: Dual temporal scale convolutional neural network for micro-expression recognition. Front. Psychol. 8, 1745 (2017)

    Article  Google Scholar 

  15. Yu, J., Zhang, C., Song, Y., Cai, W.: ICE-GAN: identity-aware and capsule-enhanced GAN for micro-expression recognition and synthesis. arXiv preprint arXiv:2005.04370 (2020)

  16. Le Ngo, A.C., Johnston, A., Phan, R.C.-W., See, J.: Micro-expression motion magnification: global Lagrangian vs. local Eulerian approaches. In: FG 2018, pp. 650–656. IEEE (2018)

    Google Scholar 

  17. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110–8119 (2020)

    Google Scholar 

  18. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR 2016, pp. 770–778 (2016)

    Google Scholar 

  19. Lee, J., Kim, D., Ponce, J., Ham, B.: SFNet: learning object-aware semantic correspondence. In: CVPR 2019, pp. 2278–2287 (2019)

    Google Scholar 

  20. See, J., Yap, M.H., Li, J., Hong, X., Wang, S.-J.: MEGC 2019 - the second facial micro-expressions grand challenge. In: FG 2019, pp. 1–5. IEEE (2019)

    Google Scholar 

  21. Van Quang, N., Chun, J., Tokuyama, T.: CapsuleNet for micro-expression recognition. In: FG 2019, pp. 1–7. IEEE (2019)

    Google Scholar 

  22. Gan, Y.S., Liong, S.-T., Yau, W.-C., Huang, Y.-C., Tan, L.-K.: OFF-ApexNet on micro-expression recognition system. Sig. Process. Image Commun. 74, 129–139 (2019)

    Article  Google Scholar 

  23. Zhou, L., Mao, Q., Xue, L.: Dual-inception network for cross-database micro-expression recognition. In: FG 2019, pp. 1–5. IEEE (2019)

    Google Scholar 

  24. Liong, S.-T., Gan, Y.S., See, J., Khor, H.-Q., Huang, Y.-C.: Shallow triple stream three-dimensional CNN (STSTNet) for micro-expression recognition. In: FG 2019, pp. 1–5. IEEE (2019)

    Google Scholar 

  25. Xia, Z., Peng, W., Khor, H.-Q., Feng, X., Zhao, G.: Revealing the invisible with model and data shrinking for composite-database micro-expression recognition. IEEE Trans. Image Process. 29, 8590–8605 (2020)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Shanghai Jiao Tong University, Shanghai, China

    Jinjie Chen, Yuzhuo Fu, YiBo Jin & Ting Liu

Authors
  1. Jinjie Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuzhuo Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. YiBo Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ting Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jinjie Chen .

Editor information

Editors and Affiliations

  1. Sampoerna University, Jakarta, Indonesia

    Teddy Mantoro

  2. Kyungpook National University, Daegu, Korea (Republic of)

    Minho Lee

  3. Sampoerna University, Jakarta, Indonesia

    Media Anugerah Ayu

  4. Murdoch University, Murdoch, WA, Australia

    Kok Wai Wong

  5. Universitas Indonesia, Depok, Indonesia

    Achmad Nizar Hidayanto

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, J., Fu, Y., Jin, Y., Liu, T. (2021). DFFCN: Dual Flow Fusion Convolutional Network for Micro Expression Recognition. In: Mantoro, T., Lee, M., Ayu, M.A., Wong, K.W., Hidayanto, A.N. (eds) Neural Information Processing. ICONIP 2021. Lecture Notes in Computer Science(), vol 13110. Springer, Cham. https://doi.org/10.1007/978-3-030-92238-2_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-92238-2_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-92237-5

  • Online ISBN: 978-3-030-92238-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation