Skip to main content

Advertisement

Springer Nature Link
Log in

LAENet for micro-expression recognition

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Micro-expression is an expression that reveals one’s true feelings and can be potentially applied in various domains such as healthcare, safety interrogation, and business negotiation. The micro-expression recognition is thus far being judged manually by psychologists and trained experts, which consumes a lot of human effort and time. Recently, the development of the deep learning network has proven promising performance in many computer vision-related tasks. Amongst, micro-expression recognition adopts the deep learning methodology to improve the feature learning capability and model generalization. This paper introduces a lightweight apex-based enhanced network that improves by extending one of the state-of-the-art, shallow triple stream three-dimensional CNN. Concretely, the network is first pre-trained with a macro-expression dataset to encounter the small data problem. The features were extracted from CASME II, SMIC, and SAMM datasets. Moreover, thorough recognition results comparison of the datasets are the optical flow-guided features. Besides, an eye masking technique is introduced to reduce noise interference such as eye blinking and glasses reflection issues. The results obtained have an accuracy of 79.19\(\%\) and an F1-score of 75.9\(\%\). Comprehensive experimentation had been conducted on the composite dataset that consists of is provided by comparing it with recent methods. Detailed qualitative and quantitative results are reported and discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Ekman, P.: Darwin, deception, and facial expression. Ann. N. Y. Acad. Sci. 1000(1), 205–221 (2003)

    Article  Google Scholar 

  2. Haggard, E.A., Isaacs, K.S.: Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of Research in Psychotherapy, pp. 154–165 (1966)

  3. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124 (1971)

    Article  Google Scholar 

  4. Ekman, P., O’Sullivan, M.: Who can catch a liar? Am. Psychol. 46(9), 913 (1991)

    Article  Google Scholar 

  5. Liong, S.-T., Gan, Y., See, J., Khor, H.-Q., Huang, Y.-C.: Shallow triple stream three-dimensional cnn (ststnet) for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5 (2019). IEEE

  6. Kharat, G.U., Dudul, S.V.: Emotion recognition from facial expression using neural networks. In: Human-Computer Systems Interaction, pp. 207–219 (2009)

  7. Rivera, A.R., Castillo, J.R., Chae, O.O.: Local directional number pattern for face analysis: Face and expression recognition. IEEE Trans. Image Process. 22(5), 1740–1752 (2012)

    Article  MathSciNet  Google Scholar 

  8. Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recogn. 29(1), 51–59 (1996)

    Article  Google Scholar 

  9. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

  10. Wang, Y., See, J., Phan, R.C.-W., Oh, Y.-H.: Efficient spatio-temporal local binary patterns for spontaneous facial micro-expression recognition. PloS one 10(5) (2015)

  11. Huang, X., Zhao, G., Hong, X., Zheng, W., Pietikäinen, M.: Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175, 564–578 (2016)

    Article  Google Scholar 

  12. Huang, X., Zhao, G.: Spontaneous facial micro-expression analysis using spatiotemporal local radon-based binary pattern. In: 2017 International Conference on the Frontiers and Advances in Data Science (FADS), pp. 159–164 (2017). IEEE

  13. Huang, X., Wang, S.-J., Liu, X., Zhao, G., Feng, X., Pietikäinen, M.: Discriminative spatiotemporal local binary pattern with revisited integral projection for spontaneous facial micro-expression recognition. IEEE Trans. Affect. Comput. 10(1), 32–47 (2017)

    Article  Google Scholar 

  14. Liong, S.-T., Phan, R.C.-W., See, J., Oh, Y.-H., Wong, K.: Optical strain based recognition of subtle emotions. In: 2014 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), pp. 180–184 (2014). IEEE

  15. Yan, W.-J., Wang, S.-J., Zhao, G., Li, X., Liu, Y.-J., Chen, Y.-H., Fu, X.: CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE 9, 86041 (2014). https://doi.org/10.1371/journal.pone.0086041

    Article  Google Scholar 

  16. Liong, S.-T., See, J., Phan, R.C.-W., Le Ngo, A.C., Oh, Y.-H., Wong, K.: Subtle expression recognition using optical strain weighted features. In: Asian Conference on Computer Vision, pp. 644–657 (2014)

  17. Happy, S., Routray, A.: Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Transactions on Affective Computing (2017)

  18. Liu, Y.-J., Zhang, J.-K., Yan, W.-J., Wang, S.-J., Zhao, G., Fu, X.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. 7(4), 299–310 (2015)

    Article  Google Scholar 

  19. Liong, S.-T., See, J., Phan, R.C.-W., Oh, Y.-H., Le Ngo, A.C., Wong, K., Tan, S.-W.: Spontaneous subtle expression detection and recognition based on facial strain. Signal Process. Image Commun. 47, 170–182 (2016)

    Article  Google Scholar 

  20. Lu, H., Kpalma, K., Ronsin, J.: Motion descriptors for micro-expression recognition. Signal Process. Image Commun. 67, 108–117 (2018)

    Article  Google Scholar 

  21. Liong, S.-T., See, J., Wong, K., Phan, R.C.-W.: Less is more: Micro-expression recognition from video using apex frame. Signal Process. Image Commun. 62, 82–92 (2018)

    Article  Google Scholar 

  22. Gu, K., Zhang, Y., Qiao, J.: Ensemble meta-learning for few-shot soot density recognition. IEEE Trans. Ind. Inf. 17(3), 2261–2270 (2020)

    Article  Google Scholar 

  23. Gu, K., Liu, H., Xia, Z., Qiao, J., Lin, W., Thalmann, D.: Pm. monitoring: Use information abundance measurement and wide and deep learning. IEEE Trans. Neural Netw. Learn. Syst. 32(10), 4278–4290 (2021)

    Article  Google Scholar 

  24. Gu, K., Xia, Z., Qiao, J., Lin, W.: Deep dual-channel neural network for image-based smoke detection. IEEE Trans. Multimedia 22(2), 311–323 (2019)

    Article  Google Scholar 

  25. Gu, K., Xia, Z., Qiao, J., Lin, W.: Deep dual-channel neural network for image-based smoke detection. IEEE Trans. Multimedia 22(2), 311–323 (2019)

    Article  Google Scholar 

  26. Gu, K., Zhai, G., Yang, X., Zhang, W., Chen, C.W.: Automatic contrast enhancement technology with saliency preservation. IEEE Trans. Circuits Syst. Video Technol. 25(9), 1480–1494 (2014)

    Google Scholar 

  27. Gu, K., Zhai, G., Lin, W., Yang, X., Zhang, W.: No-reference image sharpness assessment in autoregressive parameter space. IEEE Trans. Image Process. 24(10), 3218–3231 (2015)

    Article  MathSciNet  Google Scholar 

  28. Patel, D., Hong, X., Zhao, G.: Selective deep features for micro-expression recognition. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2258–2263 (2016). IEEE

  29. Li, X., Pfister, T., Huang, X., Zhao, G., Pietikäinen, M.: A spontaneous micro-expression database: Inducement, collection and baseline. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–6 (2013). IEEE

  30. Li, J., Wang, Y., See, J., Liu, W.: Micro-expression recognition based on 3d flow convolutional neural network. Pattern Anal. Appl. 22(4), 1331–1339 (2019)

    Article  MathSciNet  Google Scholar 

  31. Wang, S.-J., Li, B.-J., Liu, Y.-J., Yan, W.-J., Ou, X., Huang, X., Xu, F., Fu, X.: Micro-expression recognition with small sample size by transferring long-term convolutional neural network. Neurocomputing 312, 251–262 (2018)

    Article  Google Scholar 

  32. Liong, S.-T., See, J., Wong, K., Le Ngo, A.C., Oh, Y.-H., Phan, R.: Automatic apex frame spotting in micro-expression database. In: 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), pp. 665–669 (2015). IEEE

  33. Gan, Y., Liong, S.-T., Yau, W.-C., Huang, Y.-C., Tan, L.-K.: Off-apexnet on micro-expression recognition system. Signal Process. Image Commun. 74, 129–139 (2019)

    Article  Google Scholar 

  34. Li, Y., Huang, X., Zhao, G.: Can micro-expression be recognized based on single apex frame? In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 3094–3098 (2018). IEEE

  35. Liong, S.-T., See, J., Wong, K., Phan, R.C.-W.: Automatic micro-expression recognition from long video using a single spotted apex. In: Asian Conference on Computer Vision, pp. 345–360 (2016)

  36. Khor, H.-Q., See, J., Liong, S.-T., Phan, R.C., Lin, W.: Dual-stream shallow networks for facial micro-expression recognition. In: 2019 IEEE International Conference on Image Processing (ICIP), pp. 36–40 (2019). IEEE

  37. Liong, S.-T., Gan, Y., Zheng, D., Li, S.-M., Xu, H.-X., Zhang, H.-Z., Lyu, R.-K., Liu, K.-H.: Evaluation of the spatio-temporal features and gan for micro-expression recognition system. J. Signal Process. Syst., 1–21 (2020)

  38. Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: Samm: A spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(1), 116–129 (2016)

    Article  Google Scholar 

  39. Qu, F., Wang, S.-J., Yan, W.-J., Li, H., Wu, S., Fu, X.: Cas (me) \(\hat{2}\): A database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Trans. Affect. Comput. 9(4), 424–436 (2017)

    Article  Google Scholar 

  40. Liu, Y., Du, H., Zheng, L., Gedeon, T.: A neural micro-expression recognizer. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–4 (2019). IEEE

  41. Zhou, L., Mao, Q., Xue, L.: Dual-inception network for cross-database micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5 (2019). IEEE

  42. Van Quang, N., Chun, J., Tokuyama, T.: Capsulenet for micro-expression recognition. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–7 (2019). IEEE

  43. Goh, K.M., Ng, C.H., Lim, L.L., Sheikh, U.U.: Micro-expression recognition: an updated review of current trends, challenges and solutions. Vis. Comput. 36(3), 445–468 (2020)

    Article  Google Scholar 

  44. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): A complete dataset for action unit and emotion-specified expression. In: 2010 Ieee Computer Society Conference on Computer Vision and Pattern Recognition-workshops, pp. 94–101 (2010). IEEE

  45. Asthana, A., Zafeiriou, S., Cheng, S., Pantic, M.: Robust discriminative response map fitting with constrained local models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3444–3451 (2013)

  46. See, J., Yap, M.H., Li, J., Hong, X., Wang, S.-J.: Megc 2019–the second facial micro-expressions grand challenge. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–5 (2019). IEEE

  47. Yan, W.-J., Wu, Q., Liu, Y.-J., Wang, S.-J., Fu, X.: Casme database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–7 (2013). IEEE

  48. Liong, S.-T., Gan, Y., Yau, W.-C., Huang, Y.-C., Ken, T.L.: Off-apexnet on micro-expression recognition system. arXiv preprint arXiv:1805.08699 (2018)

  49. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Adv. Neural Inf. Process. Syst. 25 (2012)

  50. Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and\(<\) 0.5 mb model size. arXiv preprint arXiv:1602.07360 (2016)

  51. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

  52. Simonyan, K., Zisserman., A.: Very deep convolutional networks for large-scale image recognition. arXiv (2014)

  53. Zhao, S., Tao, H., Zhang, Y., Xu, T., Zhang, K., Hao, Z., Chen, E.: A two-stage 3d cnn based learning method for spontaneous micro-expression recognition. Neurocomputing 448, 276–289 (2021)

    Article  Google Scholar 

  54. Liu, K.-H., Jin, Q.-S., Xu, H.-C., Gan, Y.-S., Liong, S.-T.: Micro-expression recognition using advanced genetic algorithm. Signal Process. Image Commun. 93, 116153 (2021)

    Article  Google Scholar 

  55. Ekman, P.: Facial expression and emotion. Am. Psychol. 48(4), 384 (1993)

    Article  Google Scholar 

  56. Esposito, A.: The amount of information on emotional states conveyed by the verbal and nonverbal channels: Some perceptual data. In: Progress in Nonlinear Speech Processing, pp. 249–268 (2007)

  57. Shreve, M., Godavarthy, S., Goldgof, D., Sarkar, S.: Macro-and micro-expression spotting in long videos using spatio-temporal strain. In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG), pp. 51–56 (2011). IEEE

  58. Shreve, M., Brizzi, J., Fefilatyev, S., Luguev, T., Goldgof, D., Sarkar, S.: Automatic expression spotting in videos. Image Vis. Comput. 32(8), 476–486 (2014)

    Article  Google Scholar 

  59. Husák, P., Cech, J., Matas, J.: Spotting facial micro-expressions “in the wild”. In: 22nd Computer Vision Winter Workshop (Retz), pp. 1–9 (2017)

Download references

Funding

This work was funded by Ministry of Science and Technology (MOST) (Grant Numbers: MOST 111-2221-E-035-059-MY3 and MOST 110-2221-E-035-052-)

Author information

Authors and Affiliations

  1. School of Architecture, Feng Chia University, Taichung, 40724, Taiwan, R.O.C.

    Y. S. Gan

  2. School of Electrical and Computer Engineering, Purdue University, West Lafayette, 47906, USA

    Sung-En Lien & Yi-Chen Chiang

  3. Department of Electronic Engineering, Feng Chia University, Taichung, 40724, Taiwan, R.O.C.

    Sze-Teng Liong

Authors
  1. Y. S. Gan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sung-En Lien
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yi-Chen Chiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sze-Teng Liong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sze-Teng Liong.

Ethics declarations

Ethical approval

The authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest (such as honoraria; educational grants; participation in speakers’ bureaus; membership, employment, consultancies, stock ownership, or other equity interest; and expert testimony or patent-licensing arrangements), or non-financial interest (such as personal or professional relationships, affiliations, knowledge or beliefs) in the subject matter or materials discussed in this manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gan, Y.S., Lien, SE., Chiang, YC. et al. LAENet for micro-expression recognition. Vis Comput 40, 585–599 (2024). https://doi.org/10.1007/s00371-023-02803-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-02803-3

Keywords