Skip to main content
SpringerLink
Log in

Hybrid Facial Regions Extraction for Micro-expression Recognition System

  • Published:
Journal of Signal Processing Systems Aims and scope Submit manuscript

Abstract

Micro-expressions can occur when a person attempts to conceal and suppress his true feelings and emotions, both deliberately or unconsciously.In recent years, facial micro-expression analysis has received tremendous attention in the field of psychology, media and computer vision. However, due to its subtlety and brief duration, development of automated micro-expression detection and recognition system are still great challenges in the field of computer vision. In this paper, we present a novel hybrid facial region extraction framework that combines heuristic and automatic approaches to better recognize spontaneous micro-expressions. Salient facial regions are statistically determined based on the occurrence frequency of facial action units instead of holistic utilization of the entire facial area. The regions were automatically selected according to the facial landmark coordinates. We tested on two recent publicly available datasets that provided sufficient samples while also fulfilling the criteria of being elicited spontaneously. To further confirm the reliability of the proposed method, two distinct feature extractors were employed to describe micro-expression information. Results show consistent and promising performance in all scenarios considered. The best result achieved is an improvement of approximately 10.5% in CASME II and an increment of nearly 10% in SMIC. We also report F-measure, precision and recall performance metrics that are most suited for the imbalanced nature of spontaneous micro-expression datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10

Similar content being viewed by others

Notes

  1. http://www.cse.oulu.fi/SMICDatabase

  2. http://fu.psych.ac.cn/CASME/casme2-en.php

References

  1. Asthana, A., Zafeiriou, S., Cheng, S., & Pantic, M. (2013). Robust discriminative response map fitting with constrained local models. In Computer vision and pattern recognition (pp. 3444–3451).

  2. Beauchemin, S.S., & Barron, J.L. (1995). The computation of optical flow. ACM Computing Surveys (CSUR), 27(3), 433–466.

    Article  Google Scholar 

  3. Cristinacce, D., & Cootes, T.F. (2006). Feature detection and tracking with constrained local models. In British machine vision conference, (Vol. 2 p. 6).

  4. Ekman, P. (2009). Lie catching and microexpressions. The Philosophy of Deception, pp. 118–133.

  5. Ekman, P. (2009). Telling lies: clues to Deceit in the marketplace, politics, and marriage. New York: W. W. Norton and Company.

    Google Scholar 

  6. Ekman, P., & Friesen, W.V. (1969). Nonverbal leakage and clues to deception. Journal for the Study of Interpersonal Processes, 32, 88–106.

    Article  Google Scholar 

  7. Ekman, P., & Friesen, W.V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17(2), 124.

    Article  Google Scholar 

  8. Ekman, P., & Friesen, W.V. (1974). Nonverbal behavior and psychopathology. The Psychology of Depression: Contemporary Theory and Research, pp. 203–232.

  9. Fan, X., & Verma, B. (2009). Selection and fusion of facial features for face recognition. Expert Systems with Applications, 36(3), 7157–7169.

    Article  Google Scholar 

  10. Fleet, D., & Weiss, Y. (2006). Optical flow estimation. In Handbook of mathematical models in computer vision (pp. 237–257).

  11. Friesen, E., & Ekman, P. (1978). Facial action coding system a technique for the measurement of facial movement. Consulting Psychologists Press, 12.

  12. Gibson, J.J. (1950). The perception of the visual world.

  13. Guo, Z., & Zhang, D. (2010). A completed modeling of local binary pattern operator for texture classification . IEEE Transactions on Image Processing, 19(6), 1657–1663.

    Article  MathSciNet  MATH  Google Scholar 

  14. Haggard, E.A., & Isaacs, K.S. (1966). Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In Methods of research in psychotherapy (pp. 154–165).

  15. Happy, S., & Routray, A. (2015). Automatic facial expression recognition using features of salient facial patches. IEEE Transactions on Affective Computing, 6(1), 1–12.

    Article  Google Scholar 

  16. Horn, B.K., & Schunck, B.G. (1981). Determining optical flow. In International society for optics and photonics (pp. 319–331).

  17. Le Ngo, A.C., Phan, R.C.W., & See, J. (2014). Spontaneous subtle expression recognition: Imbalanced databases & solutions. In Asian conference on computer vision.

  18. Li, X., Pfister, T., Huang, X., Zhao, G., & Pietikainen, M. (2013). A spontaneous micro-expression database inducement, collection and baseline. In Automatic face and gesture recognition (pp. 1–6).

  19. Liong, S.T., Phan, R.C.-W., See, J., Oh, Y.H., & Wong, K. (2014). Optical strain based recognition of subtle emotions. In International symposium on intelligent signal processing and communication systems.

  20. Liong, S.T., See, J., Phan, R.C.W., Le Ngo, A.C., Oh, Y.H., & Wong, K. (2014). Subtle expression recognition using optical strain weighted features. In Asian conference on computer vision workshops on computer vision for affective computing.

  21. Liong, S.T., See, J., Phan, R.C.W., Oh, Y.H., Ngo, A.C.L., Wong, K., & Tan, S.W. (2016). Spontaneous subtle expression detection and recognition based on facial strain. arXiv.

  22. Liong, S.T., See, J., Phan, R.C.W., & Wong, K. (2016), Less is more: Micro-expression recognition from video using apex frame. arXiv.

  23. Liu, M., Shan, S., Wang, R., & Chen, X. (2014). Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In Computer vision and pattern recognition (pp. 1749–1756).

  24. Meng, H., & Bianchi-Berthouze, N. (2014). Affective state level recognition in naturalistic facial and vocal expressions. IEEE Transactions on Cybernetics, 44(3), 315–328.

    Article  Google Scholar 

  25. Oh, Y.H., Le Ngo, A.C., See, J., Liong, S.T., Phan, R.C.W., & Ling, H.C. (2015). Monogenic riesz wavelet representation for micro-expression recognition. In Digital signal processing (pp. 1237–1241). Los Alamitos: IEEE.

  26. Ojala, T., Pietikainen, M., & Maenpaa, T. (2002). Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. Pattern Analysis and Machine Intelligence, 24(7), 971–987.

    Article  MATH  Google Scholar 

  27. Pfister, T., Li, X., Zhao, G., & Pietikainen, M. (2011). Recognising spontaneous facial micro-expressions. In International conference on computer vision (pp. 1449–1456).

  28. Polikovsky, S., Kameda, Y., & Ohta, Y. (2009). Facial micro-expressions recognition using high speed camera and 3d-gradient descriptor. In Crime detection and prevention (pp. 16–16).

  29. Porter, S., & ten Brinke, L. (2008). Reading between the lies identifying concealed and falsified emotions in universal facial expressions. Psychological Science, 19.5, 508– 514.

    Article  Google Scholar 

  30. Saragih, J.M., Lucey, S., & Cohn, J.F. (2011). Deformable model fitting by regularized landmark mean-shift. International Journal of Computer Vision, 91(2), 200–215.

    Article  MathSciNet  MATH  Google Scholar 

  31. Senechal, T., Bailly, K., & Prevost, L. (2014). Impact of action unit detection in automatic emotion recognition. Pattern Analysis and Applications, 17(1), 51–67.

    Article  MathSciNet  Google Scholar 

  32. Shan, C., Gong, S., & Mcowan, P.W. (2009). Facial expression recognition based on local binary patterns a comprehensive study. Image and Vision Computing, 27(6), 803–816.

    Article  Google Scholar 

  33. Shreve, M., Brizzi, J., Fefilatyev, S., Luguev, T., Goldgof, D., & Sarkar, S. (2014). Automatic expression spotting in videos. Image and Vision Computing, 32(8), 476–486.

    Article  Google Scholar 

  34. Shreve, M., Godavarthy, S., Manohar, V., Goldgof, D., & Sarkar, S. (2009). Towards macro-and micro-expression spotting in video using strain patterns. In Applications of computer vision (WACV) (pp. 1–6).

  35. Wang, L., Li, R.F., Wang, K., & Chen, J. (2014). Feature representation for facial expression recognition based on facs and lbp. Pattern Analysis and Applications, 11(5), 459–468.

    Google Scholar 

  36. Wang, S.-J., Chen, H.-L., Yan, W.-J., Chen, Y.-H., & Fu, X. (2014). Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine. Neural Processing Letters, 39(1), 25–43.

    Article  Google Scholar 

  37. Wang, S.J., Yan, W.J., Zhao, G., Fu, X., & Zhou, C.G. (2014). Micro-expression recognition using robust principal component analysis and local spatiotemporal directional features. In ECCV workshop on spontaneous facial behavior analysis.

  38. Wang, Y., See, J., Phan, R.C.W., & Oh, Y.H. (2015). Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition. In Computer vision–ACCV (pp. 525–537). Los Alamitos: IEEE.

  39. Yan, W.J., Wang, S.J., Liu, Y.J., Wu, Q., & Fu, X. (2014). For micro-expression recognition database and suggestions. Neurocomputing, 136, 82–87.

    Article  Google Scholar 

  40. Yan, W.-J., Wang, S.-J., Zhao, G., Li, X., Liu, Y.-J., Chen, Y.-H., & Fu, X. (2014). CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE, 9, e86041.

    Article  Google Scholar 

  41. Zhang, J., Shan, S., Kan, M., & Chen, X. (2014). Coarse-to-fine auto-encoder networks (cfan) for real-time face alignment. In Computer vision–ECCV (pp. 1–16).

  42. Zhao, G., & Pietikainen, M. (2009). Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(6), 915–928.

    Article  Google Scholar 

  43. Zhong, L., Liu, Q., Yang, P., Huang, J., & Metaxas, D.N. (2014). Learning multiscale active facial patches for expression analysis. IEEE Transactions on Cybernetics.

  44. Zhu, X., & Ramanan, D. (2012). Face detection, pose estimation, and landmark localization in the wild. In Computer vision and pattern recognition (pp. 2879–2886).

Download references

Author information

Authors and Affiliations

  1. Faculty of Computer Science and Information Technology, University of Malaya, 50603, Kuala Lumpur, Malaysia

    Sze-Teng Liong

  2. Faculty of Engineering, Multimedia University, 63100, Cyberjaya, Malaysia

    Sze-Teng Liong, Raphael C.-W. Phan & Su-Wei Tan

  3. Faculty of Information and Communication Technology, Xiamen University, 43900, Sepang, Selangor, Malaysia

    Sze-Teng Liong

  4. Faculty of Computing and Informatics, Multimedia University, 63100, Cyberjaya, Malaysia

    John See

  5. School of Information Technology, Monash University Malaysia, 47500, Bandar Sunway, Malaysia

    KokSheik Wong

Authors
  1. Sze-Teng Liong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John See
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Raphael C.-W. Phan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. KokSheik Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Su-Wei Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sze-Teng Liong.

Additional information

This research was funded in part by TM under the projects UbeAware and 2beAware.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liong, ST., See, J., Phan, RW. et al. Hybrid Facial Regions Extraction for Micro-expression Recognition System. J Sign Process Syst 90, 601–617 (2018). https://doi.org/10.1007/s11265-017-1276-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11265-017-1276-0

Keywords

Search

Navigation