Skip to main content
SpringerLink
Log in

Manifold feature integration for micro-expression recognition

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Recognition of micro-expressions depends on the key features provided in the form of the temporal information. It needs considerable effort, however, to manually design useful characteristics. Subtle or micro-facial expressions are much difficult than regular facial expressions rich in emotional expressions in a true environment to be identified. An easy solution is discussed in this paper to recognise facial micro-expressions that utilizes an algorithm mix for facial identification, feature extraction and classification. The technique proposed is a framework which incorporates handcrafted features and deep features. Local Binary Pattern-Three Orthogonal Planes (LBP-TOP) is the handcraft feature which combines spatial and time analysis to encapsulate regional facet movements. The deep feature model is a micro-expression fine-tuned model based on Convolutional Neural Network (CNN). Two classifiers, i.e. SVM and Softmax are trained with combined feature vectors produced by LBP-TOP and CNN functionalities. All seven widely-used micro-expression databases are evaluated in an experiment. Our research can be claimed as the first extensive experimental study on a big amount of the datasets to train and test the suggested model. The findings in the document show that the method proposed, although simple and straightforward, achieves a substantial increase in precision relative to other commonly recognized micro-expression techniques, which are trained and tested with just a few datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Aadit, M.N.A., Mahin, M.T., Juthi, S.N.: Spontaneous micro-expression recognition using optimal firefly algorithm coupled with iso-flann classification. In: 2017 IEEE Region 10 Humanitarian Technology Conference (R10-HTC), pp. 714–717. IEEE (2017)

  2. Ben, X., Zhang, P., Yan, R., Yang, M., Ge, G.: Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation. Neural Comput. Appl. 27(8), 2629–2646 (2016)

    Article  Google Scholar 

  3. Ben-Hur, A., Weston, J.: A user’s guide to support vector machines. In: Clifton, N.J. (ed.) Data Mining Techniques for the Life Sciences, vol. 609, pp. 223–239. Springer (2010). https://doi.org/10.1007/978-1-60327-241-4_13

  4. Burges, C.J.: A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Discov. 2(2), 121–167 (1998)

    Article  Google Scholar 

  5. Campos, V., Jou, B., Giro-i Nieto, X.: From pixels to sentiment: fine-tuning cnns for visual sentiment prediction. Image Vis. Comput. 65, 15–22 (2017)

    Article  Google Scholar 

  6. Cireşan, D.C., Meier, U., Masci, J., Gambardella, L.M., Schmidhuber, J.: High-performance neural networks for visual object classification. arXiv preprint arXiv:1102.0183 (2011)

  7. Davison, A., Merghani, W., Yap, M.: Objective classes for micro-facial expression recognition. J. Imaging 4(10), 119 (2018)

    Article  Google Scholar 

  8. Davison, A.K., Lansley, C., Costen, N., Tan, K., Yap, M.H.: Samm: a spontaneous micro-facial movement dataset. IEEE Trans. Affect. Comput. 9(1), 116–129 (2016)

    Article  Google Scholar 

  9. Dixit, M., Kwitt, R., Niethammer, M., Vasconcelos, N.: Aga: attribute-guided augmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7455–7463 (2017)

  10. Ekman, P.: The argument and evidence about universals in facial expressions. Handbook of Social Psychophysiology, pp. 143–164. Wiley, England (1989)

    Google Scholar 

  11. Gunn, S.R., et al.: Support vector machines for classification and regression. ISIS Tech. Rep. 14(1), 5–16 (1998)

    Google Scholar 

  12. Happy, S., Routray, A.: Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Trans. Affect. Comput. 10, 394–406 (2019)

    Article  Google Scholar 

  13. He, J., Hu, J.F., Lu, X., Zheng, W.S.: Multi-task mid-level feature learning for micro-expression recognition. Pattern Recognit. 66, 44–52 (2017)

    Article  Google Scholar 

  14. Hu, C., Jiang, D., Zou, H., Zuo, X., Shu, Y.: Multi-task micro-expression recognition combining deep and handcrafted features. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 946–951. IEEE (2018)

  15. Huang, G.B., Mattar, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. In: Workshop on Faces in ’Real-Life’ Images: Detection, Alignment, and Recognition (2008)

  16. Huang, X., Zhao, G., Hong, X., Zheng, W., Pietikäinen, M.: Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175, 564–578 (2016)

    Article  Google Scholar 

  17. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on Multimedia, pp. 675–678. ACM (2014)

  18. Jung, H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2983–2991 (2015)

  19. Karpathy, A., et al.: Cs231n convolutional neural networks for visual recognition. Neural Netw. 1, (2016)

  20. Kim, D.H., Baddar, W.J., Ro, Y.M.: Micro-expression recognition with expression-state constrained spatio-temporal feature representations. In: Proceedings of the 24th ACM International Conference on Multimedia, pp. 382–386. ACM (2016)

  21. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

  22. Le Ngo, A.C., Oh, Y.H., Phan, R.C.W., See, J.: Eulerian emotion magnification for subtle expression recognition. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1243–1247. IEEE (2016)

  23. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., et al.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  24. Li, X., Hong, X., Moilanen, A., Huang, X., Pfister, T., Zhao, G., Pietikäinen, M.: Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans. Affect. Comput. 9(4), 563–577 (2017)

    Article  Google Scholar 

  25. Li, X., Pfister, T., Huang, X., Zhao, G., Pietikäinen, M.: A spontaneous micro-expression database: inducement, collection and baseline. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–6. IEEE (2013)

  26. Liong, S.T., Gan, Y., Zheng, D., Xua, H.X., Zhang, H.Z., Lyu, R.K., Liu, K.H., et al.: Evaluation of the spatio-temporal features and gan for micro-expression recognition system. arXiv preprint arXiv:1904.01748 (2019)

  27. Liong, S.T., See, J., Wong, K., Phan, R.C.W.: Less is more: micro-expression recognition from video using apex frame. Signal Process. Image Commun. 62, 82–92 (2018)

    Article  Google Scholar 

  28. Liu, Y.J., Zhang, J.K., Yan, W.J., Wang, S.J., Zhao, G., Fu, X.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. 7(4), 299–310 (2015)

    Article  Google Scholar 

  29. Mishra, A.: Metrics to evaluate your machine learning algorithm. Towards Data Science (2018). https://towardsdatascience.com/metrics-toevaluate-your-machine-learning-algorithm-f10ba6e38234. Accessed 15 Jan 2019

  30. Muna, N., Rosiani, U.D., Yuniamo, E.M., Pumomo, M.H.: Subpixel subtle motion estimation of micro-expressions multiclass classification. In: 2017 IEEE 2nd International Conference on Signal and Image Processing (ICSIP), pp. 325–330. IEEE (2017)

  31. Ng, H.W., Nguyen, V.D., Vonikakis, V., Winkler, S.: Deep learning for emotion recognition on small datasets using transfer learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 443–449. ACM (2015)

  32. Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study of texture measures with classification based on featured distributions. Pattern Recognit. 29(1), 51–59 (1996)

    Article  Google Scholar 

  33. Ojala, T., Pietikäinen, M., Mäenpää, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 7, 971–987 (2002)

    Article  Google Scholar 

  34. Ouyang, Y., Sang, N.: A facial expression recognition method by fusing multiple sparse representation based classifiers. In: International Symposium on Neural Networks, pp. 479–488. Springer (2013)

  35. Parkhi, O.M., Vedaldi, A., Zisserman, A., et al.: Deep face recognition. bmvc 1, 6 (2015)

    Google Scholar 

  36. Patel, D., Hong, X., Zhao, G.: Selective deep features for micro-expression recognition. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2258–2263. IEEE (2016)

  37. Peng, M., Wang, C., Chen, T., Liu, G., Fu, X.: Dual temporal scale convolutional neural network for micro-expression recognition. Front. Psychol. 8, 1745 (2017)

    Article  Google Scholar 

  38. Pfister, T., Li, X., Zhao, G., Pietikäinen, M.: Recognising spontaneous facial micro-expressions. In: 2011 International Conference on Computer Vision, pp. 1449–1456. IEEE (2011)

  39. Pietikäinen, M., Hadid, A., Zhao, G., Ahonen, T.: Computer Vision Using Local Binary Patterns, vol. 40. Springer Science & Business Media, Berlin (2011)

    Book  Google Scholar 

  40. Polikovsky, S., Kameda, Y., Ohta, Y.: Facial micro-expressions recognition using high speed camera and 3d-gradient descriptor. IET (2009)

  41. Qu, F., Wang, S.J., Yan, W.J., Li, H., Wu, S., Fu, X.: Cas(me)\(^2\): a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Trans. Affect. Comput. 9(4), 424–436 (2017)

    Article  Google Scholar 

  42. Reddy, S.P.T., Karri, S.T., Dubey, S.R., Mukherjee, S.: Spontaneous facial micro-expression recognition using 3d spatiotemporal convolutional neural networks. arXiv preprint arXiv:1904.01390 (2019)

  43. Shreve, M., Godavarthy, S., Goldgof, D., Sarkar, S.: Macro-and micro-expression spotting in long videos using spatio-temporal strain. In: Face and Gesture 2011, pp. 51–56. IEEE (2011)

  44. Simard, P.Y., Steinkraus, D., Platt, J.C., et al.: Best practices for convolutional neural networks applied to visual document analysis. In: Icdar, vol. 3 (2003)

  45. Song, Y., Morency, L.P., Davis, R.: Learning a sparse codebook of facial and body microexpressions for emotion recognition. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, pp. 237–244. ACM (2013)

  46. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

  47. Takalkar, M., Xu, M., Wu, Q., Chaczko, Z.: A survey: facial micro-expression recognition. Multimed. Tools Appl. 77(15), 19301–19325 (2018)

    Article  Google Scholar 

  48. Takalkar, M.A., Xu, M.: Image based facial micro-expression recognition using deep learning on small datasets. In: 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp. 1–7. IEEE (2017)

  49. Takalkar, M.A., Zhang, H., Xu, M.: Improving micro-expression recognition accuracy using twofold feature extraction. In: International Conference on Multimedia Modeling, pp. 652–664. Springer (2019)

  50. Wang, S.J., Yan, W.J., Li, X., Zhao, G., Fu, X.: Micro-expression recognition using dynamic textures on tensor independent color space. In: 2014 22nd International Conference on Pattern Recognition, pp. 4678–4683. IEEE (2014)

  51. Wang, S.J., Yan, W.J., Li, X., Zhao, G., Zhou, C.G., Fu, X., Yang, M., Tao, J.: Micro-expression recognition using color spaces. IEEE Trans. Image Process. 24(12), 6034–6047 (2015)

    Article  MathSciNet  Google Scholar 

  52. Wang, Y., See, J., Oh, Y.H., Phan, R.C.W., Rahulamathavan, Y., Ling, H.C., Tan, S.W., Li, X.: Effective recognition of facial micro-expressions with video motion magnification. Multimed. Tools Appl. 76(20), 21665–21690 (2017)

    Article  Google Scholar 

  53. Wang, Y., See, J., Phan, R.C.W., Oh, Y.H.: Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition. In: Asian Conference on Computer Vision, pp. 525–537. Springer (2014)

  54. Warren, G., Schertler, E., Bull, P.: Detecting deception from emotional and unemotional cues. J. Nonverbal Behav. 33(1), 59–69 (2009)

    Article  Google Scholar 

  55. Weinberger, S.: Airport security: intent to deceive? Nat. News 465(7297), 412–415 (2010)

    Article  Google Scholar 

  56. Widen, S.C., Russell, J.A., Brooks, A.: Anger and disgust: discrete or overlapping categories. In: 2004 APS Annual Convention, Boston College, Chicago, IL (2004)

  57. Wolf, L., Hassner, T., Maoz, I.: Face recognition in unconstrained videos with matched background similarity. In: IEEE (2011)

  58. Wu, Q., Shen, X., Fu, X.: The machine knows what you are hiding: an automatic micro-expression recognition system. In: International Conference on Affective Computing and Intelligent Interaction, pp. 152–162. Springer (2011)

  59. Yan, W.J., Li, X., Wang, S.J., Zhao, G., Liu, Y.J., Chen, Y.H., Fu, X.: Casme ii: an improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9(1), e86041 (2014)

    Article  Google Scholar 

  60. Yan, W.J., Wu, Q., Liang, J., Chen, Y.H., Fu, X.: How fast are the leaked facial expressions: the duration of micro-expressions. J. Nonverbal Behav. 37(4), 217–230 (2013)

    Article  Google Scholar 

  61. Zheng, H.: Micro-expression recognition based on 2d gabor filter and sparse representation. In: Journal of Physics: Conference Series, vol. 787, p. 012013. IOP Publishing (2017)

Download references

Author information

Authors and Affiliations

  1. Faculty of Engineering and Information Technology, School of Electrical and Data Engineering, University of Technology Sydney, Ultimo, Australia

    Madhumita A. Takalkar, Min Xu & Zenon Chaczko

Authors
  1. Madhumita A. Takalkar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Min Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zenon Chaczko
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Madhumita A. Takalkar.

Additional information

Communicated by I. IDE.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The preliminary version of this work is presented in MultiMedia Modelling (MMM 2019) [49].

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Takalkar, M.A., Xu, M. & Chaczko, Z. Manifold feature integration for micro-expression recognition. Multimedia Systems 26, 535–551 (2020). https://doi.org/10.1007/s00530-020-00663-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-020-00663-8

Keywords

Search

Navigation