Skip to main content
SpringerLink
Log in

Recognizing hidden emotions from difference image using mean local mapped pattern

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Recent progress in computer vision has pushed the limit of facial recognition from human identification to micro-expressions (MEs). However, the visual analysis of MEs is still a very challenging task because of the short occurrence and insignificant intensity of the underlying signals. To date, the accuracy of recognizing hidden emotions from frames using conventional methods is still far from reaching saturation. To address this, we have proposed a new ME recognition approach based on Mean Local Mapped Pattern (M-LMP) as a texture feature, which outperforms other state-of-the art features in terms of accuracy due to its capability of capturing small pixel transitions. Inspired by previous work, we applied M-LMP to the difference image computed from an onset frame and an apex frame, where the former represents the frame with neutral emotion and the latter consists of the frame with the largest ME intensity. The extracted local features were classified using support vector machine (SVM) and K nearest neighbourhood (KNN) classifiers. The validation of the proposed approach was performed on the CASME II and CAS(ME)2 datasets, and the results were compared with other similar state-of-the-art approaches. Comprehensive experiments were conducted using various parameters to show the robustness of our approach in the imbalanced and small dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Ekman P, Friesen WV (1969) Nonverbal leakage and clues to deception. Psychiatry 32(1):88–106

    Article  Google Scholar 

  2. Polikovsky S, Kameda Y, Ohta Y (2009) Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In: 3rd International Conference on Imaging for Crime Detection and Prevention (ICDP 2009), pp 1–6

  3. Ekman P. (2009) Lie catching and microexpressions. The philosophy of deception:118–133

  4. Liong S-T, See J, Wong K, RC-W P (2018) Less is more: Micro-expression recognition from video using apex frame. Signal Process Image Commun 62:82–92

    Article  Google Scholar 

  5. Ekman P (2003) Darwin, deception, and facial expression. Ann N Y Acad Sci 1000(1):205–221

    Article  Google Scholar 

  6. Ekman P, Friesen WV (1978) Facial Action Coding System. vol v. 1. Consulting Psychologists Press

  7. Li X, X H MA, Huang X, Pfister T, Zhao G, Pietikainen M (2017) Towards Reading Hidden Emotions: A Comparative Study of Spontaneous Micro-expression Spotting and Recognition Methods. IEEE Trans Affect Comput PP(99):1–1

    Google Scholar 

  8. Li X, Yu J, Zhan S (2016) Spontaneous facial micro-expression detection based on deep learning. In: IEEE 13th International Conference on Signal Processing (ICSP), pp 1130–1134

  9. X-b S, Wu Q, X-l F (2012) Effects of the duration of expressions on the recognition of microexpressions. J Zhejiang Univ Sci B 13(3):221–230. https://doi.org/10.1631/jzus.B1100063

    Article  Google Scholar 

  10. Frank MG, Herbasz M, Sinuk K, Keller A, Kurylo A, Nolan C (2009) I see how you feel: training laypeople and professionals to recognize fleeting emotions. In: Annual meeting of the International Communication Association, Sheraton New York, Unpublished manuscript

  11. Liu Y, Nie L, Liu L, Rosenblum DS (2016) From action to activity: Sensor-based activity recognition. Neurocomputing 181:108–115. https://doi.org/10.1016/j.neucom.2015.08.096

    Article  Google Scholar 

  12. Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2015) Action2Activity: recognizing complex activities from sensor data. In: Twenty-fourth International Joint Conference on Artificial Intelligence, pp 1617-1623

  13. Liu Y, Zhang L, Nie L, Yan Y, Rosenblum DS (2016) Fortune teller: predicting your career path. In: ​Thirtieth AAAI Conference on Artificial Intelligence, pp 201-207.

  14. Yan W-J, Li X, Wang S-J, Zhao G, Liu Y-J, Chen Y-H, Fu X (2014) CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9(1):e86041

    Article  Google Scholar 

  15. Li X, Pfister T, Huang X, Zhao G, Pietikäinen M (2013) A spontaneous micro-expression database: inducement, collection and baseline. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp 1–6

  16. Qu F, Wang SJ, Yan WJ, Li H, Wu S, Fu X (2017) CAS(ME)^2: A Database for Spontaneous Macro-expression and Micro-expression Spotting and Recognition. IEEE Trans Affect Comput PP(99):1–1. https://doi.org/10.1109/TAFFC.2017.2654440

    Google Scholar 

  17. Davison AK, Lansley C, Costen N, Tan K, Yap MH (2016) SAMM: A Spontaneous Micro-Facial Movement Dataset. IEEE Trans Affect Comput PP(99):1–1

    Google Scholar 

  18. Wang Y, See J, Oh Y-H, Phan RC-W, Rahulamathavan Y, Ling H-C, Tan S-W, Li X (2017) Effective recognition of facial micro-expressions with video motion magnification. Multimed Tools Appl 76(20):21665–21690

    Article  Google Scholar 

  19. Ferraz CT, Gonzaga A (2017) Object classification using a local texture descriptor and a support vector machine. Multimed Tools Appl 76(20):20609–20641

    Article  Google Scholar 

  20. Huang X, Wang SJ, Zhao G, Piteikäinen M (2015) Facial Micro-Expression Recognition Using Spatiotemporal Local Binary Pattern with Integral Projection. In: IEEE International Conference on Computer Vision Workshop (ICCVW), pp 1–9

  21. Lyons MJ, Akamatsu S, Kamachi M, Gyoba J, Budynek J (1998) The Japanese female facial expression (JAFFE) database. In: Proceedings of third international conference on automatic face and gesture recognition, pp 14–16

  22. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Automatic Face and Gesture Recognition, 2000. Proceedings. Fourth IEEE International Conference on, IEEE, pp 46–53

  23. Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression analysis. In: Multimedia and Expo, 2005. ICME 2005. IEEE International Conference on, IEEE, p 5 pp.

  24. Shreve M, Godavarthy S, Manohar V, Goldgof D, Sarkar S (2009) Towards macro- and micro-expression spotting in video using strain patterns. In: Workshop on Applications of Computer Vision (WACV), pp 1–6

  25. Wen-Jing Y, Wu Q, Yong-Jin L, Su-Jing W, Fu X (2013) CASME database: A dataset of spontaneous micro-expressions collected from neutralized faces. In: 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp 1–7

  26. Le Ngo AC, Phan RC-W, See J (2015) Spontaneous Subtle Expression Recognition: Imbalanced Databases and Solutions. In, Cham. Computer Vision -- ACCV 2014. Springer International Publishing, pp 33–48

  27. Wang Y, See J, Phan RCW, Oh Y-H (2015) Efficient Spatio-Temporal Local Binary Patterns for Spontaneous Facial Micro-Expression Recognition. PLoS One 10(5):e0124674

    Article  Google Scholar 

  28. Guo Y, Xue C, Wang Y, Yu M (2015) Micro-expression recognition based on CBP-TOP feature with ELM. Optik - International Journal for Light and Electron Optics 126(23):4446–4451

    Article  Google Scholar 

  29. Huang XH, Wang SJ, Liu X, Zhao G, Feng X, Pietikainen M (2017) Discriminative Spatiotemporal Local Binary Pattern with Revisited Integral Projection for Spontaneous Facial Micro-Expression Recognition. IEEE Trans Affect Comput PP(99):1–1

    Google Scholar 

  30. Wang SJ, Yan WJ, Li X, Zhao G, Fu X (2014) Micro-expression Recognition Using dynamic textures on tensor independent color space. In: 22nd International Conference on Pattern Recognition, pp 4678–4683

  31. Wang SJ, Yan WJ, Li X, Zhao G, Zhou CG, Fu X, Yang M, Tao J (2015) Micro-Expression Recognition Using Color Spaces. IEEE Trans Image Process 24(12):6034–6047

    Article  MathSciNet  MATH  Google Scholar 

  32. Liong S-T, Phan RC-W, See J, Oh Y-H, Wong K (2014) Optical strain based recognition of subtle emotions. In: International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), IEEE, pp 180–184

  33. Liong S-T, See J, Phan RCW, Oh Y-H, Cat Le Ngo A, Wong K, Tan S-W (2016) Spontaneous subtle expression detection and recognition based on facial strain. Signal Process Image Commun 47:170–182

    Article  Google Scholar 

  34. Liong S-T, See J, Phan RC-W, Le Ngo AC, Oh Y-H, Wong K (2015) Subtle expression recognition using optical strain weighted features. In, Cham, Computer Vision - ACCV 2014 Workshops. Springer International Publishing, pp 644–657

  35. Happy SL, Routray A (2017) Fuzzy Histogram of Optical Flow Orientations for Micro-expression Recognition. IEEE Trans Affect Comput PP(99):1–1

    Article  Google Scholar 

  36. Xu F, Zhang J, Wang JZ (2017) Microexpression Identification and Categorization Using a Facial Dynamics Map. IEEE Trans Affect Comput 8(2):254–267

    Article  Google Scholar 

  37. Liu YJ, Zhang JK, Yan WJ, Wang SJ, Zhao G, Fu X (2016) A Main Directional Mean Optical Flow Feature for Spontaneous Micro-Expression Recognition. IEEE Trans Affect Comput 7(4):299–310

    Article  Google Scholar 

  38. Wang S-J, Yan W-J, Zhao G, Fu X, Zhou C-G (2015) Micro-Expression Recognition Using Robust Principal Component Analysis and Local Spatiotemporal Directional Features. In: Agapito L, Bronstein MM, Rother C (eds) Computer Vision - ECCV 2014 Workshops: Zurich, Switzerland, September 6–7 and 12, 2014, Proceedings. Part I. Springer International Publishing, Cham, pp 325–338

    Chapter  Google Scholar 

  39. Huang X, Zhao G, Hong X, Zheng W, Pietikäinen M (2016) Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns. Neurocomputing 175:564–578

    Article  Google Scholar 

  40. Wang S-J, Chen H-L, Yan W-J, Chen Y-H, Fu X (2014) Face Recognition and Micro-expression Recognition Based on Discriminant Tensor Subspace Analysis Plus Extreme Learning Machine. Neural Process Lett 39(1):25–43

    Article  Google Scholar 

  41. He J, Hu J-F, Lu X, Zheng W-S (2017) Multi-task mid-level feature learning for micro-expression recognition. Pattern Recogn 66:44–52

    Article  Google Scholar 

  42. Zheng H, Geng X, Yang Z (2016) A Relaxed K-SVD Algorithm for Spontaneous Micro-Expression Recognition. In, Cham, PRICAI 2016: Trends in Artificial Intelligence. Springer International Publishing, pp 692–699

  43. Zheng H, Zhu J, Yang Z, Jin Z (2017) Effective micro-expression recognition using relaxed K-SVD algorithm. Int J Mach Learn Cybern 8(6):2043–2049

    Article  Google Scholar 

  44. Patel D, Hong X, Zhao G (2016) Selective deep features for micro-expression recognition. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp 2258–2263

  45. Takalkar MA, Xu M (2017) Image based facial micro-expression recognition using deep learning on small datasets. In: International Conference on Digital Image Computing: Techniques and Applications (DICTA), pp 1–7

  46. Mayya V, Pai RM, Pai MMM (2016) Combining temporal interpolation and DCNN for faster recognition of micro-expressions in video sequences. In: International Conference on Advances in Computing, Communications and Informatics (ICACCI), pp 699–703

  47. Peng M, Wang C, Chen T, Liu G, Fu X (2017) Dual Temporal Scale Convolutional Neural Network for Micro-Expression Recognition. Front In Psychol 8:1745

    Article  Google Scholar 

  48. Ferraz CT, Junior OP, Gonzaga A (2014) Feature description based on Mean Local Mapped Pattern. In: Workshop de Visão Computacional-WVC, pp 32–37

  49. Goh KM, Ng CH, Lim LL, Sheikh U. (2018) Micro-expression recognition: an updated review of current trends, challenges and solutions. Vis Computer:1–24

  50. Davison AK, Yap MH, Lansley C (2015) Micro-facial movement detection using individualised baselines and histogram-based descriptors. In: IEEE International Conference on Systems, Man, and Cybernetics, pp 1864–1869

  51. Davison AK, Lansley C, Ng C-C, Tan K, Yap MH (2016) Objective Micro-Facial Movement Detection Using FACS-Based Regions and Baseline Evaluation. CoRR abs/1612.05038

  52. Duque C, Alata O, Emonet R, Legrand A-C, Konik H (2018) Micro-expression spotting using the Riesz Pyramid. In: WACV 2018

  53. Tran T-K, Hong X, Zhao G (2017) Sliding window based micro-expression spotting: a benchmark. In: Cham. Advanced Concepts for Intelligent Vision Systems. Springer International Publishing, pp 542–553

  54. Xia Z, Feng X, Peng J, Peng X, Zhao G (2016) Spontaneous micro-expression spotting via geometric deformation modeling. Comput Vis Image Und 147:87–94

    Article  Google Scholar 

  55. Liong S-T, See J, Wong K, Phan RC-W (2017) Automatic Micro-expression Recognition from Long Video Using a Single Spotted Apex. In, Cham, Computer Vision – ACCV 2016 Workshops. Springer International Publishing, pp 345–360

  56. Liong S, See J, Wong K, Ngo ACL, Oh Y, Phan R (2015) Automatic apex frame spotting in micro-expression database. In: 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), pp 665–669. https://doi.org/10.1109/ACPR.2015.7486586

  57. Zafeiriou S, Petrou M (2010) Sparse representations for facial expressions recognition via l 1 optimization. In: Computer Vision and Pattern Recognition Workshops (CVPRW), 2010 IEEE Computer Society Conference on, IEEE, pp 32–39

  58. Ferraz CT, Pereira Jr O, Gonzaga A (2014) Feature description based on center-symmetric local mapped patterns. In: Proceedings of the 29th Annual ACM Symposium on Applied Computing, ACM, pp 39–44

  59. Bai G, Zhu Y, Ding Z (2008) A hierarchical face recognition method based on local binary pattern. In: 2008 Congress on Image and Signal Processing, pp 610–614. https://doi.org/10.1109/CISP.2008.520

  60. Farnebäck G (2003) Two-frame motion estimation based on polynomial expansion. In: Scandinavian conference on Image analysis, Springer, pp 363–370

  61. Chaudhry R, Ravichandran A, Hager G, Vidal R (2009) Histograms of oriented optical flow and Binet-Cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 1932–1939

Download references

Acknowledgements

This work was supported by the Ministry of Higher Education (MOHE), Malaysia, under the Fundamental Research Grant Scheme (FRGS) (Ref: FRGS/1/2016/TK04/TARUC/02/1).

Author information

Authors and Affiliations

  1. Faculty of Engineering and Technology, Tunku Abdul Rahman University College, 53300, Kuala Lumpur, Malaysia

    Kam Meng Goh

  2. School of Electrical Engineering, Faculty of Engineering, Universiti Teknologi Malaysia, 81310, Johor Bahru, Malaysia

    Usman Ullah Sheikh

  3. School of Computer Science, University of Nottingham Malaysia Campus, 43500, Semenyih, Selangor Darul Ehsan, Malaysia

    Tomás H. Maul

Authors
  1. Kam Meng Goh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Usman Ullah Sheikh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tomás H. Maul
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kam Meng Goh.

Ethics declarations

Conflict of interest

The author declares that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Goh, K.M., Sheikh, U.U. & Maul, T.H. Recognizing hidden emotions from difference image using mean local mapped pattern. Multimed Tools Appl 78, 21485–21520 (2019). https://doi.org/10.1007/s11042-019-7385-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7385-y

Keywords

Search

Navigation