Skip to main content
SpringerLink
Log in

Going deeper in hidden sadness recognition using spontaneous micro expressions database

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Recognition of facial micro-expressions (MEs), which indicates conscious or unconscious suppressing of true emotions, is still a challenging task in the affective computing and computer vision. There are two main reasons for that: First, the lack of spontaneous MEs databases, preferably focused on one emotion. So far, posed facial MEs databases were developed, and in the most cases, machines were trained on this posed MEs, which are stronger and more visible than spontaneous ones. Second, in order to achieve high recognition rate, deep learning structures are required that can achieve the best performance with very large number of data. To address these challenges, we make the following contributions: (i) extension of our MEs spontaneous database by adding new subjects; (ii) We analysed spontaneous MEs in long videos only for hidden sadness; (iii) We presented deeper analysis for automatic hidden sadness detection algorithm with deep learning architecture and compared results with standard machine learning techniques for hidden sadness detection. It is shown that with our method 99.08% recognition performance has been achieved observing only the eye region of the face.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. https://github.com/TadasBaltrusaitis/OpenFace

  2. http://www.cs.waikato.ac.nz/ml/weka

  3. https://www.tensorflow.org/

  4. https://www.youtube.com/watch?v=6dpO9cypJcA

References

  1. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Proces Syst 16(3):1097–1105

    Google Scholar 

  2. Anbarjafari G (2013) Face recognition using color local binary pattern from mutually independent color channels. EURASIP J Adv Signal Process 2013(1):6

    Article  Google Scholar 

  3. Baltrušaitis T, Robinson P, Morency L-P, An open source facial behavior analysis toolkit (2016). In: IEEE Winter conference on applications of computer vision

  4. Birchfield S (1997) Derivation of kanade-lucas-tomasi tracking equation, unpublished notes

  5. Bolotnikova A, Demirel H, Anbarjafari G (2017) Real-time ensemble based face recognition system for nao humanoids using local binary pattern. Analog Integr Circ Sig Process 92(3):467–475

    Article  Google Scholar 

  6. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    Article  MATH  Google Scholar 

  7. Chang C-C, Lin C-J (2011) Libsvm: a library for support vector machines. ACM Trans Intell Syst Technol (TIST) 2(3):27

    Google Scholar 

  8. Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 23(6):681–685

    Article  Google Scholar 

  9. Corneanu C, Noroozi F, Kaminska D, Sapinski T, Escalera S, Anbarjafari G (2018) Survey on emotional body gesture recognition. In: IEEE transactions on affective computing

  10. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297

    MATH  Google Scholar 

  11. Davison AK, Lansley C, Costen N, Tan K, Yap M H (2017) Samm: a spontaneous micro-facial movement dataset. In: IEEE transactions on affective computing

  12. Demirel H, Anbarjafari G (2009) Data fusion boosted face recognition based on probability distribution functions in different colour channels. EURASIP J Adv Signal Process 2009:25

    Article  MATH  Google Scholar 

  13. Eisenbarth GW, Hedwig, Alpers (2011) Happy mouth and sad eyes: scanning emotional facial expressions. Emotion 11 [Online]. Available: http://gen.lib.rus.ec/scimag/index.php?s=10.1037/a0022758

  14. Ekman P Mett and sett online tools. [Online]. Available: http://www.ekmaninternational.com/paul-ekman-international-plc-home/training-courses/online-training.aspx

  15. Ekman P, Friesen WV (1974) Nonverbal behavior and psychopathology. In: The psychology of depression: contemporary theory and research, p 3

  16. Ekman P, Friesen WV (2003) Unmasking the face: a guide to recognizing emotions from facial clues. Ishk

  17. Fischer AH, Manstead AS, Zaalberg R (2003) Social influences on the emotion process. Eur Rev Soc Psychol 14(1):171–201

    Article  Google Scholar 

  18. Fung G, Mangasarian OL (2000) Data selection for support vector machine classifiers. In: Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, pp 64–70

  19. Gorbova J, Avots E, Lüsi I, Fishel M, Escalera S, Anbarjafari G (2018) Integrating vision and language for first-impression personality analysis. IEEE MultiMed 25(2):24–33

    Article  Google Scholar 

  20. Gorbova J, Marjanović M, Njeguš A, Čolović M, Hasan D, Gholammreza A (2017) Automatic hidden sadness detection using micro-expressions. In: Proceedings from the 12th IEEE international conference on automatic face and gesture recognition (FG 2017). IEEE

  21. Guo J, Lei Z, Wan J, Avots E, Hajarolasvadi N, Knyazev B, Kuharenko A, Junior J C S J, Baró X, Demirel H et al (2018) Dominant and complementary emotion recognition from still images of faces. IEEE Access 6:26391–26403

    Article  Google Scholar 

  22. Guo Z, Zhang L, Zhang D (2010) A completed modeling of local binary pattern operator for texture classification. IEEE Trans Image Process 19(6):1657–1663

    Article  MathSciNet  MATH  Google Scholar 

  23. Haamer RE, Kulkarni K, Imanpour N, Haque MA, Avots E, Breisch M, Nasrollahi K, Escalera S, Ozcinar C, Baro X et al (2018) Changes in facial expression as biometric: a database and benchmarks of identification. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). IEEE, pp 621–628

  24. Haggard EA, Isaacs KS (1966) Micro-momentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of research in psychotherapy, pp 154–165

  25. Heisele TPB, Ho P (2001) Face recognition with support vector machines: global versus component-based approach. In: IEEE international conference on computer vision

  26. Howard A, Zhang C, Horvitz E (2017) Addressing bias in machine learning algorithms: a pilot study on emotion recognition for intelligent systems. In: 2017 IEEE workshop on advanced robotics and its social impacts (ARSO). IEEE, pp 1–7

  27. Hsu S-C, Huang H-H, Huang C-L (2017) Facial expression recognition for human-robot interaction. In: IEEE international conference on robotic computing (IRC). IEEE, pp 1–7

  28. Jia J, Xu Y, Zhang S, Xue X (2016) The facial expression recognition method of random forest based on improved pca extracting feature. In: 2016 IEEE international conference on signal processing, communications and computing (ICSPCC). IEEE, pp 1–5

  29. Junior J, Jacques C, Ozcinar C, Marjanovic M, Baró X, Anbarjafari G, Escalera S (2019) On the effect of age perception biases for real age regression. arXiv preprint arXiv:1902.07653

  30. Kulkarni K, Corneanu C, Ofodile I, Escalera S, Baró X, Hyniewska S, Allik J, Anbarjafari G (2018) Automatic recognition of facial displays of unfelt emotions. In: IEEE transactions on affective computing

  31. Lévesque J, Eugène F, Joanette Y, Paquette V, Mensour B, Beaudoin G, Leroux J-M, Bourgouin P, Beauregard M (2003) Neural circuitry underlying voluntary suppression of sadness. Biol Psychiatry 53(6):502–510

    Article  Google Scholar 

  32. Li X, Pfister T, Huang X, Zhao G, Pietikäinen M (2013) A spontaneous micro-expression database: Inducement, collection and baseline. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG),. IEEE, pp 1–6

  33. Li X, Yu J, Zhan S (2016) Spontaneous facial micro-expression detection based on deep learning. In: 2016 IEEE 13th international conference on signal processing (ICSP). IEEE, pp 1130–1134

  34. Li X, Xiaopeng H, Moilanen A, Huang X, Pfister T, Zhao G, Pietikainen M (2017) Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. In: IEEE transactions on affective computing

  35. Liong S-T, See J, Wong K, Phan R C-W (2018) Less is more: micro-expression recognition from video using apex frame. Signal Process Image Commun 62:82–92

    Article  Google Scholar 

  36. Liu Y-J, Zhang J-K, Yan W-J, Wang S-J, Zhao G, Fu X (2016) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310

    Article  Google Scholar 

  37. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 94–101

  38. Mansoorizadeh M, Charkari NM (2010) Multimodal information fusion application to human emotion recognition from face and speech. Multimed Tools Appl 49(2):277–297

    Article  Google Scholar 

  39. Martinez B, Binefa X, Pantic M (2010) Facial component detection in thermal imagery. In: 2010 IEEE Computer Society conference on computer vision and pattern recognition workshops (CVPRW). IEEE, pp 48–54

  40. Mstafa RJ, Elleithy KM (2016) A video steganography algorithm based on kanade-lucas-tomasi tracking algorithm and error correcting codes. Multimed Tools Appl 75(17):10311–10333

    Article  Google Scholar 

  41. Parkhi OM, Vedaldi A, Zisserman A et al (2015) Deep face recognition. BMVC 1(3):6

    Google Scholar 

  42. Patel D, Hong X, Zhao G (2016) Selective deep features for micro-expression recognition. In: 2016 23rd international conference on pattern recognition (ICPR). IEEE, pp 2258–2263

  43. Pavlović M, Zlatanović L (2011) Facijalni znaci eksperimentalnoizazvane tuge kod ispitanika muškog i ženskog pola. Godisnjak za psihologiju 8(10):11–11

    Google Scholar 

  44. Pavlović M, Zlatanović L (2011) Facijalni znaci maskiranja, blokiranja i suzbijanja eksperimentalno izazvane emocije tuge. Godišnjak za psihologiju 8 (10):152–168

    Google Scholar 

  45. Pfister T, Li X, Zhao G, Pietikäinen M (2011) Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework. In: IEEE international conference on computer vision workshops (ICCV Workshops)

  46. Pfister T, Li X, Zhao G, Pietikäinen M (2011) Recognising spontaneous facial micro-expressions. In: 2011 IEEE international conference on computer vision (ICCV). IEEE, pp 1449–1456

  47. Qi Z, Tian Y, Shi Y (2013) Robust twin support vector machine for pattern classification. Pattern Recogn 46(1):305–316

    Article  MATH  Google Scholar 

  48. Rahman A, Anam AI, Yeasin M (2017) Emoassist: emotion enabled assistive tool to enhance dyadic conversation for the blind. Multimed Tools Appl 76(6):7699–7730

    Article  Google Scholar 

  49. Schmid PC, Schmid-Mast M (2010) Mood effects on emotion recognition. Motiv Emot 34:288–292

    Article  Google Scholar 

  50. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  51. Smith ML, Cottrell GW, Gosselin F, Schyns PG (2005) Transmitting and decoding facial expressions. Psychol Sci 16(3):184–189

    Article  Google Scholar 

  52. Tertychnyi P, Ozcinar C, Anbarjafari G (2018) Low-quality fingerprint classification using deep neural network. IET Biom 7(6):550–556

    Article  Google Scholar 

  53. Wang S-J, Yan W-J, Li X, Zhao G, Fu X (2014) Micro-expression recognition using dynamic textures on tensor independent color space. In: 2014 22nd international conference on pattern recognition (ICPR). IEEE, pp 4678–4683

  54. Wang S-J, Yan W-J, Sun T, Zhao G, Fu X (2016) Sparse tensor canonical correlation analysis for micro-expression recognition. Neurocomputing 214:218–232

    Article  Google Scholar 

  55. Wang Y, See J, Oh Y-H, Phan R C-W, Rahulamathavan Y, Ling H-C, Tan S-W, Li X (2017) Effective recognition of facial micro-expressions with video motion magnification. Multimed Tools Appl 76(20):21665–21690

    Article  Google Scholar 

  56. Warren G, Schertler E, Bull P (2009) Detecting deception from emotional and unemotional cues. J Nonverbal Behav 33(1):59–69

    Article  Google Scholar 

  57. Wu Q, Shen X, Fu X (2011) The machine knows what you are hiding: an automatic micro-expression recognition system. In: International conference on affective computing and intelligent interaction. Springer, pp 152–162

  58. Yan W-J, Wu Q, Liang J, Chen Y-H, Fu X (2013) How fast are the leaked facial expressions: the duration of micro-expressions. J Nonverbal Behav 37(4):217–230

    Article  Google Scholar 

  59. Yan W-J, Wu Q, Liu Y-J, Wang S-J, Fu X (2013) Casme database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). IEEE, pp 1–7

  60. Yan W-J, Li X, Wang S-J, Zhao G, Liu Y-J, Chen Y-H, Fu X (2014) Casme ii: an improved spontaneous micro-expression database and the baseline evaluation. PloS One 9(1):e86041

    Article  Google Scholar 

  61. Yang B, Cao J-M, Jiang D-P, Lv J-D (2017) Facial expression recognition based on dual-feature fusion and improved random forest classifier. Multimed Tools Appl 77:20477–20499

Download references

Acknowledgments

This work is supported Estonian Research Council Grant (PUT638), the Scientific and Technological Research Council of Turkey (TÜBITAK) (Project 1001 - 116E097), and the Estonian Centre of Excellence in IT (EXCITE) funded by the European Regional Development Fund. The authors also gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan X and Titan V Pascal GPUs. The author would like to thank Miss Dariia Temirova for helping with deep neural network codes.

Author information

Authors and Affiliations

  1. iCV Lab, Institute of Technology, University of Tartu, Tartu, Estonia

    Jelena Gorbova & Gholamreza Anbarjafari

  2. Singidunum University, Belgrade, Serbia

    Milica Colovic, Marina Marjanovic & Angelina Njegus

  3. Institute of Digital Technologies, Loughborough University London, London, UK

    Gholamreza Anbarjafari

  4. Faculty of Engineering, Hasan Kalyoncu University, Gaziantep, Turkey

    Gholamreza Anbarjafari

Authors
  1. Jelena Gorbova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Milica Colovic
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marina Marjanovic
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Angelina Njegus
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gholamreza Anbarjafari
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gholamreza Anbarjafari.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gorbova, J., Colovic, M., Marjanovic, M. et al. Going deeper in hidden sadness recognition using spontaneous micro expressions database. Multimed Tools Appl 78, 23161–23178 (2019). https://doi.org/10.1007/s11042-019-7658-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7658-5

Keywords

Search

Navigation