Skip to main content

Advertisement

SpringerLink
Log in

Invisible emotion magnification algorithm (IEMA) for real-time micro-expression recognition with graph-based features

  • 1197: Advances in Soft Computing Techniques for Visual Information-based Systems
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The interest in real-time micro-expression recognition has increased with the current trend in human-computer interaction applications. Presently, there are several approaches which include geometrical analysis for fast facial muscle movement feature representations. However, due to the subtleness of emotions, recognition accuracy remains a challenge in this field. In order to address this challenge, this paper put forward a novel geometrical-based invisible emotion magnification algorithm (IEMA). This algorithm utilizes facial landmarks of on-set and apex frames to compute magnication elements as well as the direction of facial movement, and then applies them to the x and y coordinates of the apex-frame to enhance the magnitude of facial muscle movement for invisible emotion. Furthermore, this algorithm integrates a parameter which is set to adjust the magnification level in order to achieve optimal performance. The proposed IEMA was analyzed using a landmark-based facial graph, and subsequently, the euclidean distance and gradient of the graph segments were presented as features. The experimental results suggest that the proposed IEMA yields the highest accuracy of 94.78% and an F1-score of 0.90 with the CASMEII dataset, while the SMIC, CAS(ME)\(^2\) and SAMM datasets yield accuracies of 87.98%, 88.89% and 94.72% with F1-score of 0.86, 0.83 and 0.88, respectively. In fact, the proposed IEMA outperformed several micro-expression magnification methods in terms of accuracy as well as the F1-score. In addition, the proposed IEMA running time is 3.41ms per sample, which translates to 293 fps. This suggests that the proposed IEMA processes samples above the minimum requirement of 200fps for micro-expression recognition in real-time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Bergstra J, Bengio Y (2012) Random search for hyper-parameter optimization. J Mach Learn Res 13:2

    MathSciNet  MATH  Google Scholar 

  2. Bowyer A (1981) Computing dirichlet tessellations. Comput J 24(2):162–166

    Article  MathSciNet  Google Scholar 

  3. Buhari AM, Ooi C-P, Baskaran VM, Phan RC, Wong K, Tan W-H (2020) Facs-based graph features for real-time micro-expression recognition. J Imaging 6(12):130

    Article  Google Scholar 

  4. Chaudhry R, Ravichandran A, Hager G, Vidal R (2009) Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1932–1939

  5. Datta S, Sen D, Balasubramanian R (2017) Integrating geometric and textural features for facial emotion classification using svm frameworks. In Proceedings of International Conference on Computer Vision and Image Processing. Springer, pp. 619–628

  6. Davison A, Merghani W, Lansley C, Ng C-C, Yap MH (2018) Objective micro-facial movement detection using facs-based regions and baseline evaluation. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 642–649

  7. Davison AK, Lansley C, Costen N, Tan K, Yap MH (2018) Samm: A spontaneous micro-facial movement dataset. IEEE Trans Affect Comput 9(1):116–129

    Article  Google Scholar 

  8. Davison AK, Yap MH, Lansley C (2015) Micro-facial movement detection using individualised baselines and histogram-based descriptors. In 2015 IEEE International Conference on Systems, Man, and Cybernetics, pp. 1864–1869

  9. Ekman P (2012) Emotions revealed: Understanding faces and feelings. Hachette UK

  10. Ekman R (1997) What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA

    Google Scholar 

  11. Esmaeili V, Shahdi SO (2020) Automatic micro-expression apex spotting using cubic-lbp. Multimedia Tools and Applications, pp. 1–19

  12. Feng Z-H, Kittler J, Awais M, Huber P, Wu X-J (2017) Face detection, bounding box aggregation and pose estimation for robust facial landmark localisation in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 160–169

  13. Gan Y, Liong S-T, Yau W-C, Huang Y-C, Tan L-K (2019) Off-apexnet on micro-expression recognition system. Signal Process Image Commun 74:129–139

    Article  Google Scholar 

  14. Golzadeh H, Faria DR, Manso LJ, Ekárt A, Buckingham CD (2018) Emotion recognition using spatiotemporal features from facial expression landmarks. In 2018 International Conference on Intelligent Systems (IS), pp. 789–794

  15. Hong X, Xu Y, Zhao G (2016) Lbp-top: a tensor unfolding revisit. In Asian Conference on Computer Vision, Springer, pp. 513–527

  16. Jeong M, Ko BC (2018) Driver’s facial expression recognition in real-time for safe driving. Sensors 18(12):4270

    Article  Google Scholar 

  17. Kazemi V, Sullivan J (2014) One millisecond face alignment with an ensemble of regression trees. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1867–1874

  18. King DE (2009) Dlib-ml: A machine learning toolkit. J Mach Learn Res 10:1755–1758

    Google Scholar 

  19. Kumar AJR, Theagarajan R, Peraza O, Bhanu B (2019) Classification of facial micro-expressions using motion magnified emotion avatar images. In CVPR Workshops, pp. 12–20

  20. Lai Z, Chen R, Jia J, Qian Y (2020) Real-time micro-expression recognition based on resnet and atrous convolutions. J Ambient Intell Humaniz Comput, pp. 1–12

  21. Le Ngo AC, Oh Y-H, Phan RC-W, See J (2016) Eulerian emotion magnification for subtle expression recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1243–1247

  22. Lee Z-C, Phan RC-W, Tan S-W, Lee K-H (2017) Multimodal decomposition for enhanced subtle emotion recognition. In 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) IEEE, pp. 665–671

  23. Li X, Hong X, Moilanen A, Huang X, Pfister T, Zhao G, Pietikäinen M (2018) Towards reading hidden emotions: A comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans Affect Comput 9(4):563–577

    Article  Google Scholar 

  24. Li X, Pfister T, Huang X, Zhao G, Pietikäinen M (2013) A spontaneous micro-expression database: Inducement, collection and baseline. In 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–6

  25. Liong S-T, See J, Wong K, Phan RC-W (2016) Less is more: Micro-expression recognition from video using apex frame. Signal Process Image Commun 62:82–92

    Article  Google Scholar 

  26. Liong S-T, Wong K (2017) Micro-expression recognition using apex frame with phase information. In 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 534–537

  27. Oh Y-H, See J, Le Ngo AC, Phan RC-W, Baskaran VM (2018) A survey of automatic facial micro-expression analysis: Databases, methods and challenges. Front Psychol 9:1128

    Article  Google Scholar 

  28. O’Sullivan M, Frank MG, Hurley CM, Tiwana J (2009) Police lie detection accuracy: The effect of lie scenario. Law Hum Behav 33(6):530

    Article  Google Scholar 

  29. Park S, Kim D (2009) Subtle facial expression recognition using motion magnification. Pattern Recogn Lett 30(7):708–716

    Article  Google Scholar 

  30. Pawar SS, Moh M, Moh T-S (2019) Micro-expression recognition using motion magnification and spatiotemporal texture map. In International Conference on Ubiquitous Information Management and Communication, Springer, pp. 351–369

  31. Qu F, Wang S-J, Yan W-J, Li H, Wu S, Fu X (2018) Cas(me)\(^2\): A database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Trans Affect Comput 9(4):424–436

  32. Soh XR, Baskaran VM, Buhari AM, Phan RC-W (2017) A real time micro-expression detection system with lbp-top on a many-core processor. In 2017 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pp. 309–315

  33. Talukder BB, Chowdhury B, Howlader T, Rahman SM (2016) Intelligent recognition of spontaneous expression using motion magnification of spatio-temporal data. In Pacific-Asia Workshop on Intelligence and Security Informatics, Springer, pp. 114–128

  34. Wadhwa N, Rubinstein M, Durand F, Freeman WT (2013) Phase-based video motion processing. ACM Trans Graph 32(4):1–10

    Article  Google Scholar 

  35. Wang Y, See J, Oh Y-H, Phan RC-W, Rahulamathavan Y, Ling H-C, Tan S-W, Li X (2017) Effective recognition of facial micro-expressions with video motion magnification. Multimed Tools Appl 76(20):21665–21690

    Article  Google Scholar 

  36. Wu H-Y, Rubinstein M, Shih E, Guttag J, Durand F, Freeman W (2012) Eulerian video magnification for revealing subtle changes in the world. ACM Trans Graph 31(4):1–8

    Article  Google Scholar 

  37. Xia Z, Hong X, Gao X, Feng X, Zhao G (2019) Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions. IEEE Trans Multimedia 22(3):626–640

    Article  Google Scholar 

  38. Xu F, Zhang J, Wang JZ (2017) Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput 8(2):254–267

    Article  Google Scholar 

  39. Yan W-J, Li X, Wang S-J, Zhao G, Liu Y-J, Chen Y-H, Fu X (2014) Casme ii: An improved spontaneous micro-expression database and the baseline evaluation. PloS One 9(1):e86041

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported in part by Telekom Malaysia Research & Development (TM R&D) Fund-Ambience, Emotionally-Aware Human-Expressive Virtual Assistant for Affective Ambience (MMUE/180021).

Author information

Authors and Affiliations

  1. Faculty of Engineering, Multimedia University, Persiaran Multimedia, 63100, Cyberjaya, Selangor, Malaysia

    Adamu Muhammad Buhari, Chee-Pun Ooi & Wooi-Haw Tan

  2. School of Information Technology, Monash University Malaysia, 47500, Bandar Sunway, Selangor, Malaysia

    Vishnu Monn Baskaran, Raphael CW Phan & KokSheik Wong

Authors
  1. Adamu Muhammad Buhari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chee-Pun Ooi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vishnu Monn Baskaran
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Raphael CW Phan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. KokSheik Wong
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Wooi-Haw Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adamu Muhammad Buhari.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Buhari, A.M., Ooi, CP., Baskaran, V.M. et al. Invisible emotion magnification algorithm (IEMA) for real-time micro-expression recognition with graph-based features. Multimed Tools Appl 81, 9151–9176 (2022). https://doi.org/10.1007/s11042-021-11625-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11625-1

Keywords

Search

Navigation