Skip to main content
SpringerLink
Log in

Automatic micro-expression apex spotting using Cubic-LBP

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

The main way to communicate is through non-verbal expressions, although it could totally be manipulated by the person to give false expression. Unlike ordinary facial expressions, facial micro-expression has characterized by subtle movement and short duration of appearance which unleashes the true expression beyond the control of the person. Due to the nature of micro-expression which is very brief in time and low in intensity, prevalent methods could not come up with its challenges. One of the well-known dynamic texture descriptors is Local Binary Patterns on Three Orthogonal Planes (LBP-TOP) which mainly lacks in grabbing most vital information. To address this issue in this paper, we propose a novel feature extractor called Cubic-LBP that computes LBP on fifteen introduced planes. We demonstrate the effectiveness of these planes to find the apex frame where maximum facial movements within video sequences have occurred. Moreover, the whole process of spotting the apex frame in this paper is done automatically. Achieving results of apex frame spotting is satisfying on CASME and CASME II databases in comparison with most relevant state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Abbreviations

MEs:

Micro-Expressions

FER:

Facial Expression Recognition

AAM:

Active Appearance Models

CLM:

Constraint Local Model

DRMF:

Discriminative Response Maps Fitting

LBP:

Local Binary Pattern

HOG:

Histograms of Oriented Gradients

ROI:

Region of interest

DTCM:

Delaunay-based Temporal Coding Model

LBP-TOP:

Local Binary Pattern on Three Orthogonal Planes

LBP-SIP:

LBP with Six Intersection Points

STCLQP:

Spatio-Temporal Completed Local Quantization Patterns

RHOOF:

Region Histogram of Oriented Optical Flow

CASME:

Chinese Academy of Sciences Micro-Expressions

AU:

Action Unit

MAE:

Mean Absolute Error

SE:

Standard Error

OS:

Optical Strain

BS-ROIs:

the Binary Search strategy to the features found within ROIs

References

  1. Ahonen T, Hadid A, Pietikäinen M (2004) Face recognition with local binary patterns. In: European conference on computer vision. Springer, p 469–481

  2. Allaert B, Bilasco IM, Djeraba CJAPA (2018) Advanced local motion patterns for macro and micro facial expression recognition. Computer Vision and Pattern Recognition (cs.CV)

  3. Bhall RK, Sharma V (2015) A review: detection and analysis of facial micro-expression. Int J Educ Sci Res Rev (IJESRR) 2(1)

  4. Chaudhry R, Ravichandran A, Hager G, Vidal R (2009) Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, IEEE, pp 1932–1939

  5. Davison AK, Yap MH, Costen N, Tan K, Lansley C, Leightley D (2014) Micro-facial movements: an investigation on spatio-temporal descriptors. In: European conference on computer vision. Springer, p 111–123

  6. Davison AK, Lansley C, Costen N, Tan K, Yap MH (2018) Samm: a spontaneous micro-facial movement dataset. IEEE Trans Affect Comput 9(1):116–129

    Article  Google Scholar 

  7. Guo Y, Tian Y, Gao X, Zhang X (2014) Micro-expression recognition based on local binary patterns from three orthogonal planes and nearest neighbor method. In: Neural Networks (IJCNN), 2014 International joint conference on, IEEE, pp 3473–3479

  8. Haggard EA, Isaacs KS (1966) Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In: Methods of research in psychotherapy. Springer, p 154–165

  9. Hong X, Xu Y, Zhao G (2016) LBP-TOP: a tensor unfolding revisit. In: Asian conference on computer vision. Springer, p 513–527

  10. Huang X, Zhao G, Hong X, Zheng W, Pietikäinen MJN (2016) Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing 175:564–578

    Article  Google Scholar 

  11. Li X, Pfister T, Huang X, Zhao G, Pietikäinen M (2013) A spontaneous micro-expression database: Inducement, collection and baseline, in Automatic face and gesture recognition (fg), 2013 10th ieee international conference and workshops on, IEEE, p 1–6

  12. Li X et al (2018) Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans Affect Comput 9(4):563–577

    Article  Google Scholar 

  13. Liong S-T, See J, Wong K, Le Ngo AC, Oh Y-H, Phan RC-W (2015) Automatic apex frame spotting in micro-expression database. ACPR, p 665–669

  14. Liong S-T, See J, Wong K, Phan RC-W (2018) Less is more: micro-expression recognition from video using apex frame. Signal Process Image Commun 62:82–92

    Article  Google Scholar 

  15. Liu Y-J, Zhang J-K, Yan W-J, Wang S-J, Zhao G, Fu XJIT o AC (2016) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310

    Article  Google Scholar 

  16. Lu Z, Luo Z, Zheng H, Chen J, Li W (2014) A Delaunay-based temporal coding model for micro-expression recognition, in Asian conference on computer vision. Springer, p 698–711

  17. Ma H, An G, Wu S, Yang F (2017) A region histogram of oriented optical flow (RHOOF) feature for apex frame spotting in micro-expression. In: Intelligent Signal Processing and Communication Systems (ISPACS), 2017 International Symposium on, IEEE, p 281–286

  18. Merghani W, Davison AK, Yap MHJAPA (2018) A review on facial micro-expressions analysis: datasets, features and metrics. Computer Vision and Pattern Recognition (cs.CV)

  19. Moilanen A, Zhao G, Pietikäinen M (2014) Spotting rapid facial movements from videos using appearance-based feature difference analysis. In: Pattern Recognition (ICPR), 2014 22nd International Conference on, IEEE, p 1722–1727

  20. Nonis F, Dagnes N, Marcolin F, Vezzetti EJAS (2019) 3D approaches and challenges in facial expression recognition algorithms—A Literature review. Appl Sci 9(18):3904

    Article  Google Scholar 

  21. Oh Y-H, See J, Le Ngo AC, Phan RC-W, Baskaran VMJF i p (2018) A survey of automatic facial micro-expression analysis: databases, methods, and challenges. Comput Vis Pattern Recogn (cs.CV) 9:1128

    Google Scholar 

  22. Ojala T, Pietikäinen M, Harwood D (1996) A comparative study of texture measures with classification based on featured distributions. Pattern Recogn 29(1):51–59

    Article  Google Scholar 

  23. Pfister T, Li X, Zhao G, Pietikäinen M (2011) Recognising spontaneous facial micro-expressions. In: Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE, p 1449–1456

  24. Polikovsky S, Kameda Y, Ohta Y (2009) Facial micro-expressions recognition using high speed camera and 3D-gradient descriptor. In: Crime Detection and Prevention (ICDP 2009), 3rd international conference on crime detection and prevention, IET, pp 1–6

  25. Qu F, Wang S-J, Yan W-J, Li H, Wu S, Fu XJIT o AC (2018) CAS (ME) ^ 2: a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Trans Affect Comput 9(4):424–436

    Article  Google Scholar 

  26. Ruiz-Hernandez JA, Pietikäinen M (2013) Encoding local binary patterns using the re-parametrization of the second order gaussian jet. In: Automatic face and gesture recognition (FG), 2013 10th IEEE International Conference and Workshops on, IEEE, pp 1–6

  27. Shan C, Gong S, McOwan PWJI, V. Computing (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27(6):803–816

    Article  Google Scholar 

  28. Tornincasa S et al (2019) 3D facial action units and expression recognition using a crisp logic. Comput-Aided Des Appl 16:256–268

    Article  Google Scholar 

  29. Wang Y, See J, Phan RC-W, Oh Y-H (2014) Lbp with six intersection points: Reducing redundant information in lbp-top for micro-expression recognition, in Asian Conference on Computer Vision. Springer, pp 525–537

  30. Yan W-J, Wu Q, Liu Y-J, Wang S-J, Fu X (2013) CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: Automatic face and gesture recognition (fg), 2013 10th ieee international conference and workshops on, IEEE, pp 1–7

  31. Yan W-J, Wang S-J, Chen Y-H, Zhao G, Fu X (2014) Quantifying micro-expressions with constraint local model and local binary pattern. In: Workshop at the European conference on computer vision. Springer, pp 296–305

  32. Yan W-J et al (2014) CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PloS one 9(1):e86041

    Article  Google Scholar 

  33. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Electrical, Biomedical and Mechatronics Engineering, Qazvin Branch, Islamic Azad University, nokhbegan Blvd., Qazvin, Iran

    Vida Esmaeili & Seyed Omid Shahdi

Authors
  1. Vida Esmaeili
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Seyed Omid Shahdi
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors took part in the work described in this paper. Seyed Omid Shahdi proposed the framework of this work, and Vida Esmaeili carried out the whole experiments and drafted the manuscript. The second author Seyed Omid Shahdi helped in supervising the experiments. The author Vida Esmaeili wrote the first version of this paper, and then, the author Seyed Omid Shahdi revised the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Seyed Omid Shahdi.

Ethics declarations

Conflict of interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Esmaeili, V., Shahdi, S.O. Automatic micro-expression apex spotting using Cubic-LBP. Multimed Tools Appl 79, 20221–20239 (2020). https://doi.org/10.1007/s11042-020-08737-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-08737-5

Keywords

Search

Navigation