Skip to main content

Advertisement

Log in

Coupled source domain targetized with updating tag vectors for micro-expression recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Micro-expression has raised increasing attention for analyzing human inner emotions. However, most micro-expression recognition methods are developed with specific feature representations and extraction methods, such as local binary pattern on three orthogonal planes (LBP-TOP) and optical flow. The performance in such micro-expression recognition models is not high due to the limited training samples and the unequal size of the sample category. To improve the performance, we present a novel algorithm, named coupled source domain targetized with updating tag vectors, and we apply it to the micro-expression recognition. This method leverages rich speech data to enhance micro-expression recognition by transferring learning from the speech to the micro-expression recognition. The method highlights are: it simultaneously projects micro-expression samples and speech samples into a common space, then minimizes the reconstruction error between the speech and micro-expression samples, with an updating tag vectors added in the reconstruction process. It performs recognition by using dictionary learning together with support vector machine (SVM). Experimental results on the CASIA Chinese emotional corpus and CASME II micro-expression database demonstrate the effectiveness of our method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Aharon M, Elad M, Bruckstein A (2006) KSVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans Signal Process 54(11):4311–4322

    Article  MATH  Google Scholar 

  2. Ben X, Meng W, Yan R, Wang K (2013) Kernel coupled distance metric learning for gait recognition and face recognition. Neurocomputing 120(10):577–589

    Article  Google Scholar 

  3. Ben X, Zhang P, Meng W, Yan R, Yang M, Liu W et al (2016) On the distance metric learning between cross-domain gaits. Neurocomputing 208:153–164

    Article  Google Scholar 

  4. Ben X, Zhang P, Yan R, Yang M, Ge G (2016) Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation. Neural Comput & Applic 27(8):2629–2646

    Article  Google Scholar 

  5. Bryt O, Elad M (2008) Compression of facial images using the k-svd algorithm. J Visual Commun Image Represent 19(4):270–282

    Article  Google Scholar 

  6. CASIA Chinese emotional corpus, (2006), http://www.chineseldx.org

  7. Chang X, Yang Y (2016) Semisupervised feature analysis by mining correlations among multiple tasks. IEEE Trans Neural Netw Learn Syst. doi:10.1109/TNNLS.2016.2582746

    Article  MathSciNet  Google Scholar 

  8. Chang X, Yu YL, Yang Y (2016) Semantic pooling for complex event analysis in untrimmed videos. IEEE Trans Pattern Anal Mach Intell. doi:10.1109/TPAMI.2016.2608901

    Article  Google Scholar 

  9. Chang X, Ma Z, Yang Y (2017) Bi-level semantic representation analysis for multimedia event detection. IEEE Trans Cybern 47(5):1180–1197

    Article  Google Scholar 

  10. Du B, Wang Z, Zhang L, Zhang L, Liu W, Shen J, Tao D (2017) Exploring representativeness and informativeness for active learning. IEEE Trans Cybern 47(1):14–26

    Article  Google Scholar 

  11. Duan X, Dai Q, Wang X, Wang Y, Hua Z (2016) Recognizing spontaneous micro-expression from eye region. Neurocomputing 217:27–36

    Article  Google Scholar 

  12. Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Mathematics 32(2):407–451

    MathSciNet  MATH  Google Scholar 

  13. Ekman, P (2009) Telling lies: Clues to deceit in the marketplace, politics, and marriage (revised edition). WW Norton & Company, New York

  14. Endres J, Laidlaw A (2009) Micro-expression recognition training in medical students: a pilot study. BMC Med Educ 9(1):47

    Article  Google Scholar 

  15. Engan K, Aase SO, Hakon Husoy J (1999) Method of optimal directions for frame design. IEEE International Conference on Acoustics, Speech, and Signal Processing 5:2443–2446

    Google Scholar 

  16. Gong C, Tao D, Fu K, Yang J (2014) Fick's law assisted propagation for semisupervised learning. IEEE Trans Neural Netw Learn Syst 26(9):2148–2162

    Article  MathSciNet  Google Scholar 

  17. Gong C, Liu T, Tao D, Fu K, Tu E, Yang J (2015) Deformed graph Laplacian for semisupervised learning. IEEE Trans Neural Netw Learn Syst 26(10):2261–2274

    Article  MathSciNet  Google Scholar 

  18. Gong C, Tao D, Maybank SJ, Liu W, Kang G, Yang J (2016) Multi-modal curriculum learning for semi-supervised image classification. IEEE Trans Image Process 25(7):3249–3260

    Article  MathSciNet  MATH  Google Scholar 

  19. Gu B, Sheng VS (2017) A robust regularization path algorithm for v-support vector classification. IEEE Trans Neural Netw Learn Syst 28(5):1241–1248

    Article  Google Scholar 

  20. Gu B, Sun X, Sheng VS (2016) Structural minimax probability machine. IEEE Trans Neural Netw Learn Syst PP(99):1–11

    Google Scholar 

  21. Guo Y, Tian Y, Gao X, Zhang X (2014) Micro-expression recognition based on local binary patterns from three orthogonal planes and nearest neighbor method. The 2014 International Joint Conference on Neural Networks (IJCNN). IEEE, Beijing, p 3473–3479

  22. Haggard EA, Isaacs KS (1966) Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. Methods of Research in Psychotherapy. Springer US. 154–165

  23. Han Y, Yang Y, Zhou X (2013) Co-regularized ensemble for feature selection. IJCAI 13:1380–1386

  24. Han Y, Yang Y, Yan Y, Ma Z (2015) Semi-supervised feature selection via spline regression for video semantic recognition. IEEE Trans Neural Netw Learn Syst 26(2):252–264

    Article  MathSciNet  Google Scholar 

  25. He J, Hu JF, Lu X, Zheng WS (2016) Multi-task mid-level feature learning for micro-expression recognition. Pattern Recogn:44–52

    Article  Google Scholar 

  26. Jia X, Ben X, Yuan H, Kpalma K, Meng W Macro-to-micro transformation model for micro-expression recognition. J Comput Sci. doi:10.1016/j.jocs.2017.03.016

    Article  Google Scholar 

  27. Kan M, Wu J, Shan S, Chen X (2014) Domain adaptation for face recognition: targetize source domain bridged by common subspace. Int J Comput Vis 109(1):94–109

    Article  MATH  Google Scholar 

  28. Li H, Liu F (2009) Image denoising via sparse and redundant representations over learned dictionaries in wavelet domain. International Conference on Image and Graphics. IEEE, Xi'an, p 754–758

  29. Li Z, Yang Y, Liu J, Zhou X, Lu H (2012) Unsupervised feature selection using nonnegative spectral analysis. Twenty-Sixth AAAI Conference on Artificial Intelligence 2:1026–1032

    Google Scholar 

  30. Li Z, Liu J, Yang Y, Zhou X (2014) Clustering-guided sparse structural learning for unsupervised feature selection. IEEE Trans Knowl Data Eng 26(9):2138–2150

    Article  Google Scholar 

  31. Liao S, Jain AK, Li SZ (2013) Partial face recognition: alignment-free approach. IEEE Trans Pattern Anal Mach Intell 35(5):1193–1205

    Article  Google Scholar 

  32. Liu YJ, Zhang JK, Yan WJ, Wang SJ, Zhao G, Fu X (2016) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310

    Article  Google Scholar 

  33. Mccree AV, Barnwell TPI (1995) A mixed excitation LPC vocoder model for low bit rate speech coding. IEEE Transactions on Speech & Audio Processing 3(4):242–250

    Article  Google Scholar 

  34. Miao Y, Gowayyed M, Metze F (2015) EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding. In Automatic Speech Recognition and Understanding (ASRU), Scottsdale, 167–174

  35. Muda L, Begam M, Elamvazuthi I (2010) Voice recognition algorithms using MEL frequency cepstral coefficient (MFCC) and dynamic time warping (DTW) techniques. arXiv preprint arXiv:1003–4083

  36. Murty KSR, Yegnanarayana B (2006) Combining evidence from residual phase and MFCC features for speaker recognition. IEEE Signal Process Lett 13(1):52–55

    Article  Google Scholar 

  37. Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359

    Article  Google Scholar 

  38. Pati YC, Rezaiifar R, Krishnaprasad PS (1993) Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition, Signals, Systems and Computers, 1993. 1993 Conference Record of The Twenty-Seventh Asilomar Conference on. Conference Record of The Twenty-Seventh Asilomar Conference on. IEEE, Pacific Grove, 40–44

  39. Qu F, Wang SJ, Yan WJ, Li H, Wu S, Fu X (2017) CAS (ME)^ 2: a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Trans Affect Comput. doi:10.1109/TAFFC.2017.2654440

    Article  Google Scholar 

  40. Ren CX, Dai DQ (2010) Incremental learning of bidirectional principal components for face recognition. Pattern Recogn 43(1):318–330

    Article  MATH  Google Scholar 

  41. Sobin C, Alpert M (1999) Emotion in speech: the acoustic attributes of fear, anger, sadness, and joy. J Psycholinguist Res 28(4):347–365

    Article  Google Scholar 

  42. Song L, Smola A, Gretton A, Borgwardt KM, Bedo J (2007) Supervised feature selection via dependence estimation. In Proceedings of the 24th international conference on Machine learning, 823–830. Corvalis, Oregon, USA — June 20–24, 2007

  43. Vergyri D, Stolcke A, Gadde VRR, Ferrer L (2003) Prosodic knowledge sources for automatic speech recognition. IEEE International Conference on Acoustics, Speech, and Signal Processing 1:208–211

    Google Scholar 

  44. Wang SJ, Yan WJ, Li X, Zhao G, Fu X (2014) Micro-expression Recognition Using Dynamic Textures on Tensor Independent Color Space. International Conference on Pattern Recognition. IEEE, Stockholm, 4678–4683

  45. Wang SJ, Chen HL, Yan WJ, Chen YH, Fu X (2014) Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine. Neural Process Lett 39(1):25–43

    Article  Google Scholar 

  46. Wang Z, Yang W, Ben X (2015) Low-resolution degradation face recognition over long distance based on cca. Neural Comput & Applic 26(7):1645–1652

    Article  Google Scholar 

  47. Wang SJ, Yan WJ, Sun T, Zhao G, Fu X (2016) Sparse tensor canonical correlation analysis for micro-expression recognition. Neurocomputing 214:218–232

    Article  Google Scholar 

  48. Xia Z, Feng X, Peng J, Peng X, Zhao G (2016) Spontaneous micro-expression spotting via geometric deformation modeling. Comput Vis Image Underst 147:87–94

    Article  Google Scholar 

  49. Xu F, Zhang J, Wang J (2016) Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput. doi:10.1109/TAFFC.2016.2518162

    Article  Google Scholar 

  50. Yan WJ, Li X, Wang SJ, Zhao G, Liu YJ, Chen YH et al (2014) CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9(1):e86041

    Article  Google Scholar 

  51. Yan Y, Nie F, Li W, Gao C, Yang Y, Xu D (2016) Image classification by cross-media active learning with privileged information. IEEE Trans Multimedia 18(12):2494–2502 57

    Article  Google Scholar 

  52. Yang Y, Yang Y, Shen HT (2011) Effective transfer tagging from image to video. Int Conf Multimedea 9:1137–1140

    Google Scholar 

  53. Yang Y, Shen HT, Ma Z, Huang Z, Zhou X (2011) L2, 1-norm regularized discriminative feature selection for unsupervised learning. International Joint Conference on Artificial Intelligence 22(1):1589–1594

  54. Yang Y, Ma Z, Hauptmann AG et al (2013) Feature selection for multimedia analysis by sharing information among multiple tasks. IEEE Trans Multimedia 15(3):661–669

    Article  Google Scholar 

  55. Yang W, Wang Z, Sun C (2015) A collaborative representation based projections method for feature extraction. Pattern Recogn 48(1):20–27

    Article  Google Scholar 

  56. Yang Y, Ma Z, Nie F, Chang X, Hauptmann AG (2015) Multi-class active learning by uncertainty sampling with diversity maximization. Int J Comput Vis 113(2):113–127

    Article  MathSciNet  Google Scholar 

  57. Yeh YR, Huang CH, Wang YCF (2014) Heterogeneous domain adaptation and classification by exploiting the correlation subspace. IEEE Trans Image Process 23(5):2009–2018

    Article  MathSciNet  MATH  Google Scholar 

  58. Zhang P, Ben X, Yan R, Wu C, Guo C (2016) Micro-expression recognition system. Optik - International Journal for Light and Electron Optics 127:1395–1400

    Article  Google Scholar 

  59. Zhang S, Feng B, Chen Z, Huang X (2017) Micro-Expression Recognition by Aggregating Local Spatio-Temporal Patterns. In: Amsaleg L., Guðmundsson G., Gurrin C., Jónsson B., Satoh S. (eds) MultiMedia Modeling. MMM 2017. Lecture Notes in Computer Science, vol 10132. Springer, Cham

  60. Zhao G, Pietikäinen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

Download references

Acknowledgments

We sincerely thank the Institute of Psychology, Chinese Academy of Sciences for granting us permission to use the CASME database. This project is supported by the Natural Science Foundation of China (Grant No. 61571275, 61672333), the Young Scholars Program of Shandong University, and the National Key Research and Development Program of China (Grant No. 2017YFC0803400).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Xianye Ben.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, X., Ben, X., Liu, S. et al. Coupled source domain targetized with updating tag vectors for micro-expression recognition. Multimed Tools Appl 77, 3105–3124 (2018). https://doi.org/10.1007/s11042-017-4943-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-4943-z

Keywords

Navigation