Abstract
Micro-expression has raised increasing attention for analyzing human inner emotions. However, most micro-expression recognition methods are developed with specific feature representations and extraction methods, such as local binary pattern on three orthogonal planes (LBP-TOP) and optical flow. The performance in such micro-expression recognition models is not high due to the limited training samples and the unequal size of the sample category. To improve the performance, we present a novel algorithm, named coupled source domain targetized with updating tag vectors, and we apply it to the micro-expression recognition. This method leverages rich speech data to enhance micro-expression recognition by transferring learning from the speech to the micro-expression recognition. The method highlights are: it simultaneously projects micro-expression samples and speech samples into a common space, then minimizes the reconstruction error between the speech and micro-expression samples, with an updating tag vectors added in the reconstruction process. It performs recognition by using dictionary learning together with support vector machine (SVM). Experimental results on the CASIA Chinese emotional corpus and CASME II micro-expression database demonstrate the effectiveness of our method.
Similar content being viewed by others
References
Aharon M, Elad M, Bruckstein A (2006) KSVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans Signal Process 54(11):4311–4322
Ben X, Meng W, Yan R, Wang K (2013) Kernel coupled distance metric learning for gait recognition and face recognition. Neurocomputing 120(10):577–589
Ben X, Zhang P, Meng W, Yan R, Yang M, Liu W et al (2016) On the distance metric learning between cross-domain gaits. Neurocomputing 208:153–164
Ben X, Zhang P, Yan R, Yang M, Ge G (2016) Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation. Neural Comput & Applic 27(8):2629–2646
Bryt O, Elad M (2008) Compression of facial images using the k-svd algorithm. J Visual Commun Image Represent 19(4):270–282
CASIA Chinese emotional corpus, (2006), http://www.chineseldx.org
Chang X, Yang Y (2016) Semisupervised feature analysis by mining correlations among multiple tasks. IEEE Trans Neural Netw Learn Syst. doi:10.1109/TNNLS.2016.2582746
Chang X, Yu YL, Yang Y (2016) Semantic pooling for complex event analysis in untrimmed videos. IEEE Trans Pattern Anal Mach Intell. doi:10.1109/TPAMI.2016.2608901
Chang X, Ma Z, Yang Y (2017) Bi-level semantic representation analysis for multimedia event detection. IEEE Trans Cybern 47(5):1180–1197
Du B, Wang Z, Zhang L, Zhang L, Liu W, Shen J, Tao D (2017) Exploring representativeness and informativeness for active learning. IEEE Trans Cybern 47(1):14–26
Duan X, Dai Q, Wang X, Wang Y, Hua Z (2016) Recognizing spontaneous micro-expression from eye region. Neurocomputing 217:27–36
Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Mathematics 32(2):407–451
Ekman, P (2009) Telling lies: Clues to deceit in the marketplace, politics, and marriage (revised edition). WW Norton & Company, New York
Endres J, Laidlaw A (2009) Micro-expression recognition training in medical students: a pilot study. BMC Med Educ 9(1):47
Engan K, Aase SO, Hakon Husoy J (1999) Method of optimal directions for frame design. IEEE International Conference on Acoustics, Speech, and Signal Processing 5:2443–2446
Gong C, Tao D, Fu K, Yang J (2014) Fick's law assisted propagation for semisupervised learning. IEEE Trans Neural Netw Learn Syst 26(9):2148–2162
Gong C, Liu T, Tao D, Fu K, Tu E, Yang J (2015) Deformed graph Laplacian for semisupervised learning. IEEE Trans Neural Netw Learn Syst 26(10):2261–2274
Gong C, Tao D, Maybank SJ, Liu W, Kang G, Yang J (2016) Multi-modal curriculum learning for semi-supervised image classification. IEEE Trans Image Process 25(7):3249–3260
Gu B, Sheng VS (2017) A robust regularization path algorithm for v-support vector classification. IEEE Trans Neural Netw Learn Syst 28(5):1241–1248
Gu B, Sun X, Sheng VS (2016) Structural minimax probability machine. IEEE Trans Neural Netw Learn Syst PP(99):1–11
Guo Y, Tian Y, Gao X, Zhang X (2014) Micro-expression recognition based on local binary patterns from three orthogonal planes and nearest neighbor method. The 2014 International Joint Conference on Neural Networks (IJCNN). IEEE, Beijing, p 3473–3479
Haggard EA, Isaacs KS (1966) Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. Methods of Research in Psychotherapy. Springer US. 154–165
Han Y, Yang Y, Zhou X (2013) Co-regularized ensemble for feature selection. IJCAI 13:1380–1386
Han Y, Yang Y, Yan Y, Ma Z (2015) Semi-supervised feature selection via spline regression for video semantic recognition. IEEE Trans Neural Netw Learn Syst 26(2):252–264
He J, Hu JF, Lu X, Zheng WS (2016) Multi-task mid-level feature learning for micro-expression recognition. Pattern Recogn:44–52
Jia X, Ben X, Yuan H, Kpalma K, Meng W Macro-to-micro transformation model for micro-expression recognition. J Comput Sci. doi:10.1016/j.jocs.2017.03.016
Kan M, Wu J, Shan S, Chen X (2014) Domain adaptation for face recognition: targetize source domain bridged by common subspace. Int J Comput Vis 109(1):94–109
Li H, Liu F (2009) Image denoising via sparse and redundant representations over learned dictionaries in wavelet domain. International Conference on Image and Graphics. IEEE, Xi'an, p 754–758
Li Z, Yang Y, Liu J, Zhou X, Lu H (2012) Unsupervised feature selection using nonnegative spectral analysis. Twenty-Sixth AAAI Conference on Artificial Intelligence 2:1026–1032
Li Z, Liu J, Yang Y, Zhou X (2014) Clustering-guided sparse structural learning for unsupervised feature selection. IEEE Trans Knowl Data Eng 26(9):2138–2150
Liao S, Jain AK, Li SZ (2013) Partial face recognition: alignment-free approach. IEEE Trans Pattern Anal Mach Intell 35(5):1193–1205
Liu YJ, Zhang JK, Yan WJ, Wang SJ, Zhao G, Fu X (2016) A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans Affect Comput 7(4):299–310
Mccree AV, Barnwell TPI (1995) A mixed excitation LPC vocoder model for low bit rate speech coding. IEEE Transactions on Speech & Audio Processing 3(4):242–250
Miao Y, Gowayyed M, Metze F (2015) EESEN: End-to-end speech recognition using deep RNN models and WFST-based decoding. In Automatic Speech Recognition and Understanding (ASRU), Scottsdale, 167–174
Muda L, Begam M, Elamvazuthi I (2010) Voice recognition algorithms using MEL frequency cepstral coefficient (MFCC) and dynamic time warping (DTW) techniques. arXiv preprint arXiv:1003–4083
Murty KSR, Yegnanarayana B (2006) Combining evidence from residual phase and MFCC features for speaker recognition. IEEE Signal Process Lett 13(1):52–55
Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359
Pati YC, Rezaiifar R, Krishnaprasad PS (1993) Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition, Signals, Systems and Computers, 1993. 1993 Conference Record of The Twenty-Seventh Asilomar Conference on. Conference Record of The Twenty-Seventh Asilomar Conference on. IEEE, Pacific Grove, 40–44
Qu F, Wang SJ, Yan WJ, Li H, Wu S, Fu X (2017) CAS (ME)^ 2: a database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Trans Affect Comput. doi:10.1109/TAFFC.2017.2654440
Ren CX, Dai DQ (2010) Incremental learning of bidirectional principal components for face recognition. Pattern Recogn 43(1):318–330
Sobin C, Alpert M (1999) Emotion in speech: the acoustic attributes of fear, anger, sadness, and joy. J Psycholinguist Res 28(4):347–365
Song L, Smola A, Gretton A, Borgwardt KM, Bedo J (2007) Supervised feature selection via dependence estimation. In Proceedings of the 24th international conference on Machine learning, 823–830. Corvalis, Oregon, USA — June 20–24, 2007
Vergyri D, Stolcke A, Gadde VRR, Ferrer L (2003) Prosodic knowledge sources for automatic speech recognition. IEEE International Conference on Acoustics, Speech, and Signal Processing 1:208–211
Wang SJ, Yan WJ, Li X, Zhao G, Fu X (2014) Micro-expression Recognition Using Dynamic Textures on Tensor Independent Color Space. International Conference on Pattern Recognition. IEEE, Stockholm, 4678–4683
Wang SJ, Chen HL, Yan WJ, Chen YH, Fu X (2014) Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine. Neural Process Lett 39(1):25–43
Wang Z, Yang W, Ben X (2015) Low-resolution degradation face recognition over long distance based on cca. Neural Comput & Applic 26(7):1645–1652
Wang SJ, Yan WJ, Sun T, Zhao G, Fu X (2016) Sparse tensor canonical correlation analysis for micro-expression recognition. Neurocomputing 214:218–232
Xia Z, Feng X, Peng J, Peng X, Zhao G (2016) Spontaneous micro-expression spotting via geometric deformation modeling. Comput Vis Image Underst 147:87–94
Xu F, Zhang J, Wang J (2016) Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affect Comput. doi:10.1109/TAFFC.2016.2518162
Yan WJ, Li X, Wang SJ, Zhao G, Liu YJ, Chen YH et al (2014) CASME II: an improved spontaneous micro-expression database and the baseline evaluation. PLoS One 9(1):e86041
Yan Y, Nie F, Li W, Gao C, Yang Y, Xu D (2016) Image classification by cross-media active learning with privileged information. IEEE Trans Multimedia 18(12):2494–2502 57
Yang Y, Yang Y, Shen HT (2011) Effective transfer tagging from image to video. Int Conf Multimedea 9:1137–1140
Yang Y, Shen HT, Ma Z, Huang Z, Zhou X (2011) L2, 1-norm regularized discriminative feature selection for unsupervised learning. International Joint Conference on Artificial Intelligence 22(1):1589–1594
Yang Y, Ma Z, Hauptmann AG et al (2013) Feature selection for multimedia analysis by sharing information among multiple tasks. IEEE Trans Multimedia 15(3):661–669
Yang W, Wang Z, Sun C (2015) A collaborative representation based projections method for feature extraction. Pattern Recogn 48(1):20–27
Yang Y, Ma Z, Nie F, Chang X, Hauptmann AG (2015) Multi-class active learning by uncertainty sampling with diversity maximization. Int J Comput Vis 113(2):113–127
Yeh YR, Huang CH, Wang YCF (2014) Heterogeneous domain adaptation and classification by exploiting the correlation subspace. IEEE Trans Image Process 23(5):2009–2018
Zhang P, Ben X, Yan R, Wu C, Guo C (2016) Micro-expression recognition system. Optik - International Journal for Light and Electron Optics 127:1395–1400
Zhang S, Feng B, Chen Z, Huang X (2017) Micro-Expression Recognition by Aggregating Local Spatio-Temporal Patterns. In: Amsaleg L., Guðmundsson G., Gurrin C., Jónsson B., Satoh S. (eds) MultiMedia Modeling. MMM 2017. Lecture Notes in Computer Science, vol 10132. Springer, Cham
Zhao G, Pietikäinen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928
Acknowledgments
We sincerely thank the Institute of Psychology, Chinese Academy of Sciences for granting us permission to use the CASME database. This project is supported by the Natural Science Foundation of China (Grant No. 61571275, 61672333), the Young Scholars Program of Shandong University, and the National Key Research and Development Program of China (Grant No. 2017YFC0803400).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zhu, X., Ben, X., Liu, S. et al. Coupled source domain targetized with updating tag vectors for micro-expression recognition. Multimed Tools Appl 77, 3105–3124 (2018). https://doi.org/10.1007/s11042-017-4943-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-017-4943-z