Skip to main content
SpringerLink
Log in

Unsupervised emotion recognition algorithm based on improved deep belief model in combination with probabilistic linear discriminant analysis

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

Since the initial weight matrix between the last hidden layer of the network and the classification layer is usually generated randomly, the weight matrix does not have the discrimination ability to accurately classify the facial expression recognition, which results in that the features obtained by the weight matrix mapping cannot be guaranteed to be suitable for classification tasks. To solve this problem, a novel linear discriminant deep belief network is proposed in this paper. Firstly, the traditional linear discriminant analysis method is improved, and a new type of inter-class dispersion matrix is designed to solve the rank limitation problem in the traditional Linear Discriminant Analysis Method (LDA). Then, the weight matrix between the last hidden layer and the classification layer of the deep belief network is initialized by the improved linear discriminant analysis method, so that the network is more suitable for the classification task. In the experiments, our proposed deep network obtains respectively the recognition rates of 78.26% and 94.48% on the JAFFE database and the Extended Cohn-Kanade database. In addition, using our proposed algorithm for aggregating linear discriminant analysis into a deep belief network, we were able to produce an accuracy of 81.03% on the challenge test set.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Kotsia I, Pitas I (2007) Facial expression recognition in image sequences using geometric deformation features and support vector machines[J]. IEEE Trans Image Process 16(1):172–187

    Article  MathSciNet  Google Scholar 

  2. Calder AJ (2003) Facial expression recognition across the life span[J]. Neuropsychologia 41(2):195–202

    Article  Google Scholar 

  3. Calder AJ, Young AW (2005) Understanding the recognition of facial identity and facial expression[J]. Nat Rev Neurosci 6(8):641–651

    Article  Google Scholar 

  4. Sprengelmeyer R, Young AW et al (2003) Facial expression recognition in people with medicated and unmedicated Parkinson's disease[J]. Neuropsychologia 41(8):1047–1057

    Article  Google Scholar 

  5. Gu W, Xiang C, Venkatesh YV et al (2012) Facial expression recognition using radial encoding of local Gabor features and classifier synthesis[J]. Pattern Recogn 45(1):80–91

    Article  Google Scholar 

  6. Kim BK, Roh J, Dong SY et al (2016) Hierarchical committee of deep convolutional neural networks for robust facial expression recognition[J]. Journal on Multimodal User Interfaces 10(2):173–189

    Article  Google Scholar 

  7. Wood A, Rychlowska M, Korb S et al (2016) Fashioning the face: sensorimotor simulation contributes to facial expression recognition[J]. Trends Cogn Sci 20(3):227–240

    Article  Google Scholar 

  8. Lopes AT, Aguiar ED, Souza AFD et al (2016) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order[J]. Pattern Recogn 61:610–628

    Article  Google Scholar 

  9. Zhao K, Chu WS, Fernando DLT, et al. (2016) Joint patch and multi-label learning for facial action unit and holistic expression recognition[J]. IEEE Trans Image Process, 1–1

  10. Zhang T, Zheng W, Cui Z et al (2016) A deep neural network-driven feature learning method for multi-view facial expression recognition[J]. IEEE T Multimedia 18(12):2528–2536

    Article  Google Scholar 

  11. Uçar A, Demir Y, Güzeliş C (2016) A new facial expression recognition based on curvelet transform and online sequential extreme learning machine initialized with spherical clustering[J]. Neural Comput & Applic 27(1):131–142

    Article  Google Scholar 

  12. Zhen Q, Huang D, Wang Y et al (2016) Muscular movement model-based automatic 3D/4D facial expression recognition[J]. IEEE T Multimedia 18(7):1438–1450

    Article  Google Scholar 

  13. Sun B, Li L, Zhou G et al (2016) Facial expression recognition in the wild based on multimodal texture features[J]. J Electron Imaging 25(6):061407

    Article  Google Scholar 

  14. Yanpeng L, Yibin L, Xin M et al (2017) Facial expression recognition with fusion features extracted from salient facial areas[J]. Sensors 17(4):712

    Article  Google Scholar 

  15. Wang Z, Ruan Q, An G (2016) Facial expression recognition using sparse local fisher discriminant analysis[J]. Neurocomputing 174(174):756–766

    Article  Google Scholar 

  16. Guo Y, Zhao G, Pietikainen M (2016) Dynamic facial expression recognition with atlas construction and sparse representation.[J]. IEEE Trans Image Process 25(5):1977–1992

    Article  MathSciNet  MATH  Google Scholar 

  17. Zheng H, Geng X, Tao D, et al. (2015) A multi-task model for simultaneous face identification and facial expression recognition[J]. Neurocomputing, S0925231215010097

  18. Kamarol SKA, Jaward MH, Parkkinen J et al (2016) Spatiotemporal feature extraction for facial expression recognition.[J]. IET Image Process 10(7):534–541

    Article  Google Scholar 

  19. Zen G, Porzi L, Sangineto E, et al. (2016) Learning personalized models for facial expression analysis and gesture recognition[J]. IEEE T Multimedia, 1–1

  20. Li H, Sun J, Xu Z, et al. (2017) Multimodal 2D+3D facial expression recognition with deep fusion convolutional neural network[J]. IEEE T Multimedia, 1–1

  21. Yan J, Zheng W, Xu Q, et al. (2016) Sparse kernel reduced-rank regression for bimodal emotion recognition from facial expression and speech[J]. IEEE T Multimedia, 1–1

  22. Chakraborty BK, Kumar S, Bhuyan MK (2016) Extraction of informative regions of a face for facial expression recognition[J]. IET Comput Vis 10(6):567–576

    Article  Google Scholar 

  23. Uddin M Z, Hassan M M, Almogren A, et al. (2017) A facial expression recognition system using robust face features from depth videos and deep learning ☆[J]. Comput Electr Eng, 63

  24. Siddiqi MH, Alam MGR, Hong CS et al (2016) A novel maximum entropy Markov model for human facial expression recognition[J]. PLoS One 11(9):e0162702

    Article  Google Scholar 

  25. Chen J, Chen Z, Chi Z, et al. (2018) Facial expression recognition in video with multiple feature fusion[J]. IEEE Trans Affect Comput, PP(99):1–1

  26. Lee SH, Ro YM (2017) Partial matching of facial expression sequence using over-complete transition dictionary for emotion recognition[J]. IEEE Trans Affect Comput 7(4):389–408

    Article  Google Scholar 

  27. Wen G, Hou Z, Li H et al (2017) Ensemble of deep neural networks with probability-based fusion for facial expression recognition[J]. Cogn Comput 9(5):597–610

    Article  Google Scholar 

  28. Wen G et al (2017) Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cogn Comput 9(5):597–610

    Article  Google Scholar 

  29. Tsai HH, Chang YC (2017) "Facial expression recognition using a combination of multiple facial features and support vector machine." Soft Comput

  30. Kamarol SKA et al (2017) Joint facial expression recognition and intensity estimation based on weighted votes of image sequences. Pattern Recogn Lett 92:25–32

    Article  Google Scholar 

  31. Yan X, Young AW, Andrews TJ (2016) "Differences in holistic processing do not explain cultural differences in the recognition of facial expression." Q J Exp Psychol :1–15

  32. Liu M et al (2015) Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. IEEE Trans Image Process 25(12):5920–5932

    Article  MATH  Google Scholar 

  33. Yan H (2016) "Transfer subspace learning for cross-dataset facial expression recognition." Neurocomputing :S0925231216304623

  34. Danelakis A, Theoharis T, Pratikakis I (2016) A spatio-temporal wavelet-based descriptor for dynamic 3D facial expression retrieval and recognition. Vis Comput 32(6–8):1001–1011

    Article  Google Scholar 

  35. Jaina DK, Zhanga Z, Huanga K (2017) Multi angle optimal pattern-based deep learning for automatic facial expression recognition. Pattern Recogn Lett

  36. Mlakar U et al (2017) Multi-objective differential evolution for feature selection in facial expression recognition systems. Expert Syst Appl 89:129–137

    Article  Google Scholar 

Download references

Funding

This work was financially supported by the Excellent Specialties Program Development of Jiangsu Higher Education Institutions (PPZY2015C240).

Author information

Authors and Affiliations

  1. School of Internet of Things Technology, Wuxi Institute of Technology, Wuxi, Jiangsu, China

    Ying Xiao, Deyan Wang & Ligong Hou

  2. College of Internet of Things Engineering, Jiangnan University, Wuxi, Jiangsu, China

    Ying Xiao

  3. Management &Science University, Shah Alam, Malaysia

    Deyan Wang

Authors
  1. Ying Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Deyan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ligong Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ying Xiao.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xiao, Y., Wang, D. & Hou, L. Unsupervised emotion recognition algorithm based on improved deep belief model in combination with probabilistic linear discriminant analysis. Pers Ubiquit Comput 23, 553–562 (2019). https://doi.org/10.1007/s00779-019-01235-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-019-01235-y

Keywords

Search

Navigation