Skip to main content
SpringerLink
Log in

Multi-level uncorrelated discriminative shared Gaussian process for multi-view facial expression recognition

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In multi-view facial expression recognition, discriminative shared Gaussian process latent variable model (DS-GPLVM) gives better performance than that of linear and nonlinear multi-view learning-based methods. However, Laplacian-based prior used in DS-GPLVM only captures topological structure of data space without considering the inter-class separability of the data, and hence the obtained latent space is suboptimal. So, we propose a multi-level uncorrelated DS-GPLVM (ML-UDSGPLVM) model which searches a common uncorrelated discriminative latent space learned from multiple observable spaces. A novel prior is proposed, which not only depends on the topological structure of the intra-class data, but also on the local-between-class-scatter-matrix of the data onto the latent manifold. The proposed approach employs an hierarchical framework, in which, expressions are first divided into three sub-categories. Subsequently, each of the sub-categories are further classified to identify the constituent basic expressions. Experimental results show that the proposed method outperforms state-of-the-art methods in many instances.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Bettadapura, V.: Face expression recognition and analysis: the state of the art (2012). arXiv preprint arXiv:1203.6722

  2. Yan, J., Zheng, W., Xu, Q., Lu, G., Li, H., Wang, B.: Sparse kernel reduced-rank regression for bimodal emotion recognition from facial expression and speech. IEEE Trans. Multimed. 18(7), 1319–1329 (2016)

    Article  Google Scholar 

  3. Ekman, P., Friesen, W.V., Press, C.P.: Pictures of Facial Affect. Consulting Psychologists Press, Mountain View (1975)

    Google Scholar 

  4. Tie, Y., Guan, L.: A deformable 3-d facial expression model for dynamic human emotional state recognition. IEEE Trans. Circuits Syst. Video Technol. 23(1), 142–157 (2013)

    Article  Google Scholar 

  5. Kumar, S., Bhuyan, M., Chakraborty, B.K.: Extraction of informative regions of a face for facial expression recognition. IET Comput. Vis. 10(6), 567–576 (2016)

    Article  Google Scholar 

  6. Siddiqi, M.H., Ali, R., Khan, A.M., Park, Y.-T., Lee, S.: Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Trans. Image Process. 24(4), 1386–1398 (2015)

    Article  MathSciNet  Google Scholar 

  7. Eleftheriadis, S., Rudovic, O., Pantic, M.: Discriminative shared Gaussian processes for multiview and view-invariant facial expression recognition. IEEE Trans. Image Process. 24(1), 189–204 (2015)

    Article  MathSciNet  Google Scholar 

  8. Moore, S., Bowden, R.: Local binary patterns for multi-view facial expression recognition. Comput. Vis. Image Underst. 115(4), 541–558 (2011)

    Article  Google Scholar 

  9. Hu, Y., Zeng, Z., Yin, L., Wei, X., Tu, J., Huang, T.S.: A study of non-frontal-view facial expressions recognition. In: 19th International Conference on Pattern Recognition, 2008. ICPR 2008, pp. 1–4. IEEE (2008)

  10. Hu, Y., Zeng, Z., Yin, L., Wei, X., Zhou, X., Huang, T.S.: Multi-view facial expression recognition. In: 8th IEEE International Conference on Automatic Face Gesture Recognition, 2008. FG ’08, pp. 1–6 (2008)

  11. Hesse, N., Gehrig, T., Gao, H., Ekenel, H.K.: Multi-view facial expression recognition using local appearance features. In: 2012 21st International Conference on Pattern Recognition (ICPR), pp. 3533–3536. IEEE (2012)

  12. Rudovic, O., Pantic, M., Patras, I.: Coupled Gaussian processes for pose-invariant facial expression recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35(6), 1357–1369 (2013)

    Article  Google Scholar 

  13. Rudovic, O., Patras, I., Pantic, M.: Coupled Gaussian process regression for pose-invariant facial expression recognition. In: Computer Vision–ECCV 2010, pp. 350–363. Springer (2010)

  14. Rudovic, O., Patras, I., Pantic, M.: Regression-based multi-view facial expression recognition. In: 2010 20th International Conference on Pattern Recognition (ICPR), pp. 4121–4124, IEEE (2010)

  15. Zheng, W.: Multi-view facial expression recognition based on group sparse reduced-rank regression. IEEE Trans. Affect. Comput. 5(1), 71–85 (2014)

    Article  Google Scholar 

  16. Tariq, U., Yang, J., Huang, T.S.: Multi-view facial expression recognition analysis with generic sparse coding feature. In: Computer Vision–ECCV 2012. Workshops and Demonstrations, pp. 578–588, Springer (2012)

  17. Zheng, W., Tang, H., Lin, Z., Huang, T.S.: Emotion recognition from arbitrary view facial images. In: Computer Vision–ECCV 2010, pp. 490–503, Springer (2010)

  18. Eleftheriadis, S., Rudovic, O., Pantic, M.: View-constrained latent variable model for multi-view facial expression classification. In: International Symposium on Visual Computing, pp. 292–303. Springer (2014)

  19. Urtasun, R., Darrell, T.: Discriminative Gaussian process latent variable model for classification. In: Proceedings of the 24th International Conference on Machine Learning, pp. 927–934, ACM (2007)

  20. Shon, A., Grochow, K., Hertzmann, A., Rao, R.P.: Learning shared latent structure for image synthesis and robotic imitation. In: Advances in Neural Information Processing Systems, pp. 1233–1240 (2005)

  21. Ek, C.H., Lawrence, P.: Shared Gaussian Process Latent Variable Models. Ph.D. dissertation, PhD thesis (2009)

  22. Christopher, M.B.: Pattern Recognition and Machine Learning, vol. 16(4). Springer, New York (2006)

    MATH  Google Scholar 

  23. Chung, F.R.: Spectral Graph Theory, vol. 92. American Mathematical Soc., New York (1997)

    MATH  Google Scholar 

  24. Zhong, G., Li, W.-J., Yeung, D.-Y., Hou, X., Liu, C.-L.: Gaussian process latent random field. In: AAAI, pp. 679–684 (2010)

  25. He, X., Niyogi, P.: Locality preserving projections. In: Mozer, M.C., Jordan, M.I., Petsche, T. (eds.) Advances in Neural Information Processing Systems, pp. 153–160. MIT Press, Cambridge (2004)

    Google Scholar 

  26. Yu, X., Wang, X.: Uncorrelated discriminant locality preserving projections. IEEE Signal Process. Lett. 15, 361–364 (2008)

    Article  Google Scholar 

  27. Kan, M., Shan, S., Zhang, H., Lao, S., Chen, X.: Multi-view discriminant analysis. IEEE Trans. Pattern Anal. Mach. Intell. 38(1), 188–194 (2016)

    Article  Google Scholar 

  28. Hu, P., Peng, D., Guo, J., Zhen, L.: Local feature based multi-view discriminant analysis. Knowl. Based Syst. 149, 34–46 (2018)

    Article  Google Scholar 

  29. Peng, X., Feng, J., Xiao, S., Yau, W.-Y., Zhou, J.T., Yang, S.: Structured autoencoders for subspace clustering. IEEE Trans. Image Process. 27(10), 5076–5086 (2018)

    Article  MathSciNet  Google Scholar 

  30. Sugiyama, M.: Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis. J. Mach. Learn. Res. 8(May), 1027–1061 (2007)

    MATH  Google Scholar 

  31. Rahulamathavan, Y., Phan, R.C.-W., Chambers, J.A., Parish, D.J.: Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Trans. Affect. Comput. 4(1), 83–92 (2013)

    Article  Google Scholar 

  32. Kumar, S., Bhuyan, M., Chakraborty, B.K.: An efficient face model for facial expression recognition. In: 2016 Twenty Second National Conference on Communication (NCC), pp. 1–6. IEEE (2016)

  33. Zhong, L., Liu, Q., Yang, P., Liu, B., Huang, J., Metaxas, D.N.: Learning active facial patches for expression analysis. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2562–2569. IEEE (2012)

  34. Liu, P., Zhou, J.T., Tsang, I.W.-H., Meng, Z., Han, S., Tong, Y.: Feature disentangling machine-a novel approach of feature selection and disentangling in facial expression analysis. In: Computer Vision–ECCV 2014, pp. 151–166. Springer (2014)

  35. Nusseck, M., Cunningham, D.W., Wallraven, C., Bülthoff, H.H.: The contribution of different facial regions to the recognition of conversational expressions. J. Vis. 8(8), 1–1 (2008)

    Article  Google Scholar 

  36. Zhu, X., Ghahramani, Z., Lafferty, J.D.: Semi-supervised learning using gaussian fields and harmonic functions. In: Proceedings of the 20th International conference on Machine learning (ICML-03), pp 912–919 (2003)

  37. Zelnik-Manor, L., Perona, P.: Self-tuning spectral clustering. In: Mozer, M.C., Jordan, M.I., Petsche, T. (eds.) Advances in Neural Information Processing Systems, pp. 1601–1608. MIT Press, Cambridge (2004)

    Google Scholar 

  38. Lawrence, N.D., Quiñonero-Candela, J.: Local distance preservation in the gp-lvm through back constraints. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 513–520. ACM (2006)

  39. Bertsekas, D.P.: Constrained Optimization and Lagrange Multiplier Methods. Academic press, Cambridge (2014)

    MATH  Google Scholar 

  40. Rasmussen, C.E.: Gaussian Processes for Machine Learning, vol. 1. MIT Press, Cambridge (2006)

    MATH  Google Scholar 

  41. Goodall, C.: Procrustes methods in the statistical analysis of shape. J. R. Stat. Soc. Ser. B (Methodol.) 53, 285–339 (1991)

    MathSciNet  MATH  Google Scholar 

  42. Hotelling, H.: Relations between two sets of variates. Biometrika 28(3/4), 321–377 (1936)

    Article  Google Scholar 

  43. Rupnik, J., Shawe-Taylor, J.: Multi-view canonical correlation analysis. In: Conference on Data Mining and Data Warehouses (SiKDD 2010), pp. 1–4 (2010)

  44. Sharma, A., Kumar, A., Daume, H., Jacobs, D.W.: Generalized multiview analysis: a discriminative latent space. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2160–2167. IEEE (2012)

  45. Dhall, A., Goecke, R., Lucey, S., Gedeon, T.: Static facial expression analysis in tough conditions: data, evaluation protocol and benchmark. In: 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), pp. 2106–2112. IEEE (2011)

  46. Zhang, T., Zheng, W., Cui, Z., Zong, Y., Yan, J., Yan, K.: A deep neural network-driven feature learning method for multi-view facial expression recognition. IEEE Trans. Multimed. 18(12), 2528–2536 (2016)

    Article  Google Scholar 

  47. Liu, Y., Zeng, J., Shan, S., Zheng, Z.: Multi-channel pose-aware convolution neural networks for multi-view facial expression recognition. In: 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018), pp. 458–465. IEEE (2018)

  48. Kang, Z., Pan, H., Hoi, S.C., Xu, Z.: Robust graph learning from noisy data. IEEE Trans. Cybern. (2019)

  49. Li, D., Li, Z., Luo, R., Deng, J., Sun, S.: Multi-pose facial expression recognition based on generative adversarial network. IEEE Access 7, 143980–143989 (2019)

    Article  Google Scholar 

  50. Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–10. IEEE (2016)

  51. Kim, B.-K., Roh, J., Dong, S.-Y., Lee, S.-Y.: Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. J. Multimodal User Interfaces 10(2), 173–189 (2016)

    Article  Google Scholar 

  52. Li, J.. Lam, E.Y.: Facial expression recognition using deep neural networks. In: 2015 IEEE International Conference on Imaging Systems and Techniques (IST), pp. 1–6. IEEE (2015)

  53. Khorrami, P., Paine, T., Huang, T.: Do deep neural networks learn facial action units when doing expression recognition? In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 19–27 (2015)

  54. Lawrence, N.D.: Large scale learning with the gaussian process latent variable model. Technical Report CS-06-05, University of Sheffield, 2006. 3, 4, 7, Technical Report (2008)

Download references

Author information

Authors and Affiliations

  1. ABV-IIITM Gwalior, Gwalior, 474015, India

    Sunil Kumar

  2. Department of Electronics and Electrical Engineering, IIT Guwahati, Guwahati, 781039, India

    M. K. Bhuyan

  3. Department of Computer Science, Chubu University, Kasugai, Aichi, 487-8501, Japan

    Yuji Iwahori

Authors
  1. Sunil Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. K. Bhuyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuji Iwahori
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sunil Kumar.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar, S., Bhuyan, M.K. & Iwahori, Y. Multi-level uncorrelated discriminative shared Gaussian process for multi-view facial expression recognition. Vis Comput 37, 143–159 (2021). https://doi.org/10.1007/s00371-019-01788-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-019-01788-2

Keywords

Search

Navigation