Skip to main content
SpringerLink
Log in

Pose-robust feature learning for facial expression recognition

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

Automatic facial expression recognition (FER) from non-frontal views is a challenging research topic which has recently started to attract the attention of the research community. Pose variations are difficult to tackle and many face analysis methods require the use of sophisticated normalization and initialization procedures. Thus head-pose invariant facial expression recognition continues to be an issue to traditional methods. In this paper, we propose a novel approach for pose-invariant FER based on pose-robust features which are learned by deep learning methods — principal component analysis network (PCANet) and convolutional neural networks (CNN) (PRP-CNN). In the first stage, unlabeled frontal face images are used to learn features by PCANet. The features, in the second stage, are used as the target of CNN to learn a feature mapping between frontal faces and non-frontal faces. We then describe the non-frontal face images using the novel descriptions generated by the maps, and get unified descriptors for arbitrary face images. Finally, the pose-robust features are used to train a single classifier for FER instead of training multiple models for each specific pose. Our method, on the whole, does not require pose/ landmark annotation and can recognize facial expression in a wide range of orientations. Extensive experiments on two public databases show that our framework yields dramatic improvements in facial expression analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Zheng W M. Multi-view facial expression recognition based on group sparse reduced-rank regression. IEEE Transactions on Affective Computing, 2014, 5(1): 71–85

    Article  Google Scholar 

  2. Eleftheriadis S, Rudovic O, Pantic M. Discriminative shared gaussian processes for multiview and view-invariant facial expression recognition. IEEE Transactions on Image Processing, 2015, 24(1): 189–204

    Article  MathSciNet  Google Scholar 

  3. Liu P, Han S Z, Meng Z B, Tong Y. Facial expression recognition via a boosted deep belief network. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1805–1812

    Google Scholar 

  4. Zeng Z, Pantic M, Roisman G I, Huang T S. A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis andMachine Intelligence, 2009, 31(1): 39–58

    Article  Google Scholar 

  5. Moore S, Bowden R. Local binary patterns for multi-view facial expression recognition. Computer Vision and Image Understanding, 2011, 115(4): 541–558

    Article  Google Scholar 

  6. Hesse N, Gehrig T, Gao H, Ekenel H K. Multi-view facial expression recognition using local appearance features. In: Proceedings of International Conference on Pattern Recognition. 2012, 3533–3536

    Google Scholar 

  7. Rudovic O, Pantic M, Patras I. Coupled Gaussian processes for poseinvariant facial expression recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(6): 1357–1369

    Article  Google Scholar 

  8. Kumano S, Otsuka K, Yamato J, Maeda E, Sato Y. Pose-invariant facial expression recognition using variable-intensity templates. International Journal of Computer Vision, 2009, 83(2): 178–194

    Article  Google Scholar 

  9. Biswas A, Ghose M K. Expression invariant face recognition using DWT sift features. International Journal of Computer Applications, 2014, 92(2): 30–32

    Article  Google Scholar 

  10. Jian S, Hu C B, Aggarwal J K. Facial expression recognition with temporal modeling of shapes. In: Proceedings of IEEE International Conference on Computer Vision. 2011, 1642–1649

    Google Scholar 

  11. Girisha H, Sreepathi B, Karibasappa K. Multi-view face recognition using local binary pattern. International Journal of Computer Science and Information Technologies, 2014, 5(3): 2978–2981

    Google Scholar 

  12. Dahmane M, Meunier J. Emotion recognition using dynamic gridbased HoG features. In: Proceedings of IEEE International Conference on Automatic Face & Gesture Recognition andWorkshops. 2011, 884–888

    Google Scholar 

  13. Hu Y X, Zeng Z, Yin L J, Wei X Z, Tu J, Huang T S. A study of non-frontal-view facial expressions recognition. In: Proceedings of International Conference on Pattern Recognition. 2008, 1–4

    Google Scholar 

  14. Rudovic O, Patras I, Pantic M. Regression-based multiview facial expression recognition. In: Proceedings of International Conference on Pattern Recognition. 2010, 4121–4124

    Google Scholar 

  15. Hu Y X, Zeng Z H, Yin L J, Wei X Z, Zhou X, Huang T S. Multi-view facial expression recognition. In: Proceedings of the 8th IEEE International Conference on Automatic Face and Gesture Recognition. 2008, 56–61

    Google Scholar 

  16. Gupta S K, Agrwal S, Meena Y K, Nain N. A hybrid method of feature extraction for facial expression recognition. In: Proceedings of Signal-Image Technology and Internet-Based Systems. 2011, 422–425

    Google Scholar 

  17. Ding C X, Tao D C. A comprehensive survey on pose-invariant face recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2015

    Google Scholar 

  18. Tong Y, Chen J X, Ji Q. A unified probabilistic framework for spontaneous facial action modeling and unerstanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(2): 258–273

    Article  Google Scholar 

  19. Zheng W M, Tang H, Lin Z C, Huang T S. Emotion recognition from arbitrary view facial images. In: Proceedings of European Conference on Computer Vision. 2010, 490–503

    Google Scholar 

  20. Sung J, Kim D. Real-time facial expression recognition using STAAM and layered GDA classifier. Image and Vision Computing, 2009, 27(9): 1315–1325

    Article  Google Scholar 

  21. Tang H, Hasegawa-Johnson M, Huang T. Non-frontal view facial expression recognition based on ergodic hidden markov model supervectors. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2010, 1202–1207

    Google Scholar 

  22. Ranzato M, Susskind J, Mnih V, Hinton G. On deep generative models with applications to recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2011, 2857–2864

    Google Scholar 

  23. Rifai S, Bengio Y, Courville A, Vincent P, Mirza M. Disentangling factors of variation for facial expression recognition. In: Proceedings of European Conference on Computer Vision. 2012, 808–822

    Google Scholar 

  24. Eleftheriadis S, Rudovic O, Pantic M. Shared Gaussian process latent variable model for multi-view facial expression recognition. In: Proceedings of International Symposium on Visual Computing. 2013, 527–538

    Google Scholar 

  25. Liu M Y, Shan S G, Wang R P, Chen X L. Learning expressionlets on spatio-temporal manifold for dynamic facial expression recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2014, 1749–1756

    Google Scholar 

  26. Rifai S, Vincent P, Muller X, Glorot X, Bengio Y. Contracting autoencoders: explicit invariance during feature extraction. In: Proceedings of International Conference on Machine Learning. 2011, 833–840

    Google Scholar 

  27. Saudagare P V, Chaudhari D S. Facial expression recognition using neural network — an overview. International Journal of Soft Computing and Engineering, 2012, 2(1): 224–227

    Google Scholar 

  28. Li J G, Zhang Y M. Learning surf cascade for fast and accurate object detection. In: Proceedings of the 26th IEEE Conference on Computer Vision and Pattern Recognition. 2013, 3468–3475

    Google Scholar 

  29. Viola P, Jones M. Rapid object detection using a boosted cascade of simple features. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2001, 511–518

    Google Scholar 

  30. Yin L J, Wei X Z, Sun Y, Wang J, Rosato M J. A 3D facial expression database for facial behavior research. In: Proceedings of Automatic face and gesture recognition. 2006, 211–216

    Google Scholar 

  31. Dhall A, Goecke R, Lucey S, Gedeon T. Static facial expressions analysis in tough conditions: data, evaluation protocol and benchmark. In: Proceedings of IEEE International Conference on Computer Vision. 2011, 2106–2112

    Google Scholar 

  32. Zheng WM, Tang H, Lin Z C, Huang T S. A novel approach to expression recognition from non-frontal face images. In: Proceedings of the 12th IEEE International Conference on Computer Vision. 2009, 1901–1908

    Google Scholar 

  33. Tariq U, Yang J, Huang T. Maximum margin gmm learning for facial expression recognition. In: Proceedings of Automatic Face and Gesture Recognition. 2013, 1–6

    Google Scholar 

  34. Tariq U, Yang J C, Huang T S. Supervised super-vector encoding for facial expression recognition. Pattern Recognition, 2014, 89–95

    Google Scholar 

  35. Jampour M, Mauthner T, Bischof H. Multi-view facial expressions recognition using local linear regression of sparse codes. In: Proceedings of the 20th Computer Vision Winter Workshop Paul Wohlhart, 2015

    Google Scholar 

  36. Tariq U, Yang J C, Huang T S. Multi-view facial expression recognition analysis with generic sparse coding feature. In: Proceedings of European Conference on Computer Vision. 2012, 578–588

    Google Scholar 

  37. Kan M N, Shan S G, Zhang H H, Lao S H, Chen X L. Multi-view discriminant analysis. In: Proceedings of European Conference on Computer Vision. 2012, 808–821

    Google Scholar 

  38. Sharma A, Kumar A, Daume H, Jacobs D W. Generalized multiview analysis: a discriminative latent space. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2012, 2160–2167

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang, 212013, China

    Feifei Zhang, Yongbin Yu, Qirong Mao, Jianping Gou & Yongzhao Zhan

Authors
  1. Feifei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongbin Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qirong Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianping Gou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yongzhao Zhan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qirong Mao.

Additional information

Feifei Zhang received her BS degree in computer science and technology from Xuchang University, China in 2013. she is currently a MS candidate in computer technology with the School of Computer Science and Communication Engineering at Jiangsu University, China. Her research interests include affect computing and pattern recognition.

Yongbin Yu received his BS degree in computer science and technology from Jiangsu University, China in 2012. He received his MS degree of computer application technology in the School of Computer Science and Communication Engineering at Jiangsu University. His research interests include affect computing and pattern recognition.

Qirong Mao received her MS and PhD degrees from Jiangsu University, China in 2002 and 2009, both in computer application technology. She is currently an associate professor of the School of Computer Science and Communication Engineering, Jiangsu University. Her research interests include affective computing, pattern recognition, and multimedia analysis. She has published over 30 technical articles, some of them in premium journals and conferences such as ACMMultimedia, IEEE Transactions onMultimedia. She is a member of the IEEE.

Jianping Gou received the BS degree in computer science from Beifang University of Nationalities, China in 2005, the MS degree in computer science from the Southwest Jiaotong University, China in 2008, and the PhD degree in computer science from University of Electronic Science and Technology of China, China in 2012. He is currently a lecturer in School of Computer Science and Telecommunication Engineering, JiangSu University, China. His current research interests include pattern classification, machine learning.

Yongzhao Zhan received his BS degree from Fuzhou University, China in 1984 and his PhD degree from Nanjing University, China in 2000, both in computer science and technology. He is currently a professor and the dean of the School of Computer Science and Communication Engineering, Jiangsu University, China. His research interests include multimedia analysis and pattern recognition. He has published over 60 technical articles.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, F., Yu, Y., Mao, Q. et al. Pose-robust feature learning for facial expression recognition. Front. Comput. Sci. 10, 832–844 (2016). https://doi.org/10.1007/s11704-015-5323-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11704-015-5323-3

Keywords

Search

Navigation