Skip to main content

Advertisement

Springer Nature Link
Log in

Deep peak-neutral difference feature for facial expression recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial expression recognition (FER) is important in vision-related applications. Deep neural networks demonstrate impressive performance for face recognition; however, it should be noted that this method relies heavily on a great deal of manually labeled training data, which is not available for facial expressions in real-world applications. Hence, we propose a powerful facial feature called deep peak–neutral difference (DPND) for FER. DPND is defined as the difference between two deep representations of the fully expressive (peak) and neutral facial expression frames. The difference tends to emphasize the facial parts that are changed in the transition from the neutral to the expressive face and to eliminate the face identity information retained in the fine-tuned deep neural network for facial expression, the network has been trained on large-scale face recognition dataset. Furthermore, unsupervised clustering and semi-supervised classification methods are presented to automatically acquire the neutral and peak frames from the expression sequence. The proposed facial expression feature achieved encouraging results on public databases, which suggests that it has strong potential to recognize facial expressions in real-world applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Almaev TR, Valstar MF (2013) Local gabor binary patterns from three orthogonal planes for automatic facial expression recognition. In: Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference IEEE, pp 356–361

  2. An L, Yang S, Bhanu B (2015) Efficient smile detection by extreme learning machine. Neurocomputing 149:354–363

    Article  Google Scholar 

  3. Arthur D, Vassilvitskii S (2007) "K-means++: the advantages of careful seeding." SODA ‘07: proceedings of the eighteenth annual ACM-SIAM symposium on discrete Algorithms, pp. 1027–1035

  4. Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 27(2):1–27

    Article  Google Scholar 

  5. Chao WL, Ding JJ, Liu JZ (2015) Facial expression recognition based on improved local binary pattern and class-regularized locality preserving projection. Signal Process 117:1–10

    Article  Google Scholar 

  6. Chen J, Chen D, Gong Y, et al (2012) Facial expression recognition using geometric and appearance features. In: Proceedings of the 4th international conference on internet multimedia computing and service. ACM, pp 29–33

  7. Chen J, Luo N, Liu Y, Liu L, Zhang K, Kolodziej J (2016) A hybrid intelligence-aided approach to affect-sensitive e-learning. Computing 98(1–2):215–233

    Article  MathSciNet  Google Scholar 

  8. Corneanu CA, Simon MO, Cohn JF, Guerrero SE (2016) Survey on rgb, 3d, thermal, and multimodal approaches for facial expression recognition: history, trends, and affect-related applications. IEEE Trans Pattern Anal Mach Intell 38(8):1548–1568

    Article  Google Scholar 

  9. Dapogny A, Bailly K, Dubuisson S (2017) Dynamic pose-robust facial expression recognition by multi-view pairwise conditional random forests. IEEE Trans Affect Comput 99:1–14

    Article  Google Scholar 

  10. Ding GG, Guo YC, Zhou JL (2016) Large-scale cross-modality search via collective matrix factorization hashing. IEEE Transactions Image Processing 25(11):5427–5440

    Article  MathSciNet  Google Scholar 

  11. Ding GG, Zhou JL, Guo YC (2017) Large-scale image retrieval with sparse embedded hashing. Neurocomputing 257:24–36

    Article  Google Scholar 

  12. Ding H, Zhou S K, Chellappa R (2017) Facenet2expnet: Regularizing a deep face recognition net for expression recognition. In: Automatic Face & Gesture Recognition (FG 2017), 2017 12th IEEE International Conference. IEEE, pp 118–126

  13. Guo YC, Ding GG, Liu L (2017) Learning to hash with optimized anchor embedding for scalable retrieval. IEEE Transactions Image Processing 26(3):1344–1354

    Article  MathSciNet  Google Scholar 

  14. Guo YC, Ding GG, Han JG (2017) Robust quantization for general similarity search. IEEE Transactions Image Processing, pp 949–963

    Article  MathSciNet  Google Scholar 

  15. Guo YC, Ding GG, Han JG (2017) Zero-shot learning with transferred samples. IEEE Transactions Image Processing 26(7):3277–3290

    Article  MathSciNet  Google Scholar 

  16. Jung H, Lee S, Yim J, et al (2015) Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE International Conference on Computer Vision IEEE, pp 2983–2991

  17. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems. NIPS, pp 1097–1105

  18. Lee S, Plataniotis K, Ro Y (2014) Intra-class variation reduction using training expression images for sparse representation based facial expression recognition. IEEE Trans Affect Comput 5(3):340–351

    Article  Google Scholar 

  19. Levi G, Hassner T (2015) Emotion recognition in the wild via convolutional neural networks and mapped binary patterns. In: Proceedings of the 2015 ACM on international conference on multimodal interaction. ACM, pp 503–510

  20. Li YF, Zhou ZH (2015) Towards making unlabeled data never hurt. IEEE Trans Pattern Anal Mach Intell 37(1):175–188

    Article  Google Scholar 

  21. Li H, Lin Z, Shen X, et al (2015) A convolutional neural network cascade for face detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition IEEE pp 5325–5334

  22. Liu M, Li S, Shan S, et al (2013) Au-aware deep networks for facial expression recognition. In: Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops. IEEE, pp 1–6

  23. Lopes AT, Aguiar ED, Souza AFD, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recogn 61:610–628

    Article  Google Scholar 

  24. Lu X, Yuan Y, Zheng X (2013) Image super-resolution via double sparsity regularized manifold learning. IEEE Trans Circuits Syst Video Technol 23(12):2022–2033

    Article  Google Scholar 

  25. Lu X, Yuan Y, Yan P (2014) Alternatively constrained dictionary learning for image super-resolution. IEEE Trans Cybern 44(3):366–377

    Article  Google Scholar 

  26. Lu X, Yuan Y, Zheng X (2017) Jointly dictionary learning for change detection in multispectral imagery. IEEE Transactions on Cybernetics (IEEE) 47(4):884–897

    Article  Google Scholar 

  27. Lu X, Li X, Zheng X (2017) Latent semantic minimal hashing for image retrieval. IEEE Transactions on Image processing (IEEE) 26(1):355–368

    Article  MathSciNet  Google Scholar 

  28. Luo Z, Liu L, Chen J, et al (2016) Spontaneous smile recognition for interest detection. In: Proceedings of the Chinese Conference on Pattern Recogntion IEEE, pp 119–130

  29. Mohammadi MR, Fatemizadeh E, Mahoor MH (2014) PCA-based dictionary building for accurate facial expression recognition via sparse representation. J Vis Commun Image Represent 25(5):1082–1092

    Article  Google Scholar 

  30. Mollahosseini A, Graitzer G, Borts E, et al (2014) Expressionbot: an emotive lifelike robotic face for face-to-face communication. In: Humanoid Robots (Humanoids), 2014 14th IEEE-RAS International Conference IEEE, pp 1098–1103

  31. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: Applications of Computer Vision (WACV), 2016 I.E. Winter Conference. IEEE, pp 1–10

  32. Ng HW, Nguyen VD, Vonikakis V, et al (2015) Deep learning for emotion recognition on small datasets using transfer learning. Proceedings of the 2015 ACM on international conference on multimodal interaction. ACM, pp 443–449

  33. Parkhi O M, Vedaldi A, Zisserman A (2015) Deep face recognition. In: Proceedings of the British machine vision conference. BMVC, pp 1–12

  34. Scherer S, Stratou G, Mahmoud M, et al (2013) Automatic behavior descriptors for psychological disorder analysis. In: Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops. IEEE, pp 1–8

  35. Szegedy C, Liu W, Jia Y, et al (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition IEEE, pp 1–9

  36. Tang Y (2013) Deep learning using linear support vector machines. arXiv preprint arXiv:1306,0239

  37. VGG face dataset. http://www.robots.ox.ac.uk/~vgg/data/vgg_face/

  38. Wang Z, Ying Z (2012) Facial expression recognition based on local phase quantization and sparse representation. In: Natural Computation (ICNC), 2012 Eighth International Conference IEEE, pp 222–225

  39. Yao XW, Han JW, Cheng G (2016) Semantic annotation of high-resolution satellite images via weakly supervised learning. IEEE Trans Geosci Remote Sens 54:3660–3671

    Article  Google Scholar 

  40. Yao XW, Han JW, Zhang DW (2017) Revisiting co-saliency detection: a novel approach based on two-stage multi-view spectral rotation co-clustering. IEEE Trans Image Process 26:3196–3209

    Article  MathSciNet  Google Scholar 

  41. Yin L, Wei X, Sun Y, et al (2006) A 3D facial expression database for facial behavior research. In: automatic face and gesture recognition, 2006 FGR 2006 7th international conference. IEEE, pp 211–216

  42. Zhang X, Mahoor M, Mavadati S (2015) Facial expression recognition using {l} _ {p} -norm MKL multiclass-SVM. Machine Vision & Applications 26(4):467–483

    Article  Google Scholar 

  43. Zhang DW, Han JW, Li C (2015) Detection of co-salient objects by looking deep and wide. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2994–3002

  44. Zhang T, Zheng W, Cui Z, Zong Y, Yan J, Yan K (2016) A deep neural network-driven feature learning method for multi-view facial expression recognition. IEEE Transactions on Multimedia 18(12):2528–2536

    Article  Google Scholar 

  45. Zhang DW, Han JW, Jiang L (2017) Revealing event saliency in unconstrained video collection. IEEE Trans Image Process 26:1746–1758

    Article  MathSciNet  Google Scholar 

  46. Zhang DW, Meng DY, Han JW (2017) Co-saliency detection via a self-paced multiple-instance learning framework. IEEE Trans Pattern Anal Mach Intell 39:865–878

    Article  Google Scholar 

  47. Zhao X, Liang X, Liu L, et al (2016) Peak-piloted deep network for facial expression recognition. In: European conference on computer vision. Springer International Publishing, pp 425–442

  48. Zhao J, Han J, Shao L (2017) Unconstrained face recognition using a set-to-set distance measure on deep learned features. IEEE Transactions on Circuits and Systems for Video Technology, (99) :1–11

Download references

Acknowledgements

This work was supported by the National Social Science Foundation of China (Grant no. 16BSH107).

Author information

Authors and Affiliations

  1. National Engineering Research Center for E-Learning, Central China Normal University, Wuhan, China

    Jingying Chen, Ruyi Xu & Leyuan Liu

Authors
  1. Jingying Chen
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Ruyi Xu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Leyuan Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Leyuan Liu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, J., Xu, R. & Liu, L. Deep peak-neutral difference feature for facial expression recognition. Multimed Tools Appl 77, 29871–29887 (2018). https://doi.org/10.1007/s11042-018-5909-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-5909-5

Keywords