Skip to main content
SpringerLink
Log in

Rapid facial expression recognition under part occlusion based on symmetric SURF and heterogeneous soft partition network

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Recently, deep learning has made great achievements in facial expression recognition. However, occlusion and large skew will greatly affect the accuracy of facial expression recognition in practice. Therefore, we propose a novel framework based on symmetric SURF and heterogeneous soft partition network to quickly recognize facial recognition under partial occlusion. In this framework, an occlusion detection module based on symmetric SURF is presented to detect the occlusion part, which helps to locate the horizontal symmetric area of the occlusion area. After that, a face inpainting module based on mirror transition is presented to rapidly accomplish the face inpainting under the unsupervised circumstance. Moreover, a recognition network based on heterogeneous soft partitioning is proposed for the facial expression recognition. After heterogeneous soft partitioning, the weights of each part are input and to into the recognition network as more prior information for training. Finally, we feed the weighted image into the trained neural network for expression recognition. Experimental results show that the accuracy of the proposed method is respectively 7% and 8% higher than the average accuracies from the state-of-the-art methods on Cohn-Kanade (CK +) and fer2013 datasets. Besides, the run time of our method is 2.38 s faster than the most advanced.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Bastanfard, A, Takahashi H, Nakajima M (2004) Toward E-appearance of human face and hair by age, expression and rejuvenation. International Conference on Cyberworlds IEEE, 2004

  2. Bay H et al (2006) SURF: speeded up robust features. European conference on computer vision 110(3):404–417

    Google Scholar 

  3. Cai J, Han H, Shan S, Chen XJaCV, Recognition P (2019) FCSR-GAN: joint face completion and super-resolution via multi-task learning. arXiv: computer vision and pattern recognition. arXiv:1911.01045

  4. Chang T, Wen G, Hu Y, Ma J (2018) Facial expression recognition based on complexity perception classification algorithm. arXiv:1803.00185

  5. Chen LC, Hsieh JW, Yan Y, Chen DY (2015) Vehicle make and model recognition using sparse representation and symmetrical SURFs. Pattern Recogn 48(6):1979–1998

    Google Scholar 

  6. Ghimire D, Lee J (2016) Geometric feature-based facial expression recognition in image sequences using multi-class AdaBoost and support vector machines. Sensors 13(6):7714–7734

    Google Scholar 

  7. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial networks. Proceedings of the international conference on neural information processing systems (NIPS 2014), 3:2672-2680

  8. Horé A, Ziou D (2010) Image quality metrics: PSNR vs. SSIM. In: 20th international conference on pattern recognition, ICPR 2010, Istanbul, Turkey, August 2010: 23–26

  9. Hsieh JW, Chen LC, Chen DY (2014) Symmetrical SURF and its applications to vehicle detection and vehicle make and model recognition. IEEE Trans Intell Transp Syst 15(1):6–20

    Google Scholar 

  10. Hsieh JW, Chen LC, Chen DY, Cheng SC (2013) Vehicle make and model recognition using symmetrical SURF. In: advanced video and signal based surveillance (AVSS), 2013 10th IEEE international conference on IEEE, 2013: 472-477

  11. Huang MW, Wang ZW, Ying ZL (2010) A new method for facial expression recognition based on sparse representation plus LBP. In: International Congress on Image & Signal Processing: 1750-1754

  12. Jabid T, Kabir MH, Chae O (2010) Robust facial expression recognition based on local directional pattern. ETRI J 32(5):784–794

    Google Scholar 

  13. Jeon J, Park J-C, Jo Y, Nam C, Bae K-H, Hwang Y, Kim D-S (2016). A real-time facial expression recognizer using deep neural network. In: proceedings of the 10th international conference on ubiquitous information management and communication, 2016:1-4

  14. Kaggle. Fer (2013) dataset. https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge/data. Accessed 1 July 2013

  15. Kan M, Shan S, Chang H, Chen X (2014) Stacked progressive auto-encoders (spae) for face recognition across poses. Proc IEEE Conf Comput Vis Pattern Recognit 2014:1883–1890

    Google Scholar 

  16. Kim B, Roh J, Dong SY, Lee SY (2016) Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. Journal on Multimodal User Interfaces 10(2):173–189

    Google Scholar 

  17. Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444

    Google Scholar 

  18. Li Y, Liu J, Wang L (2018) Lightweight network research based on deep learning: a review. In: 2018 37th Chinese control conference (CCC), 2018:9021-9026

  19. Li Y, Liu S, Yang J, Yang M (2017) Generative face completion. arXiv:1704.05838

  20. Li D, Wen G, Hou Z, Huan E, Hu Y, Li H (2019) RTCRelief-F: an effective clustering and ordering-based ensemble pruning algorithm for facial expression recognition. Knowl Inf Syst 59(1):219–250

    Google Scholar 

  21. Liu Z, Luo P, Wang X, Tang X (2015) Deep learning face attributes in the wild. Proceedings of the IEEE international conference on computer vision 2015:3730–3738

    Google Scholar 

  22. Lowe DG (2004) Distinctive image features from scale-invariant Keypoints. Int J Comput Vis 60(2):91–110

    Google Scholar 

  23. Lucey P, Cohn JF, Kanade T, Saragih J, Matthews I (2010) The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. Computer Vision & Pattern Recognition Workshops 2010:94–101

    Google Scholar 

  24. Lyons MJ, Akamatsu S, Kamachi MG, Gyoba J (1998) Coding facial expressions with Gabor wavelets. In: Proceedings of the 3rd IEEE international conference on automatic face and gesture recognition. Nara: IEEE, 1998: 200–205

  25. Meng Z, Liu P, Cai J, Han S, Tong Y (2017) Identity-aware convolutional neural network for facial expression recognition. In: 2017 12th IEEE international conference on Automatic Face & Gesture Recognition (FG 2017), 2017. IEEE, pp 558-565

  26. Ojansivu V, Heikkilä J (2018) Blur insensitive texture classification using local phase quantization. International Conference on Image & Signal Processing 2008:236–243

    Google Scholar 

  27. Online. Available: http://opencv.org/. Accessed 3 Aug 2017

  28. Patel MJ, Pandya MS, Shah VJ (2018) Review on generative adversarial networks. International Journal of Technical Innovation in Modern Engineering & Science (IJTIMES) 4(7):1230–1238

    Google Scholar 

  29. Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros A (2016) Context encoders: feature learning by Inpainting. arXiv: computer vision and pattern recognition. arXiv:1604.07379

  30. Qayyum H, Majid M, Anwar SM, Khan BJ (2017) Facial expression recognition using stationary wavelet transform features. Math Probl Eng 2017:1–9

    Google Scholar 

  31. Saunders C, Stitson MO, Weston J, Holloway R, Bottou L, Scholkopf B, Smola A (2002) Support vector machine. Computer ence 1(4):1–28

    Google Scholar 

  32. Sun Y, Zheng L, Yang Y, Tian Q, Wang S (2018) Beyond part models: person retrieval with refined part pooling (and a strong convolutional baseline). Proceedings of the European Conference on Computer Vision (ECCV) 2018:480–496

    Google Scholar 

  33. Wang KF, Gou C, Duan YJ, Lin YL, Wang FY (2017) Generative adversarial networks: the state of the art and beyond. Acta Automat Sin 43(3):321–332

    Google Scholar 

  34. Yang Y, Guo X, Ma J, Ma L, Ling H (2019) LaFIn: generative landmark guided face Inpainting. arXiv: computer vision and pattern recognition. arXiv:1911.11394

  35. Yao NM, Guo QP, Qiao FC, Chen H, Wang HA (2016) Robust facial expression recognition with generative adversarial networks. Zidonghua Xuebao/acta Automatica Sinica 44(5):865–877

    Google Scholar 

  36. Yong LI, Lin XZ, Jiang MY (2018) Facial expression recognition with cross-connect LeNet-5 network. Acta Automat Sin 44(1):176–182

    Google Scholar 

  37. Yu Z (2015) Image based static facial expression recognition with multiple deep network learning. Acm on International Conference on Multimodal Interaction 2015:435–442

    Google Scholar 

  38. Zhang J, Kan M, Shan S, Chen X (2016) Occlusion-free face alignment: deep regression networks coupled with de-corrupt autoencoders. Proc IEEE Conf Comput Vis Pattern Recognit 2016:3428–3437

    Google Scholar 

  39. Zhu Z, Luo P, Wang X, Tang X (2013) Deep learning identity-preserving face space. International conference on computer vision:113–120

Download references

Acknowledgements

Special thanks to Mr. Junan Chen and Prof. Weiwen Zhang for their participation in writing or technical editing of the manuscript. In addition, this work was supported by the Key-Area Research and Development Program of Guangdong Province under Grant 2019B010153002, the National Natural Science Foundation of China under Grant 61702111, the National Nature Science Foundation of China-Guangdong Joint Fund under Grant 83-Y40G33-9001-18/20, the National Key Research and Development Program of China under Grant 2017YFB1201203, the Guangdong Provincial Key Laboratory of Cyber-Physical System under Grant 2016B030301008, the National Natural Science Foundation of Guangdong Joint Fund under Grant U1801263, the National Natural Science Foundation of Guangdong Joint Fund under Grant U1701262, the Key-Area Research and Development Program of Guangdong Province under Grant 2018B010109007, the “Blue Fire Plan” (Huizhou) Industry-University-Research Joint Innovation Fund 2017 Project of the Ministry of Education under Grant CXZJHZ201730, and Key-Area Research and Development Program of Guangdong Province under Grant 2019B010109001.

Author information

Authors and Affiliations

  1. School of Computers, Guangdong University of Technology, Guangzhou, 510006, China

    Ke Hu, Guoheng Huang, Ying Yang & Lianglun Cheng

  2. Department of Computer and Information Science, University of Macau, Macau, 999078, SAR, China

    Chi-Man Pun

  3. School of Information Engineering, Guangdong University of Technology, Guangzhou, 510006, China

    Wing-Kuen Ling

Authors
  1. Ke Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guoheng Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ying Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chi-Man Pun
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wing-Kuen Ling
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lianglun Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Guoheng Huang, Chi-Man Pun, Wing-Kuen Ling or Lianglun Cheng.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, K., Huang, G., Yang, Y. et al. Rapid facial expression recognition under part occlusion based on symmetric SURF and heterogeneous soft partition network. Multimed Tools Appl 79, 30861–30881 (2020). https://doi.org/10.1007/s11042-020-09566-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09566-2

Keywords

Search

Navigation