Skip to main content
SpringerLink
Log in

Deep Partial Occlusion Facial Expression Recognition via Improved CNN

  • Conference paper
  • First Online:
Advances in Visual Computing (ISVC 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12509))

Included in the following conference series:

Abstract

Facial expression recognition (FER) can indicate a person’s emotion state, that is of great importance in virtual human modelling and communication. However, FER suffers from a partial occlusion problem when applied under an unconstrained environment. In this paper, we propose to use facial expressions with partial occlusion for FER. This differs from the most conventional FER problems which assume that facial images are detected without any occlusion. To this end, by reconstructing the partially occluded facial expression database, we propose a 20-layer “VGG + residual” CNN network based on the improved VGG16 network, and adapt a hybrid feature strategy to parallelize the Gabor filter with the above CNN. We also optimize the components of the model by LMCL and momentum SGD. The results are then combined with a certain weight to get the classification results. The advantages of this method are demonstrated by multiple sets of experiments and cross-database tests.

Supported by the Natural Science Foundation of China under grant nos. 61672375 and 61170118.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Brink, H., Vadapalli, H.B.: Deformable part models with CNN features for facial landmark detection under occlusion. In: ACM Press the South African Institute of Computer Scientists and Information Technologists, pp. 681–685 (2017)

    Google Scholar 

  2. Cheng, Y., Jiang, B., Jia, K.: A deep structure for facial expression recognition under partial occlusion. In: IEEE Tenth International Conference on Intelligent Information Hiding and Multimedia Signal Processing, pp. 211–214 (2014)

    Google Scholar 

  3. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124–129 (1971)

    Article  Google Scholar 

  4. García-Rojas, A., et al.: Emotional face expression profiles supported by virtual human ontology: research articles. Comput. Animation Virtual Worlds 17(3–4), 259–269 (2006)

    Article  Google Scholar 

  5. Goeleven, E., De-Raedt, R., Leyman, L., Verschuere, B.: The Karolinska directed emotional faces: a validation study. Cogn. Emot. 22(6), 1094–1118 (2008)

    Article  Google Scholar 

  6. Hammal, Z., Arguin, M.: Comparing a novel model based on the transferable belief model with humans during the recognition of partially occluded facial expressions. J. Vis. 9(2), 22–28 (2009)

    Article  Google Scholar 

  7. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–12 (2015)

    Google Scholar 

  8. Kanopoulos, N., Vasanthavada, N., Baker, R.L.: Design of an image edge detection filter using the Sobel operator. IEEE J. Solid State Circuits 23(2), 358–367 (1988)

    Article  Google Scholar 

  9. Kotsia, I., Zafeiriou, S., Pitas, I.: Texture and shape information fusion for facial expression and facial action unit recognition. Pattern Recogn. 41(3), 833–851 (2008)

    Article  Google Scholar 

  10. Liu, P., Han, S., Meng, Z., Tong, Y.: Facial expression recognition via a boosted deep belief network. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2014)

    Google Scholar 

  11. Liu, S., Yang, X., Wang, Z., Xiao, Z., Zhang, J.: Real-time facial expression transfer with single video camera. Comput. Animation Virtual Worlds 27(3–4), 301–310 (2016)

    Article  Google Scholar 

  12. Liu, W., Wen, Y., Yu, Z., Yang, M.: Large-margin softmax loss for convolutional neural networks. In: International Conference on Machine Learning, pp. 507–516 (2016)

    Google Scholar 

  13. Lyons, M.J., Akamatsu, S., Kamachi, M.: Coding facial expressions with Gabor wavelets. In: IEEE International Conference on Automatic Face and Gesture Recognition, pp. 200–205 (1998)

    Google Scholar 

  14. Pantic, M., Rothkrantz, L.J.M.: Automatic analysis of facial expressions the state of the art. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1424–1445 (2000)

    Article  Google Scholar 

  15. Qiao, F., Yao, N., Jiao, Z., Li, Z.: Emotional facial expression transfer from a single image via generative adversarial nets. Comput. Animation Virtual Worlds 29(6), e1819 (2018)

    Article  Google Scholar 

  16. Shi, J., Ray, N., Zhang, H.: Shape based local thresholding for binarization of document images. Pattern Recogn. Lett. 33(1), 24–32 (2012)

    Article  Google Scholar 

  17. Simard, P., Steinkraus, D., Platt, J.C.: Best practices for convolutional neural networks applied to visual document analysis. In: International Conference on Document Analysis and Recognition, pp. 958–963 (2003)

    Google Scholar 

  18. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference of Learning Representation, pp. 1409–1417 (2014)

    Google Scholar 

  19. Towner, H., Slater, M.: Reconstruction and recognition of occluded facial expressions using PCA. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 36–47. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74889-2_4

    Chapter  Google Scholar 

  20. Wallace, C.: A note on Darwins work on the expression of the emotions in man and animals. J. Abnorm. Psychol. Soc. Psychol. 16(5), 356–366 (1921)

    Google Scholar 

  21. Wang, H.: Cosface large margin cosine loss for deep face recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1801–1807 (2018)

    Google Scholar 

  22. Wen, Y., Zhang, K., Li, Z., Qiao, Yu.: A discriminative feature learning approach for deep face recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 499–515. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_31

    Chapter  Google Scholar 

  23. Yang, L., Wang, S., Zhao, W., Zhao, Y.: Wgan-based robust occluded facial expression recognition. IEEE Access 7, 93594–93610 (2019)

    Article  Google Scholar 

  24. Zhang, L., Brijesh, V., Dian, T., Vinod, C.: Facial expression analysis under partial occlusion: a survey. ACM Comput. Surv. 51(2), 1–49 (2018)

    Article  Google Scholar 

  25. Zhuo, J., Chen, Z., Lai, J., Wang, G.: Occluded person re-identification. In: International Conference on Multimedia and Expo, pp. 1–6 (2018)

    Google Scholar 

  26. Tősér, Z., Jeni, L.A., Lőrincz, A., Cohn, J.F.: Deep learning for facial action unit detection under large head poses. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9915, pp. 359–371. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-49409-8_29

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. College of Intelligence and Computing, Tianjin University, Tianjin, 300350, People’s Republic of China

    Yujian Chen & Shiguang Liu

Authors
  1. Yujian Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shiguang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shiguang Liu .

Editor information

Editors and Affiliations

  1. University of Nevada Reno, Reno, NV, USA

    George Bebis

  2. Stony Brook University, Stony Brook, NY, USA

    Zhaozheng Yin

  3. Drexel University, Philadelphia, PA, USA

    Edward Kim

  4. RWTH Aachen University, Aachen, Germany

    Jan Bender

  5. University of Edinburgh, Edinburgh, UK

    Kartic Subr

  6. IBM Research – Cambridge, Cambridge, MA, USA

    Bum Chul Kwon

  7. University of Waterloo, Waterloo, ON, Canada

    Jian Zhao

  8. Graz University of Technology, Graz, Austria

    Denis Kalkofen

  9. The Hong Kong Polytechnic University, Hong Kong, Hong Kong

    George Baciu

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, Y., Liu, S. (2020). Deep Partial Occlusion Facial Expression Recognition via Improved CNN. In: Bebis, G., et al. Advances in Visual Computing. ISVC 2020. Lecture Notes in Computer Science(), vol 12509. Springer, Cham. https://doi.org/10.1007/978-3-030-64556-4_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-64556-4_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-64555-7

  • Online ISBN: 978-3-030-64556-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation