Skip to main content
SpringerLink
Log in
Book cover

Asian Conference on Computer Vision

ACCV 2018: Computer Vision – ACCV 2018 pp 19–34Cite as

Zero-Shot Facial Expression Recognition with Multi-label Label Propagation

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11363))

Abstract

Facial expression recognition classifies a face image into one of several discrete emotional categories. We have a lot of exclusive or non-exclusive emotional classes to describe the varied and nuancing meaning conveyed by facial expression. However, it is almost impossible to enumerate all the emotional categories and collect adequate annotated samples for each category. To this end, we propose a zero-shot learning framework with multi-label label propagation (Z-ML\(^2\)P). Z-ML\(^2\)P is built on existing multi-class datasets annotated with several basic emotions and it can infer the existence of other new emotion labels via a learned semantic space. To evaluate the proposed method, we collect a multi-label FER dataset FaceME. Experimental results on FaceME and two other FER datasets demonstrate that Z-ML\(^2\)P framework improves the state-of-the-art zero-shot learning methods in recognizing both seen or unseen emotions.

This work is done by Zijia Lu during his internship in Institute of Computing Technology, Chinese Academy of Sciences. We gratefully acknowledge the supports from National Key R&D Program of China (grant 2017YFA0700800), National Natural Science Foundation of China (grant 61702481), and External Cooperation Program of CAS (grant GJHZ1843). We also thank Yong Li for his help in adapting the annotation tool to Windows OS.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Akata, Z., Perronnin, F., Harchaoui, Z., Schmid, C.: Label-embedding for image classification. IEEE T-PAMI 38(7), 1425–1438 (2016)

    Article  Google Scholar 

  2. Akata, Z., Reed, S., Walter, D., Lee, H., Schiele, B.: Evaluation of output embeddings for fine-grained image classification. In: Proceedings of CVPR (2015)

    Google Scholar 

  3. Bucak, S.S., Mallapragada, P.K., Jin, R., Jain, A.K.: Efficient multi-label ranking for multi-class learning: application to object recognition. In: Proceedings of ICCV (2009)

    Google Scholar 

  4. Changpinyo, S., Chao, W.L., Gong, B., Sha, F.: Synthesized classifiers for zero-shot learning. In: Proceedings of CVPR (2016)

    Google Scholar 

  5. Cohn, J.F., De la Torre, F.: Automated face analysis for affective computing. In: The Oxford Handbook of Affective Computing (2014)

    Google Scholar 

  6. De la Torre, F., Cohn, J.F.: Facial expression analysis. In: Moeslund, T., Hilton, A., Krüger, V., Sigal, L. (eds.) Visual Analysis of Humans, pp. 377–409. Springer, London (2011). https://doi.org/10.1007/978-0-85729-997-0_19

    Chapter  Google Scholar 

  7. Du, S., Tao, Y., Martinez, A.M.: Compound facial expressions of emotion. Proc. Nat. Acad. Sci. 111(15), E1454–E1462 (2014)

    Article  Google Scholar 

  8. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. 17(2), 124–9 (1971)

    Article  Google Scholar 

  9. Ekman, P., Rosenberg, E.L.: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, Oxford (1997)

    Google Scholar 

  10. Eleftheriadis, S., Rudovic, O., Pantic, M.: Discriminative shared gaussian processes for multiview and view-invariant facial expression recognition. IEEE Trans. Image Process. 24(1), 189–204 (2015)

    Article  MathSciNet  Google Scholar 

  11. Fabian Benitez-Quiroz, C., Srinivasan, R., Martinez, A.M.: Emotionet: an accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In: Proceedings of CVPR (2016)

    Google Scholar 

  12. Faddeev, D.K., Faddeeva, V.N.: Computational methods of linear algebra. J. Sov. Math. 15(5), 531–650 (1981)

    Article  Google Scholar 

  13. Frome, A., Corrado, G., Shlens, J.: Devise: a deep visual-semantic embedding model. In: Proceedings of NIPS (2013)

    Google Scholar 

  14. Fu, Y., Hospedales, T.M., Xiang, T., Gong, S.: Transductive multi-view zero-shot learning. IEEE T-PAMI 37(11), 2332–2345 (2015)

    Article  Google Scholar 

  15. Gaure, A., Gupta, A., Verma, V.K., Rai, P.: A probabilistic framework for zero-shot multi-label learning. In: Proceedings of UAI (2017)

    Google Scholar 

  16. Gibaja, E., Ventura, S.: A tutorial on multilabel learning. ACM Comput. Surv. 47(3), 52:1–52:38 (2015)

    Article  Google Scholar 

  17. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of CVPR (2015)

    Google Scholar 

  18. Kacem, A., Daoudi, M., Amor, B.B., Alvarez-Paiva, J.C.: A novel space-time representation on the positive semidefinite cone for facial expression recognition. In: Proceedings of ICCV (2017)

    Google Scholar 

  19. Kodirov, E., Xiang, T., Gong, S.: Semantic autoencoder for zero-shot learning. In: Proceedings of CVPR (2017)

    Google Scholar 

  20. Kosti, R., Alvarez, J.M., Recasens, A., Lapedriza, A.: Emotion recognition in context. In: Proceedings of CVPR (2017)

    Google Scholar 

  21. Li, A., Lu, Z., Wang, L., Xiang, T., Li, X., Wen, J.R.: Zero-Shot Fine-Grained Classification by Deep Feature Learning with Semantics. CoRR abs/1707.00785, pp. 1–10 (2017)

    Google Scholar 

  22. Li, S., Deng, W., Du, J.: Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: Proceedings of CVPR (2017)

    Google Scholar 

  23. Liu, M., Shan, S., Wang, R., Chen, X.: Learning expressionlets via universal manifold model for dynamic facial expression recognition. IEEE Trans. Image Process. 25(12), 5920–5932 (2016)

    Article  MathSciNet  Google Scholar 

  24. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: Proceedings of CVPR (2010)

    Google Scholar 

  25. McDonald, J.B., Xu, Y.J.: A generalization of the beta distribution with applications. J. Econometrics 66(1), 133–152 (1995)

    Article  Google Scholar 

  26. Mehrabian, A.: Framework for a comprehensive description and measurement of emotional states. Genet. Soc. Gen. Psychol. Monogr. 121(3), 339–361 (1995)

    Google Scholar 

  27. Mikolov, T., Corrado, G., Chen, K., Dean, J.: Efficient estimation of word representations in vector space. In: Proceedings of ICLR (2013)

    Google Scholar 

  28. Mollahosseini, A., Hasani, B., Mahoor, M.H.: Affectnet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. (2017)

    Google Scholar 

  29. Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: Proceedings of BMVC (2015)

    Google Scholar 

  30. Patterson, G., Hays, J.: Sun attribute database: discovering, annotating, and recognizing scene attributes. In: Proceedings of CVPR (2012)

    Google Scholar 

  31. Peng, G., Wang, S.: Weakly supervised facial action unit recognition through adversarial training. In: Proceedings of CVPR (2018)

    Google Scholar 

  32. Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: Proceedings of EMNLP (2014)

    Google Scholar 

  33. Sang, D.V., Dat, N.V., Thuan, D.P.: Facial expression recognition using deep convolutional neural networks. In: Proceedings of KSE (2017)

    Google Scholar 

  34. Soleymani, M., Asghari-Esfeden, S., Fu, Y., Pantic, M.: Analysis of eeg signals and facial expressions for continuous emotion detection. IEEE Trans. Affect. Comput. 7(1), 17–28 (2016)

    Article  Google Scholar 

  35. Wang, P., Liu, L., Shen, C.: Multi-attention network for one shot learning. In: Proceedings of CVPR (2017)

    Google Scholar 

  36. Xian, Y., Akata, Z., Sharma, G., Nguyen, Q., Hein, M., Schiele, B.: Latent embeddings for zero-shot classification. In: Proceedings of CVPR (2016)

    Google Scholar 

  37. Yang, H., Ciftci, U., Yin, L.: Facial expression recognition by de-expression residue learning. In: Proceedings of CVPR (2018)

    Google Scholar 

  38. Zhang, F., Zhang, T., Mao, Q., Xu, C.: Joint pose and expression modeling for facial expression recognition. In: Proceedings of CVPR (2018)

    Google Scholar 

  39. Zhang, Y., Gong, B., Shah, M.: Fast zero-shot image tagging. In: Proceedings of CVPR (2016)

    Google Scholar 

  40. Zhang, Y., Dong, W., Hu, B.G., Ji, Q.: Classifier learning with prior probabilities for facial action unit recognition. In: Proceedings of CVPR (2018)

    Google Scholar 

  41. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE T-PAMI 29(6), 915–928 (2007)

    Article  Google Scholar 

  42. Zhao, K., Chu, W., la Torre, F.D., Cohn, J.F., Zhang, H.: Joint patch and multi-label learning for facial action unit and holistic expression recognition. IEEE Trans. Image Process. 25(8), 3931–3946 (2016)

    Article  MathSciNet  Google Scholar 

  43. Zhao, K., Chu, W.S., Martinez, A.M.: Learning facial action units from web images with scalable weakly supervised clustering. In: Proceedings of CVPR (2018)

    Google Scholar 

  44. Zhu, X., Ghahramani, Z., Lafferty, J.: Semi-supervised learning using Gaussian fields and harmonic functions. In: Proceedings of ICML (2003)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS, Beijing, 100190, China

    Zijia Lu, Jiabei Zeng, Shiguang Shan & Xilin Chen

  2. NYU Shanghai, Shanghai, China

    Zijia Lu

  3. University of Chinese Academy of Sciences, Beijing, 100190, China

    Shiguang Shan & Xilin Chen

  4. CAS Center for Excellence in Brain Science and Intelligence Technology, Shanghai, 200031, China

    Shiguang Shan

Authors
  1. Zijia Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiabei Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shiguang Shan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xilin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiabei Zeng .

Editor information

Editors and Affiliations

  1. IIIT Hyderabad, Hyderabad, India

    C. V. Jawahar

  2. ANU, Canberra, ACT, Australia

    Hongdong Li

  3. Simon Fraser University, Burnaby, BC, Canada

    Greg Mori

  4. ETH Zurich, Zurich, Zürich, Switzerland

    Konrad Schindler

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1167 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lu, Z., Zeng, J., Shan, S., Chen, X. (2019). Zero-Shot Facial Expression Recognition with Multi-label Label Propagation. In: Jawahar, C., Li, H., Mori, G., Schindler, K. (eds) Computer Vision – ACCV 2018. ACCV 2018. Lecture Notes in Computer Science(), vol 11363. Springer, Cham. https://doi.org/10.1007/978-3-030-20893-6_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-20893-6_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-20892-9

  • Online ISBN: 978-3-030-20893-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation