Skip to main content
Springer Nature Link
Log in

Modeling Stylized Character Expressions via Deep Learning

  • Conference paper
  • First Online:
Computer Vision – ACCV 2016 (ACCV 2016)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 10112))

Included in the following conference series:

Abstract

We propose DeepExpr, a novel expression transfer approach from humans to multiple stylized characters. We first train two Convolutional Neural Networks to recognize the expression of humans and stylized characters independently. Then we utilize a transfer learning technique to learn the mapping from humans to characters to create a shared embedding feature space. This embedding also allows human expression-based image retrieval and character expression-based image retrieval. We use our perceptual model to retrieve character expressions corresponding to humans. We evaluate our method on a set of retrieval tasks on our collected stylized character dataset of expressions. We also show that the ranking order predicted by the proposed features is highly correlated with the ranking order provided by a facial expression expert and Mechanical Turk experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Project page: http://grail.cs.washington.edu/projects/deepexpr/.

References

  1. Lasseter, J.: Principles of traditional animation applied to 3D computer animation. SIGGRAPH Comput. Graph. 21, 35–44 (1987)

    Article  Google Scholar 

  2. Porter, T., Susman, G.: On site: creating lifelike characters in pixar movies. Commun. ACM 43, 25 (2000)

    Article  Google Scholar 

  3. Bates, J.: The role of emotion in believable agents. Commun. ACM 37, 122–125 (1994)

    Article  Google Scholar 

  4. Pelachaud, C., Poggi, I.: Subtleties of facial expressions in embodied agents. J. Vis. Comput. Anim. 13, 301–312 (2002)

    Article  MATH  Google Scholar 

  5. Amini, R., Lisetti, C.: HapFACS: an open source API/Software to generate FACS-based expressions for ECAs animation and for corpus generation. In: 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), pp. 270–275 (2013)

    Google Scholar 

  6. Buhrmester, M., Kwang, T., Gosling, S.D.: Amazon’s Mechanical Turk: a new source of inexpensive, yet high-quality, data? Perspect. Psychol. Sci. 6, 3–5 (2011)

    Article  Google Scholar 

  7. Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1717–1724 (2014)

    Google Scholar 

  8. Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)

    Google Scholar 

  9. Roesch, E.B., Tamarit, L., Reveret, L., Grandjean, D., Sander, D., Scherer, K.R.: FACSGen: a tool to synthesize emotional facial expressions through systematic manipulation of facial action units. J. Nonverbal Behav. 35, 1–16 (2011)

    Article  Google Scholar 

  10. Sandbach, G., Zafeiriou, S., Pantic, M., Yin, L.: Static and dynamic 3D facial expression recognition: a comprehensive survey. Image Vis. Comput. 30, 683–697 (2012)

    Article  Google Scholar 

  11. Adolphs, R.: Recognizing emotion from facial expressions: psychological and neurological mechanisms. Behav. Cogn. Neurosci. Rev. 1(1), 21–62 (2002)

    Article  Google Scholar 

  12. Pereira, F.C., Ebrahimi, T.: The MPEG-4 Book. Prentice Hall PTR, Upper Saddle River (2002)

    Google Scholar 

  13. Deng, Z., Ma, X.: Perceptually guided expressive facial animation. In: Proceedings of the 2008 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, SCA 2008 Eurographics Association (2008)

    Google Scholar 

  14. Jolliffe, I.: Principal Component Analysis. Wiley, Hoboken (2002)

    MATH  Google Scholar 

  15. Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20, 273–297 (1995)

    MATH  Google Scholar 

  16. Kobayashi, H., Hara, F.: Facial interaction between animated 3D face robot and human beings. In: 1997 IEEE International Conference on Systems, Man, and Cybernetics, Computational Cybernetics and Simulation, vol. 4, pp. 3732–3737. IEEE (1997)

    Google Scholar 

  17. Dibeklioglu, H., Salah, A., Gevers, T.: Like father, like son: facial expression dynamics for kinship verification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1497–1504 (2013)

    Google Scholar 

  18. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27, 803–816 (2009)

    Article  Google Scholar 

  19. Liu, C., Wechsler, H.: Independent component analysis of Gabor features for face recognition. IEEE Trans. Neural Netw. 14, 919–928 (2003)

    Article  Google Scholar 

  20. Whitehill, J., Littlewort, G., Fasel, I., Bartlett, M., Movellan, J.: Toward practical smile detection. IEEE Trans. Pattern Anal. Mach. Intell. 31, 2106–2111 (2009)

    Article  Google Scholar 

  21. Shu, C., Ding, X., Fang, C.: Histogram of the oriented gradient for face recognition. Tsinghua Sci. Technol. 16, 216–224 (2011)

    Article  Google Scholar 

  22. Kenji, M.: Recognition of facial expression from optical flow. IEICE Trans. Inf. Syst. 74, 3474–3483 (1991)

    Google Scholar 

  23. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29, 915–928 (2007)

    Article  Google Scholar 

  24. Mahoor, M.H., Zhou, M., Veon, K.L., Mavadati, S.M., Cohn, J.F.: Facial action unit recognition with sparse representation. In: 2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), pp. 336–342. IEEE (2011)

    Google Scholar 

  25. Lin, Y., Song, M., Quynh, D.T.P., He, Y., Chen, C.: Sparse coding for flexible, robust 3D facial-expression synthesis. IEEE Comput. Graph. Appl. 32, 76–88 (2012)

    Article  Google Scholar 

  26. Jeni, L.A., Lőrincz, A., Szabó, Z., Cohn, J.F., Kanade, T.: Spatio-temporal event classification using time-series kernel based structured sparsity. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 135–150. Springer, Heidelberg (2014). doi:10.1007/978-3-319-10593-2_10

    Google Scholar 

  27. Tan, X., Triggs, B.: Fusing Gabor and LBP feature sets for kernel-based face recognition. In: Zhou, S.K., Zhao, W., Tang, X., Gong, S. (eds.) AMFG 2007. LNCS, vol. 4778, pp. 235–249. Springer, Heidelberg (2007). doi:10.1007/978-3-540-75690-3_18

    Chapter  Google Scholar 

  28. Ying, Z.-L., Wang, Z.-W., Huang, M.-W.: Facial expression recognition based on fusion of sparse representation. In: Huang, D.-S., Zhang, X., Reyes García, C.A., Zhang, L. (eds.) ICIC 2010. LNCS (LNAI), vol. 6216, pp. 457–464. Springer, Heidelberg (2010). doi:10.1007/978-3-642-14932-0_57

    Google Scholar 

  29. Mollahosseini, A., Chan, D., Mahoor, M.H.: Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE (2016)

    Google Scholar 

  30. Yu, Z., Zhang, C.: Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 435–442. ACM (2015)

    Google Scholar 

  31. Yu, X., Yang, J., Luo, L., Li, W., Brandt, J., Metaxas, D.: Customized expression recognition for performance-driven cutout character animation. In: Winter Conference on Computer Vision (2016)

    Google Scholar 

  32. Liu, M., Li, S., Shan, S., Chen, X.: Au-aware deep networks for facial expression recognition. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–6. IEEE (2013)

    Google Scholar 

  33. Jung, H., Lee, S., Yim, J., Park, S., Kim, J.: Joint fine-tuning in deep neural networks for facial expression recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2983–2991 (2015)

    Google Scholar 

  34. Zhong, L., Liu, Q., Yang, P., Liu, B., Huang, J., Metaxas, D.N.: Learning active facial patches for expression analysis. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2562–2569. IEEE (2012)

    Google Scholar 

  35. Dumas, M.: Emotional expression recognition using support vector machines. In: Proceedings of International Conference on Multimodal Interfaces. Citeseer (2001)

    Google Scholar 

  36. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 94–101. IEEE (2010)

    Google Scholar 

  37. Mavadati, S.M., Mahoor, M.H., Bartlett, K., Trinh, P., Cohn, J.F.: Disfa: a spontaneous facial action intensity database. IEEE Trans. Affect. Comput. 4, 151–160 (2013)

    Article  Google Scholar 

  38. Lundqvist, D., Flykt, A., Öhman, A.: The Karolinska directed emotional faces-KDEF. CD-ROM from department of clinical neuroscience, psychology section, Karolinska Institutet, Stockholm, Sweden. Technical report (1998). ISBN 91-630-7164-9

    Google Scholar 

  39. Pantic, M., Valstar, M., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: 2005 IEEE International Conference on Multimedia and Expo, ICME 2005, p. 5. IEEE (2005)

    Google Scholar 

  40. Xiong, X., Torre, F.: Supervised descent method and its applications to face alignment. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 532–539 (2013)

    Google Scholar 

  41. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

    Google Scholar 

  42. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the ACM International Conference on Multimedia, pp. 675–678. ACM (2014)

    Google Scholar 

  43. Lin, J.: Divergence measures based on the Shannon entropy. IEEE Trans. Inf. Theor. 37, 145–151 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  44. Kullback, S., Leibler, R.A.: On information and sufficiency. Ann. Math. Stat. 22, 79–86 (1951)

    Article  MathSciNet  MATH  Google Scholar 

  45. Müller, H., Marchand-Maillet, S., Pun, T.: The truth about corel - evaluation in image retrieval. In: Lew, M.S., Sebe, N., Eakins, J.P. (eds.) CIVR 2002. LNCS, vol. 2383, pp. 38–49. Springer, Heidelberg (2002). doi:10.1007/3-540-45479-9_5

    Chapter  Google Scholar 

  46. Spearman, C.: The proof and measurement of association between two things. Am. J. Psychol. 15, 72–101 (1904)

    Article  Google Scholar 

  47. Kendall, M.G.: A new measure of rank correlation. Biometrika 30, 81–93 (1938)

    Article  MATH  Google Scholar 

  48. Character Animator: Adobe After Effects CC 2016. Adobe Systems Incorporated, San Jose, CA 95110–2704 (2016)

    Google Scholar 

Download references

Acknowledgements

We would like to thank Jamie Austad for creating our stylized character database. We would also like to thank the creators of the rigs we used in our project: Mery (www.meryproject.com), Ray (CGTarian Online School), Malcolm (www.animSchool.com), Aia & Jules (www.animationmentor.com), and Bonnie (Josh Sobel Rigs).

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, University of Washington, Seattle, Washington, USA

    Deepali Aneja, Linda Shapiro & Barbara Mones

  2. Zillow Group, Seattle, Washington, USA

    Alex Colburn

  3. Gage Academy of Art, Seattle, Washington, USA

    Gary Faigin

Authors
  1. Deepali Aneja
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alex Colburn
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gary Faigin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Linda Shapiro
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Barbara Mones
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deepali Aneja .

Editor information

Editors and Affiliations

  1. National Tsing Hua University, Hsinchu, Taiwan

    Shang-Hong Lai

  2. Graz University of Technology, Graz, Austria

    Vincent Lepetit

  3. Drexel University, Philadelphia, Pennsylvania, USA

    Ko Nishino

  4. The University of Tokyo, Tokyo, Japan

    Yoichi Sato

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 396 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Aneja, D., Colburn, A., Faigin, G., Shapiro, L., Mones, B. (2017). Modeling Stylized Character Expressions via Deep Learning. In: Lai, SH., Lepetit, V., Nishino, K., Sato, Y. (eds) Computer Vision – ACCV 2016. ACCV 2016. Lecture Notes in Computer Science(), vol 10112. Springer, Cham. https://doi.org/10.1007/978-3-319-54184-6_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-54184-6_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-54183-9

  • Online ISBN: 978-3-319-54184-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics