Abstract
Analysis of facial expressions is a task of increasing interest in Computer Vision, with many potential applications. However, collecting images with labeled expression for many subjects is a quite complicated operation. In this paper, we propose a solution that use a particular 3D morphable model (3DMM) that, starting from a neutral image of a target subject, is capable of producing a realistic expressive face image of the same subject. This is possible thanks to the fact the used 3DMM can effectively and efficiently fit to 2D images, and then deform itself under the action of deformation parameters that are learned expression-by-expression in a subject-independent manner. Ultimately, the application of such deformation parameters to the neutral model of a subject allows the rendering of realistic expressive images of the subject. In the experiments, we demonstrate that such deformation parameters can be learned even from a small set of training data using simple statistical tools; despite this simplicity, we show that very realistic subject-dependent expression renderings can be obtained with our method. Furthermore, robustness to cross dataset tests is also evidenced.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Amberg, B., Romdhani, S., Vetter, T.: Optimal step nonrigid ICP algorithms for surface registration. In: IEEE International Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, pp. 1–8, June 2007
Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: ACM Conference on Computer Graphics and Interactive Techniques (1999)
Blanz, V., Vetter, T.: Face recognition based on fitting a 3D morphable model. IEEE Trans. Pattern Anal. Mach. Intell. 25(9), 1063–1074 (2003)
Booth, J., Antonakos, E., Ploumpis, S., Trigeorgis, G., Panagakis, Y., Zafeiriou, S.: 3D face morphable models “in-the-wild”. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5464–5473, July 2017. https://doi.org/10.1109/CVPR.2017.580
Booth, J., Roussos, A., Zafeiriou, S., Ponniahand, A., Dunaway, D.: A 3D morphable model learnt from 10,000 faces. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 5543–5552 (2016)
Cosker, D., Krumhuber, E., Hilton, A.: Perception of linear and nonlinear motion properties using a FACS validated 3D facial model. In: ACM Applied Perception in Graphics and Vision (2010)
Cosker, D., Krumhuber, E., Hilton, A.: A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling. In: International Conference on Computer Vision (2011)
Cosker, D., Krumhuber, E., Hilton, A.: Perceived emotionality of linear and non-linear AUs synthesised using a 3D dynamic morphable facial model. In: Proceedings of the Facial Analysis and Animation, FAA 2015, p. 7:1. ACM (2015)
Dou, P., Shah, S.K., Kakadiaris, I.A.: End-to-end 3D face reconstruction with deep neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1503–1512, July 2017. https://doi.org/10.1109/CVPR.2017.164
Ekman, P.: Facial expression and emotion. Am. Anthropol. 48(4), 384–392 (1992)
Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, CA (1978)
Ferrari, C., Lisanti, G., Berretti, S., Del Bimbo, A.: A dictionary learning-based 3D morphable shape model. IEEE Trans. Multimedia 19(12), 2666–2679 (2017). https://doi.org/10.1109/TMM.2017.2707341
Hu, G., et al.: Face recognition using a unified 3D morphable model. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 73–89. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_5
Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E.: Labeled faces in the wild: a database for studying face recognition in unconstrained environments. Technical report 07-49, University of Massachusetts, Amherst, October 2007
Huang, Y., Khan, S.M.: DyadGAN: generating facial expressions in dyadic interactions. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2259–2266, July 2017. https://doi.org/10.1109/CVPRW.2017.280
Huber, P., Kopp, P., Rätsch, M., Christmas, W.J., Kittler, J.: 3D face tracking and texture fusion in the wild. CoRR abs/1605.06764 (2016). http://arxiv.org/abs/1605.06764
Kazemi, V., Sullivan, J.: One millisecond face alignment with an ensemble of regression trees. In: IEEE Conference on Computer Vision and Pattern Recognition (2014)
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: IEEE Conference on Computer Vision and Pattern Recognition-Workshops (2010)
Mairal, J., Bach, F., Ponce, J., Sapiro, G.: Online dictionary learning for sparse coding. In: International Conference on Machine Learning (2009)
Masi, I., Ferrari, C., Del Bimbo, A., Medioni, G.: Pose independent face recognition by localizing local binary patterns via deformation components. In: International Conference on Pattern Recognition (2014)
Patel, A., Smith, W.A.P.: 3D morphable face models revisited. In: IEEE Conference on Computer Vision and Pattern Recognition (2009)
Paysan, P., Knothe, R., Amberg, B., Romdhani, S., Vetter, T.: A 3D face model for pose and illumination invariant face recognition. In: IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), pp. 296–301 (2009)
Qiao, F., Yao, N., Jiao, Z., Li, Z., Chen, H., Wang, H.: Geometry-contrastive generative adversarial network for facial expression synthesis. CoRR abs/1802.01822 (2018). http://arxiv.org/abs/1802.01822
Ramanathan, S., Kassim, A., Venkatesh, Y.V., Wah, W.S.: Human facial expression recognition using a 3D morphable model. In: International Conference on Image Processing (2006)
Romdhani, S., Vetter, T.: Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior. In: IEEE Conference on Computer Vision and Pattern Recognition (2005)
Savran, A., et al.: Bosphorus database for 3D face analysis. In: Schouten, B., Juul, N.C., Drygajlo, A., Tistarelli, M. (eds.) BioID 2008. LNCS, vol. 5372, pp. 47–56. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-89991-4_6
Tran, A.T., Hassner, T., Masi, I., Medioni, G.: Regressing robust and discriminative 3D morphable models with a very deep neural network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5163–5172, July 2017
Ujir, H., Spann, M.: Facial expression recognition using FAPs-based 3DMMM. In: Tavares, J., Natal Jorge, R. (eds.) Topics in Medical Image Processing and Computational Vision. LNCVB, vol. 8, pp. 33–47. Springer, Dordrecht (2013). https://doi.org/10.1007/978-94-007-0726-9_2
Yi, D., Lei, Z., Li, S.Z.: Towards pose robust face recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (2013)
Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.: A 3D facial expression database for facial behavior research. In: IEEE International Conference on Automatic Face and Gesture Recognition (2006)
Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S.Z.: Face alignment across large poses: a 3D solution. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Ferrari, C., Berretti, S., Pala, P., Del Bimbo, A. (2018). Learning 3DMM Deformation Coefficients for Rendering Realistic Expression Images. In: Basu, A., Berretti, S. (eds) Smart Multimedia. ICSM 2018. Lecture Notes in Computer Science(), vol 11010. Springer, Cham. https://doi.org/10.1007/978-3-030-04375-9_27
Download citation
DOI: https://doi.org/10.1007/978-3-030-04375-9_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-04374-2
Online ISBN: 978-3-030-04375-9
eBook Packages: Computer ScienceComputer Science (R0)