Skip to main content
Log in

Synthesis of expressive facial animations: A multimodal caricatural mirror

  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

This paper describes a natural and intuitive way to create expressive facial animations, using a novel approach based on the so-called ‘multimodal caricatural mirror’ (MCM). Taking as an input an audio-visual video sequence of the user’s face, the MCM generates a facial animation, in which the prosody and the facial expressions of emotions can either be reproduced or amplified. The user can thus simulate an emotion and see almost instantly the animation it produced, like with a regular mirror. In addition, the MCM also enables to amplify the emotions of selected parts of the input video sequence, leaving other parts unchanged. It therefore constitutes a novel approach to the design of very expressive facial animation, as the affective content of the animation can be modified by post-processing operations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

8. References

  1. M. S. Bartlett, G. Littlewort, I. Fasel, and J. R. Movellan, “Real Time Face Detection and Facial Expression Recognition: Development and Applications to Human Computer Interaction”, inProceedings of Conference on Computer Vision and Pattern Recognition Workshop, vol. 5, (Madison, Wisconsin), pp. 53–58, 16–22 June 2003. 21

    Article  Google Scholar 

  2. P. S. Aleksic and A. K. Katsaggelos, “Speech-To-Video Synthesis Using MPEG-4 Compliant Visual Features”,IEEE Transactions on Circuits and Systems for Video Technology, vol. 14, pp. 682–692, May 2004. 21

    Article  Google Scholar 

  3. Z. Deng, U. Neumann, J. Lewis, T.-Y. Kim, M. Bulut, and S. Narayanan, “Expressive Facial Animation Synthesis by Learning Speech Coarticulation and Expression Spaces”,IEEE Transactions on Visualization and Computer Graphics, vol. 12, pp. 1523–1534, November/December 2006. 21, 22

    Article  Google Scholar 

  4. M. Pantic and L. Rothkrantz, “Automatic Analysis of Facial Expressions: The State of the Art.”,IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, pp. 1424–1445, December 2000. 21

    Article  Google Scholar 

  5. B. Fasel and J. Luettin, “Automatic Facial Expression Analysis: A Survey.”,Pattern Recognition, vol. 1, no. 30, pp. 259–275, 2003. 21

    Article  Google Scholar 

  6. I. Cohen, N. Sebe, S. Garg, L. S. Chen, and T. S. Huanga, “Facial expression recognition from video sequences: temporal and static modelling”,Computer Vision and Image Understanding, vol. 91, pp. 160–187, 2003. 21

    Article  Google Scholar 

  7. I. Kotsia and I. Pitas, “Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines”,IEEE Transactions on Image Processing, vol. 16, pp. 172–187, January 2007. 21, 25, 26

    Article  MathSciNet  Google Scholar 

  8. J. Tao, Y. Kang, and A. Li, “Prosody Conversion From Neutral Speech to Emotional Speech”,IEEE Transactions on Audio, Speech and Language Processing, vol. 14, pp. 1145–1154, July 2006. 21

    Article  Google Scholar 

  9. D. H. Kim, S. U. Jungt, K. H. An, H. S. Lee, and M. J. Chung, “Development of a Facial Expression Imitation System”,IEEE Transactions On Visualization And Computer Graphic, vol. 12, pp. 1523–1534, November/December 2006. 21

    Article  Google Scholar 

  10. P. Ekman and W. V. Friesen,Manual for the Facial Action Coding System. Consulting Psychologists Press, 1977. 22

  11. D. Comaniciu, V. Ramesh, and P.Meer, “Real-Time Tracking of Non-Rigid Objects using Mean-Shift”, inin Proceedinds of IEEE Conference on Computer Vision and Pattern Recognition, (Hilton Head Island, South Carolina), 2000. 23

  12. J. Allen, R. Xu, and J. Jin, “Object Tracking Using CamShift Algorithm and Multiple Quantized Feature Spaces”, inin Proceedinds of the Pan-Sydney Area Workshop on Visual Information Processing (VIP2003), (Sydney, Australia), 2003. 23

  13. K. Sobottka and I. Pitas, “Segmentation and Tracking of Faces in Color Images”, inin Proceedings of the 2nd International Conference on Automatic Face and Gesture Recognition, (Killington, Vermont, USA), 14–16 October 1996. 24

  14. J. Ahlberg, “CANDIDE-3 — an updated parameterized face”, Tech. Rep. Technical report no. LiTH-ISY-R-2326, Dept. of Electrical Engineering, Linkoping University, 2001. 24, 28

  15. T. Kanade, J. F. Cohn, and Y. L. Tian, “Comprehensive database for facial expression analysis”, inin Proceedinds of the 4th IEEE International Conference on Automatic Face and Gesture Recognition (FG’00), 2000. 24, 26

  16. J. Y. Bouguet, “Pyramidal implementation of the Lucas-Kanade feature tracker”, tech. rep., Intel Corporation, Microprocessor Research Labs, 1999. 24, 25

  17. P. Ekman and W. V. Friesen,Emotion in the Human Face. New Jersey: Prentice Hall, 1975. 25

    Google Scholar 

  18. P. Oudeyer, “The production and recognition of emotions in speech: features and algorithms”,International Journal on Human-Computer Studies, vol. 59, pp. 157–183, 2003. 26, 27

    Article  Google Scholar 

  19. P. Boersma, “Accurate short-term analysis of the fundamental frequency and the harmonics-to-noise ratio of a sampled sound”, inin Proceedings of the the Institute of Phonetic Sciences, pp. 97–110, 1995. 26

Download references

Author information

Authors and Affiliations

Authors

Rights and permissions

Reprints and permissions

About this article

Cite this article

Martin, O., Kotsia, I., Pitas, I. et al. Synthesis of expressive facial animations: A multimodal caricatural mirror. J Multimodal User Interfaces 1, 21–30 (2007). https://doi.org/10.1007/BF02884429

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02884429

Keywords

Navigation