Skip to main content
SpringerLink
Log in

Modeling and Animating Realistic Faces from Images

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

We present a new set of techniques for modeling and animating realistic faces from photographs and videos. Given a set of face photographs taken simultaneously, our modeling technique allows the interactive recovery of a textured 3D face model. By repeating this process for several facial expressions, we acquire a set of face models that can be linearly combined to express a wide range of expressions. Given a video sequence, this linear face model can be used to estimate the face position, orientation, and facial expression at each frame. We illustrate these techniques on several datasets and demonstrate robust estimations of detailed face geometry and motion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • Akimoto, T., Suenaga, Y., and Wallace, R. 1993. Automatic creation of 3D facial models. IEEE Computer Graphics and Applications, 13(5):16–22.

    Google Scholar 

  • Ali. 1995. Alias V7.0.

  • Anjyo, K., Usami, Y., and Kurihara, T. 1992. A simple method for extracting the natural beauty of hair. In SIGGRAPH 92 Conference Proceedings, ACM SIGGRAPH, pp. 111–120.

  • Ballard, D. and Brown, C. 1982. Computer Vision. Prentice-Hall: Englewood Cliffs, NJ.

    Google Scholar 

  • Black, M. and Yacoob, Y. 1995. Tracking and recognizing rigid and non-rigid facial motions using local parametric models of image otions. In Proceedings, International Conference on Computer Vision, pp. 374–381.

  • Blake, A. and Isard, M. 1998. Active Contrours: The Application of Techniques from Graphics, Vision, Control Theory and Statistics to Visual Tracking of Shapes in Motion. Addison Wesley: Reading, MA.

    Google Scholar 

  • Blanz, T. and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In SIGGRAPH 99 Conference Proceedings, ACM SIGGRAPH.

  • Bregler, C., Covell, M., and Slaney, M. 1997. Video rewrite: Driving visual speech with audio. In SIGGRAPH 97 Conference Proceedings, ACM SIGGRAPH, pp. 353–360.

  • Bright Start Technologies Inc. 1993. In Beginning Reading Software. Sierra On-Line, Inc.

  • Cascia, M.L., Isidoro, J., and Sclaroff, S. 1998. Head tracking via robust registration in texture map images. In Proceedings, IEEE Conference on Computer Vision and Pattern Recognition.

  • Chen, D., State, A., and Banks, D. 1995. Interactive shape metamorphosis. In 1995 Symposium on Interactive 3D Graphics, ACM SIGGRAPH, pp. 43–44.

  • Choi, C., Aizawa, K., Harshima, H., and Takebe, T. 1994. Analysis and synthesis of facial image sequences in model-based image coding. IEEE Transactions on Circuits and Systems for Video Technology, 4(3):257–274.

    Google Scholar 

  • Choi, C.S., Kiyoharu, Harashima, H., and Takebe, T. 1994. Analysis and synthesis of facial image sequences in model-based image coding. IEEE Transactions on Circuits and Systems for Video Technology, 4:257–275.

    Google Scholar 

  • Covell, M. 1996. Eigen-points: Control-point location using principal component analysis. In Proceedings, Second International Conference on Automatic Face and Gesture Recognition, pp. 122–127.

  • Cyb. 1990. 4020/RGB 3D Scanner with Color Digitizer.

  • Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. In SIGGRAPH 2000 Conference Proceedings, ACM SIGGRAPH, pp. 35–42.

  • Debevec, P., Taylor, C., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In SIGGRAPH 96 Conference Proceedings, ACM SIGGRAPH, pp. 11–20.

  • Decarlo, D. and Metaxas, D. 1998. Deformable model-based shape and motion analysis from images using motion residual error. In Proceedings, First International Conference on Computer Vision, pp. 113–119.

  • Devernay, F. and Faugeras, O. 1994. Computing differential properties of 3d shapes from stereoscopic images without 3d models. In Proceedings, IEEE Conference on Computer Vision and Pattern Recognition, pp. 208–213.

  • Edwards, G., Taylor, C., and Cootes, T. 1998. Interpreting face images using active appearance models. In Proceedings, Third Workshop on Face and Gesture Recognition, pp. 300–305.

  • Essa, I. and Pentland, A. 1997. Coding, analysis, interpretation, and recognition of facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7):757–763.

    Google Scholar 

  • Faugeras, O. 1993. Three-Dimensional Computer Vision: A Geometric Viewpoint. MIT Press: Cambridge, MA.

    Google Scholar 

  • Fua, P. and Miccio, C. 1998. From regular images to animated heads: Aleast square approach. In Proceedings, European Conference on Computer Vision, pp. 188–202.

  • Golub, G. and Van Loan, C. 1996. Matrix Computation, 3rd edn. The John Hopkins University Press: Baltimore/London.

    Google Scholar 

  • Gortler, S., Grzeszczuk, R., Szeliski, R., and Cohen, M. 1996. The Lumigraph. In SIGGRAPH 96 Conference Proceedings, ACM SIGGRAPH, pp. 43–54.

  • Guenter, B., Grimm, C., Wood, D., Malvar, H., and Pighin, F. 1998. Making faces. In SIGGRAPH 98 Conference Proceedings, ACM SIGGRAPH, pp. 55–66.

  • Hager, G. and Belhumeur, P. 1996. Real-time tracking of image regions with changes in geometry and illumination. In Proceedings, Computer Vision and Pattern Recognition, pp. 403–410.

  • Hallinan, P. 1994. A low-dimensional representation of human faces for arbitrary lighting conditions. In Proceedings, IEEE Conference on Computer Vision and Pattern Recognition, pp. 58–66.

  • Hanrahan, P. and Kruger, W. 1993. Reflection from layered surfaces due to surface scattering. In SIGGRAPH 93 Conference Proceedings, pp. 165–174.

  • Ip, H. and Yin, L. 1996. Constructing a 3D individualized head model from two orthogonal views. The Visual Computer, 12:254–266.

    Google Scholar 

  • Jones, M. and Poggio, T. 1998. Hierarchical morphable models. In Proceedings, International Conference on Computer Vision, pp. 820–826.

  • Kass, M., Witkin, A., and Terzopoulos, D. 1987. Snakes: Active contour models. In Proceedings, First International Conference on Computer Vision, pp. 259–268.

  • Koch, R., Gross, M., Carls, F.R., von Büren, D., Fankhauser, G., and Parish, Y. 1996. Simulating facial surgery using finite element methods. In SIGGRAPH 96 Conference Proceedings, ACM SIGGRAPH, pp. 421–428.

  • Kurihara, T. and Arai, K. 1991. A transformation method for modeling and animation of the human face from photographs. In Computer Animation, Vol. 91, N.M. Thalmann and D. Thalmann (Eds.), Springer-Verlag, Tokyo, pp. 45–58.

    Google Scholar 

  • Lanitis, A., Taylor, C., and Cootes, T. 1995. A unified approach for coding and interpreting face images. In Fifth International Conference on Computer Vision (ICCV 95), Cambridge, Massachusetts, pp. 368–373.

  • Lawson, C. and Hansen, R. 1974. Solving Least Squares Problems. Prentice-Hall: Englewood Cliffs.

    Google Scholar 

  • Leclerc, Y. and Bobick, A. 1991. The direct computation of height from shading. In Proceedings, IEEE Conference on Computer Vision and Pattern Recognitio n.

  • Lee, S., Chwa, K., Shin, S., and Wolberg, G. 1995. Image metamorphosis using snakes and free-form deformations. In SIGGRAPH 95 Conference Proceedings, ACM SIGGRAPH, pp. 439–448.

  • Lee, S., Wolberg, G., Chwa, K., and Shin, S. 1996. Image metamorphosis with scattered feature constraints. In IEEE Transactions on Visualization and Computer Graphics, 2(4):337–354.

    Google Scholar 

  • Lee, Y., Terzopoulos, D., and Waters, K. 1995. Realistic modeling for facial animation. In SIGGRAPH 95 Conference Proceedings, ACM SIGGRAPH, pp. 55–62.

  • Lengagne, R., Fua, P., and Monga, O. 1998. 3d face modeling using stereo and differential constraints. In Proceedings of the Third International Conference on Automatic Face and Gesture Recognition.

  • Li, H., Roivainen, P., and Forchheimer, R. 1993. 3d motion estimation in model-based facial image coding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(6):545–555.

    Google Scholar 

  • Matsino, K., Lee, C., Kimura S., and Tsuji, S. 1995. Automatic recognition of human facial expressions. In Proceedings of the IEEE, pp. 352–359.

  • Moffitt, F. and Mikhail, E. 1980. Photogrammetry, 3rd edn. Harper & Row, New York.

    Google Scholar 

  • Nielson, G. 1993. Scattered data modeling. IEEE Computer Graphics and Applications, 13(1):60–70.

    Google Scholar 

  • Ostby, E. Pixar animation studios: 1997, Personal communication.

  • Parke, F. 1972. Computer generated animation of faces. In Proceedings ACM annual conference.

  • Parke, F. 1974. A Parametric Model for Human Faces. PhD Thesis, University of Utah, Salt Lake City, Utah. UTEC-CSc-75-047.

    Google Scholar 

  • Pighin, F., Hecker, J., Lischinski, D., Szeliski, R., and Salesin, D. 1998. Synthesizing realistic facial expressions from photographs. In SIGGRAPH 98 Conference Proceedings, ACM SIGGRAPH, pp. 75–84.

  • Pighin, F., Szeliski, R., and Salesin, D. 1999. Resynthesizing facial animation through 3d model-based tracking. In Proceedings, International Conference on Computer Vision.

  • Press, W., Flannery, B., Teukolsky, S., and Vetterling, W. 1992. Numerical Recipes in C: The Art of Scientific Computing. 2nd. edn., Cambridge University Press: Cambridge, England.

    Google Scholar 

  • Proesman, M., Gool, L.V., and Oosterlinck, A. 1996. Active acuisition of 3d shape for moving objects. International Conference on Image Processing.

  • Pulli, K., Cohen, M., Duchamp, T., Hoppe, H., Shapiro, L., and Stuetzle, W. 1997. View-based rendering: Visualizing real objects from scanned range and color data. In Proc. 8th Eurographics Workshop on Rendering.

  • Rosenblum, R., Carlson, W., and 1991. Physically-based facial modeling, analysis, and animation. Journal of Visualization and Computer Animation, 2(4):141–148.

    Google Scholar 

  • Schodl, A., Ario, A., and Essa, I. 1998. Head tracking using a textured polygonal model. In Workshop on Perceptual User Interfaces, pp. 43–48.

  • Slama, C. (Ed.). 1980. In Manual of Photogrammetry. 4th edn., American Society of Photogrammetry, Falls Church, Virginia.

    Google Scholar 

  • Szeliski, R. and Kang, S. 1994. Recovering 3D shape and motion from image streams using nonlinear least squares. Journal of Visual Communication and Image Representation, 5(1):10–28.

    Google Scholar 

  • Szeliski, R. and Shum, H. 1997. Creating full view panoramic image mosaics and texture-mapped models. In SIGGRAPH 97 Conference Proceedings, ACM SIGGRAPH, pp. 251–258.

  • Terzopoulos, D. and Waters, K. 1993. Analysis and synthesis of facial image sequences using physical an anatomical models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(6):569–579.

    Google Scholar 

  • Thomas, F., Johnston, O., and Johnston, C. 1995. The Illusion of Life. Hyperion.

  • Thórisson, K. 1997. Gandalf: An embodied humanoid capable of real-time multimodal dialogue with people. In First ACM International Conference on Autonomous Agents.

  • Turk, M. and Pentland, A. 1987. Eigenfaces for recognition. Journal of Cognitive Neuroscience, 3(1):71–87.

    Google Scholar 

  • Vannier, M., Marsh, J., and Warren, J.O. 1983. Three-dimentional computer graphics for craniofacial surgical planning and evaluation. In SIGGRAPH 83 Conference Proceedings, Vol. 17, ACM SIGGRAPH, pp. 263–273.

    Google Scholar 

  • Vetter, T. and Blanz, V. 1998. Estimating coloured 3d face models from single images: An example based approach. In Proceedings, European Conference on Computer Vision, pp. 499–513.

  • Watanabe, Y. and Suenaga, Y. 1992. A trigonal prism-based method for hair image generation. IEEE Computer Graphics and Applications, 12(1):47–53.

    Google Scholar 

  • Williams, L. 1990. Performance-driven facial animation. In SIGGRAPH 90 Conference Proceedings, Vol. 24, pp. 235–242.

    Google Scholar 

  • Yu, Y. and Malik, J. 1998. Recovering photometric properties of architectural scenes from photographs. In SIGGRAPH 98 Conference Proceeding s, ACM SIGGRAPH, pp. 207–217.

Download references

Author information

Authors and Affiliations

  1. Institute for Creative Technologies, University of Southern California, USA

    Frédéric Pighin

  2. Microsoft Research, USA

    Richard Szeliski

  3. Department of Computer Science, University of Washington; Microsoft Research, USA

    David H. Salesin

Authors
  1. Frédéric Pighin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Richard Szeliski
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David H. Salesin
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Pighin, F., Szeliski, R. & Salesin, D.H. Modeling and Animating Realistic Faces from Images. International Journal of Computer Vision 50, 143–169 (2002). https://doi.org/10.1023/A:1020393915769

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1020393915769

Search

Navigation