Skip to main content
SpringerLink
Log in

Face Hallucination: Theory and Practice

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

In this paper, we study face hallucination, or synthesizing a high-resolution face image from an input low-resolution image, with the help of a large collection of other high-resolution face images. Our theoretical contribution is a two-step statistical modeling approach that integrates both a global parametric model and a local nonparametric model. At the first step, we derive a global linear model to learn the relationship between the high-resolution face images and their smoothed and down-sampled lower resolution ones. At the second step, we model the residue between an original high-resolution image and the reconstructed high-resolution image after applying the learned linear model by a patch-based non-parametric Markov network to capture the high-frequency content. By integrating both global and local models, we can generate photorealistic face images. A practical contribution is a robust warping algorithm to align the low-resolution face images to obtain good hallucination results. The effectiveness of our approach is demonstrated by extensive experiments generating high-quality hallucinated face images from low-resolution input with no manual alignment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Baker, S. and Kanade, T. 2000a. Hallucinating faces. In IEEE International Conference on Automatic Face and Gesture Recognition.

  • Baker, S. and Kanade, T. 2000b. Limits on super-resolution and how to break them. In Proc. IEEE Conf. Computer Vision and Pattern Recognition.

  • Baker, S. and Matthews, I. 2004. Lucas-kanade 20 years on: A unifying framework. International Journal on Compter Vision, 56(3):221–255.

    Article  Google Scholar 

  • Blake, A., Bascle, B., and Zisserman, A. 1996. Motion deblurring and super-resolution from an image sequence. In Proc. European Conference on Computer Vision, pp. 312–320.

  • De Bonet, J. 1997. Multiresolution sampling procedure for analysis and synthesis of texture images. In Proceedings of SIGGRAPH 97, pp. 361–368.

  • Capel, D. and Zisserman, A. 2001. Super-resolution from multiple views using learnt image models. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 627–634.

  • Chen, H., Xu, Y.Q., Shum, H.Y., Zhu, S.C., and Zheng, N.N. 2001. Example-based facial sketch generation with non-parametric sampling. In Proc. IEEE Int’l Conf. Computer Vision, pp. 433– 438.

  • Cootes, T.F. and Taylor, C.J. 2000. Statistical models of appearance for computer vision. Technical report, University of Manchester.

  • Dedeoglu, G., Baker, S., and Kanade, T. 2006. Resolution-aware fitting of active appearance models to low-resolution images. In Proc. European Conference on Computer Vision. Springer, pp. 83– 97.

  • Dedeoglu, G., Kanade, T., and August, J. 2004. High-zoom video hallucination by exploiting spatio-temporal regularities. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 151–158.

  • Efros, A.A. and Freeman, W.T. 2001. Quilting for texture synthesis and transfer. In Proceedings of SIGGRAPH 2001, pp. 341–346.

  • Fitzgibbon, A.W., Wexler, Y., and Zisserman, A. 2003. Image-based rendering using image-based priors. In Proc. IEEE Int’l Conf. Computer Vision, pp. 1176–1183.

  • Freeman, W.T., Jones, T.R., and Pasztor, E.C. 2002. Example-based super-resolution. IEEE Computer Graphics and Applications, 22(2):56–65.

    Article  Google Scholar 

  • Freeman, W.T., Pasztor, E.C., and Carmichael, O.T. 2000. Learning low-level vision. International Journal on Compter Vision, 40(1):25–47.

    Article  MATH  Google Scholar 

  • Greenspan, H., Anderson, C., and Akber, S. 2000. Image enhancement by nonlinear extrapolation in frequency space. IEEE Trans. on Image Processing, 9(6).

  • Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., and Salesin, D.H. 2001. Image analogies. In Proceedings of SIGGRAPH 2001.

  • Hou, H.H. and Andrews, H.C. 1978. Cubic splines for image interpolation and digital filtering. IEEE Trans. Acoust. Speech Signal Proc., 26(6):508–517.

    Article  MATH  Google Scholar 

  • Huang, T.S. and Tsai, R.Y. 1984. Multi-frame image restoration and registration. Advances in Computer Vision and Image Processing, 1:317–339.

    Google Scholar 

  • Jia, K. and Gong, S. 2005. Multi-modal tensor face for simultaneous super-resolution and recognition. In Proc. IEEE Int’l Conf. Computer Vision, pp. 1683–1690.

  • Jordan, M.I. 1998. (ed.) Learning in Graphical Models. MIT Press.

  • Li, Y. and Lin, X. 2004. Face hallucination with pose variation. In IEEE International Conference on Automatic Face and Gesture Recognition, pp. 723–728.

  • Liang, L., Liu, C., Xu, Y.Q., Guo, B., and Shum, H.Y. 2001. Real-time texture synthesis by patch-based sampling. ACM Trans. Graph., 20(3):127–150.

    Google Scholar 

  • Liu, C., Freeman, W.T., Szeliski, R., and Kang, S.B. 2006. Noise estimation from a single image. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 901–908.

  • Liu, C., Shum, H.Y., and Zhang, C.S. 2001. A two-step approach to hallucinating faces: Global parametric model and local nonparametric model. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 192–198.

  • Liu, C., Zhu, S.C., and Shum, H.Y. 2001. Learning inhomogeneous Gibbs model of faces by minimax entropy. In Proc. IEEE Int’l Conf. Computer Vision, pp. 281–287.

  • Lucas, B.D. and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In the 7th International Joint Conference on Artificial Intelligence (IJCAI ’81), pp. 674–679.

  • Martinez, A. and Benavente, R. 1998. The AR face database. Technical report, CVC Techinical Report No. 24.

  • Martinez, D. 1986. Model-based Motion Estimation and its Application to Restoration and Interpolation of Motion Pictures. PhD thesis, Massachusetts Institute of Technology.

  • Morse, B. and Schwartzwald, D. 2001. Image magnification using level set reconstruction. In Proc. IEEE Int’l Conf. Computer Vision, pp. 333–341.

  • Oppenheim, A.V., Willsky, A.S., and Nawab, S.H. 1997. Signal and Systems 2nd edition. Prentice Hall, Inc.

  • Philips, P., Moon, H., Pauss, P., and Rivzvi. S. 1997. The feret evaluation methodology for face-recognition algorithms. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 137–143.

  • Press, W., Teukolsky, S., Vetterling, W., and Flannery, B. 1992. Numerical Recipes in C 2nd edition. Cambridge University Press.

  • Rowley, H.A., Baluja, S., and Kanade, T. 1998. Neural network-based face detection. IEEE Trans. on Pattern Analysis and Machine Intelligence, 20(1):23–38.

    Article  Google Scholar 

  • Schultz, R.R. and Stevenson, R.L. 1994. A Bayesian approach to image expansion for improved definition. IEEE Trans. Image Processing, 3(3):233–242.

    Article  Google Scholar 

  • Shechtman, E., Caspi, Y., and Irani, M. 2005. Space-time super-resolution. IEEE Trans. on Pattern Analysis and Machine Intelligence, 27(4):531–545.

    Article  Google Scholar 

  • Strang, G. 1988. Linear Algebra and Its Applications 3rd edn. Thomson Learning, Inc.

  • Sun, J., Zheng, N.N., Tao, H., and Shum, H.-Y. 2003. Generic image hallucination with primal sketch prior. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 729– 736.

  • Tappen, M.F., Russell, B.C., and Freeman, W.T. 2004. Efficient graphical models for processing images. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 673–680.

  • Tomasi, C. and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Proc. IEEE Int’l Conf. Computer Vision, pp. 839– 846.

  • Turk, M. and Pentland, A. 1991. Eigenfaces for recognition. Journal of Cognitive Neurosciences, 3:71–86.

    Google Scholar 

  • Viola, P. and Jones, M. 2001. Rapid object detection using a boosted cascade of simple features. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 511–518.

  • Wang, X.G. and Tang, X. 2005. Hallucinating face by eigentransformation. IEEE Transactions on Systems, Man, and Cybernetics, Part C, 35(3):425–434.

    Article  Google Scholar 

  • Xiao, R., Li, M.J., and Zhang, H.J. 2004. Robust multipose face detection in images. IEEE Trans. Circuits Syst. Video Techn., 14(1):31– 41.

    Article  Google Scholar 

  • Zhou, Y., Gu, L., and Zhang, H.J. 2003. Bayesian tangent shape model: Estimating shape and pose parameters via bayesian inference. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 109–118.

  • Zhu, S., Wu, Y., and Mumford, D. Filters random fields and maximum entropy (FRAME): To a unified theory for texture modeling. International Journal on Compter Vision, 27:1–20, 1998.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology, Massachusetts, USA

    Ce Liu & William T. Freeman

  2. Microsoft Research Asia, Beijing, China

    Heung-Yeung Shum

Authors
  1. Ce Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Heung-Yeung Shum
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. William T. Freeman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ce Liu.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Liu, C., Shum, HY. & Freeman, W.T. Face Hallucination: Theory and Practice. Int J Comput Vis 75, 115–134 (2007). https://doi.org/10.1007/s11263-006-0029-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-006-0029-5

Keywords

Search

Navigation