Skip to main content
SpringerLink
Log in

Appearance-Cloning: Photo-Consistent Scene Recovery from Multi-View Images

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

This paper introduces the novel volumetric methodology “appearance-cloning” as a viable solution for achieving a more improved photo-consistent scene recovery, including a greatly enhanced geometric recovery performance, from a set of photographs taken at arbitrarily distributed multiple camera viewpoints. We do so while solving many of the problems associated with previous stereo-based and volumetric methodologies. We redesign the photo-consistency decision problem of individual voxel in volumetric space as the photo-consistent shape search problem in image space, by generalizing the concept of the point correspondence search between two images in stereo-based approach, within a volumetric framework.

In detail, we introduce a self-constrained greedy-style optimization methodology, which iteratively searches a more photo-consistent shape based on the probabilistic shape photo-consistency measure, by using the probabilistic competition between candidate shapes. Our new measure is designed to bring back the probabilistic photo-consistency of a shape by comparing the appearances captured from multiple cameras with those rendered from that shape using the per-pixel Maxwell model in image space.

Through various scene recoveries experiments including specular and dynamic scenes, we demonstrate that if sufficient appearances are given enough to reflect scene characteristics, our appearance-cloning approach can successfully recover both the geometry and photometry information of a scene without any kind of scene-dependent algorithm tuning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Baker, H.H. and Binford, T.O. 1981. Depth from edge and intensity based stereo. In International Joint Conference on Artificial Intelligence, pp. 631–639.

  • Bhotika, R., Fleet, D., and Kutulakos, K. 2002. A probabilistic theory of occupancy and emptiness. In Proc. European Conference on Computer Vision, vol. 3, pp. 112–132.

  • Bhotika, R. 2003. Scene-space methods for bayesian inference of 3d shape and motion, Ph. D. Thesis, University of Rochester, Computer Science Department.

  • Broadhurst, A. 2001. A probabilistic framework for space carving. Ph.D. Thesis, Dept. of Engineering, University of Cambridge.

  • Broadhurst, A. and Cipolla, R. 2000. A statistical consistency check for the space carving algorithm. In British Machine Vision Conference, pp. 282–291.

  • Broadhurst, A., Drummond, T., and Cipolla, R. 2001. A probabilistic framework for space carving. In International Conference of Computer Vision, pp. 388–393.

  • Cheung, K.M. 2003. Visual hull construction, alignment and refinement for human kinematic modeling, motion tracking and rendering. Doctoral dissertation, Tech. Report CMU-RI-TR-03-44, Robotics Institute, Carnegie Mellon University.

  • Culbertson, W.B., Malzbender, T., and Slabaugh, G. 1999. Generalized voxel coloring. In International Conference on Computer Vision Workshop on Vision Algorithms, Theory and Practice, Springer-Verlag Lecture Notes in Computer Science Vol. 1883, pp. 100–115.

  • Dinh, H.Q., Turk, G., and Slabaugh, G. 2002. Reconstructing surfaces by volumetric regularization using radial basis functions. IEEE Trans. on Pattern Analysis and Machine Intelligence, 24(10):1358–1371.

    Google Scholar 

  • Faugeras, O. 1993. Three-Dimensional Computer Vision: A Geometrical Viewpoint. The MIT Press: Cambridge, MA (second printing).

    Google Scholar 

  • Faugeras, O. and Keriven, R. 1998. Variational principles, surface evolution, PDE's, level set methods and the stereo problem, Special Issue on Partial Differential Equations and Geometry-Driven Diffusion in Image Processing and Analysis of the IEEE Transactions on Image Processing, 7(3):336–344.

  • Garland, M. and Heckbert, P.S. 1997. Surface simplification using quadric error metrics. In Computer Graphics (SIGGRAPH97).

  • Ikeuchi, K., Nakazawa, A., Hasegawa, K., and Ohishi, T. 2003. The great buddha project: Modeling cultural heritage for VR systems through observation. In IEEE and ACM International Symposium on Mixed and Augmented Reality.

  • Kim, J.S., Kim, H.W., and Kweon, I.S. 2002. A camera calibration method using concentric circles for vision applications. In Proc. 5th Asian Conference on Computer Vision, pp. 515–520.

  • Kutulakos, K.N. 2000. Approximate N-view stereo. In Proc. 6th European Conference on Computer Vision, pp. 67–83.

  • Kutulakos, K.N. and Seitz, M. 1998. What do N photographs tell us about 3D shape? Technical Report TR680, Computer Science Dept., U. Rochester, January.

  • Kutulakos, K.N. and Seitz, S. 1999. A theory of shape by space carving. In International Conference on Computer Vision, pp. 307–314.

  • Kutulakos, K.N. and Seitz, S. 2000. A theory of shape by space carving. International Journal of Computer Vision, 38(3):199–218.

    Article  Google Scholar 

  • Laurentini, A. 1994. The visual hull concept for silhouette-based image understanding. IEEE Trans. on Pattern Recognition and Machine Intelligence, 16(2):150–162.

    Google Scholar 

  • Levoy, M., Pulli, K., Curless, B., Rusinkiewicz, S., Koller, D., Pereira, L., Ginzton, M., Anderson, S., David, J., Ginsberg, J., Shade, J., and Fulk, D. 2000. The Digital Michelangelo project: 3d scanning of large statues. In Computer Graphics (SIGGRAPH2000).

  • Lorensen, W.E. and Cline, H.E. 1987. Marching Cubes: A high resolution 3D surface reconstruction algorithm. Computer Graphics, 21(4):163–169.

    Google Scholar 

  • Marr, D. and Poggio, T. 1976. Cooperative computation of stereo disparity. Science 194:283–287.

    Google Scholar 

  • Mortensen, E. and Barrett, W. 1995. Intelligent scissors for image composition. In Computer Graphics (SIGGGRAPH’95), pp. 191–198.

  • Nayar, S.K., Ikeuchi, K., and Kanade, T. 1991. Shape from Interreflections. International Journal of Computer Vision, 6(3):173–195.

    Article  Google Scholar 

  • Prock, A. and Dyer, C. 1998. Towards real-time voxel coloring. In Image Understanding Workshop, pp. 315–321.

  • Seitz, S. and Dyer, C. 1997. Photorealistic scene reconstruction by voxel coloring. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1067–1073.

  • Slabaugh, G. 2002. Novel volumetric scene reconstruction methods for new view synthesis. Ph.D. Thesis, Georgia Institute of Technology.

  • Slabaugh, G., Culbertson, W.B., Malzbender, T., and Schafer, R. 2001. A survey of volumetric scene reconstruction methods from photographs. In Proc. International Workshop on Volume Graphics, pp. 81–100.

  • Slabaugh, G., Culbertson, W.B., Malzbender, T., Stevens, M.R., and Schafer, R.W. 2004. Methods for volumetric reconstruction of visual scenes. International Journal of Computer Vision, 57(3):179–199.

    Article  Google Scholar 

  • Stevens, M., Culbertson, W.B., and Malzbender, T. 2002. A histogram-based color consistency test for voxel coloring. In Proc. International Conference on Pattern Recognition, 4:118–121.

  • Szeliski, R. 1993. Rapid octree construction from image sequences. Computer Vision, Graphics and Image Processing: Image Understanding, 58(1):23–32.

    Google Scholar 

  • Vedula, S., Baker, S., Seitz, S.M., and Kanade, T. 2000. Shape and motion carving in 6D. In Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 324–344.

  • Yang, R., Pollefeys, M., and Welch, G. 2003. Dealing with textureless regions and specular highlights—A progressive space carving scheme using a novel photo-consistency measure. In Proc. International Conference on Computer Vision.

  • Yezzi, A., Slabaugh, G., Broadhurst, A., Cipolla, R., and Schafer, R. 2002. A surface evolution approach to probabilistic space carving. In International Symposium on 3D Processing, Visualization and Transmission, pp. 618–621.

  • Yoon, K.J. and Kweon, I.S. 2004. Voting-based separation of diffuse and specular pixels. Electronics Letters, 40(20):1260–1261.

    Article  Google Scholar 

  • Zhang, L., Curless, B., and Seitz, S.M. 2003. Spacetime stereo: Shape recovery for dynamic scenes. In Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 367–374.

Download references

Author information

Authors and Affiliations

  1. Department of Electrical Engineering and Computer Science, Korea Advanced Institute of Science and Technology, 373-1 Guseong-dong, Yuseong-gu, Daejeon, Korea

    Howon Kim & In So Kweon

Authors
  1. Howon Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. In So Kweon
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Howon Kim.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kim, H., Kweon, I.S. Appearance-Cloning: Photo-Consistent Scene Recovery from Multi-View Images. Int J Comput Vision 66, 163–192 (2006). https://doi.org/10.1007/s11263-005-3956-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-005-3956-7

Keywords

Search

Navigation