Skip to main content
SpringerLink
Log in

A Theory of Shape by Space Carving

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

In this paper we consider the problem of computing the 3D shape of an unknown, arbitrarily-shaped scene from multiple photographs taken at known but arbitrarily-distributed viewpoints. By studying the equivalence class of all 3D shapes that reproduce the input photographs, we prove the existence of a special member of this class, the photo hull, that (1) can be computed directly from photographs of the scene, and (2) subsumes all other members of this class. We then give a provably-correct algorithm, called Space Carving, for computing this shape and present experimental results on complex real-world scenes. The approach is designed to (1) capture photorealistic shapes that accurately model scene appearance from a wide range of viewpoints, and (2) account for the complex interactions between occlusion, parallax, shading, and their view-dependent effects on scene-appearance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • Alfvin, R.L. and Fairchild, M.D. 1997. Observer variability in metameric color matches using color reproduction media. Color Research & Application, 22(3):174–178.

    Google Scholar 

  • Aloimonos, Y. 1988. Visual shape computation. Proc. IEEE, 76:899–916.

    Google Scholar 

  • Armstrong, M.A. 1983. Basic Topology. Springer-Verlag.

  • Bascle, B. and Deriche, R. 1993. Stereo matching, reconstruction and refinement of 3D curves using deformable contours. In Proc. 4th Int. Conf. Computer Vision, pp. 421–430.

  • Beardsley, P., Torr, P., and Zisserman, A. 1996. 3D model acquisition from extended image sequences. In Proc. 4th European Conf. on Computer Vision, pp. 683–695.

  • Belhumeur, P.N. 1996. A bayesian approach to binocular stereopsis. Int. J. on Computer Vision, 19(3):237–260.

    Google Scholar 

  • Belhumeur, P.N. and Kriegman, D.J. 1996. What is the set of images of an object under all possible lighting conditions? In Proc. Computer Vision and Pattern Recognition, pp. 270–277.

  • Bolles, R.C., Baker, H.H., and Marimont, D.H. 1987. Epipolar-plane image analysis: An approach to determining structure from motion. Int. J. Computer Vision, 1:7–55.

    Google Scholar 

  • Bolles, R.C. and Cain, R.A. 1982. Recognizing and locating partially-visible objects: The local-feature-focus method. Int. J. Robotics Research, 1(3):57–82.

    Google Scholar 

  • Cipolla, R. and Blake, A. 1992. Surface shape from the deformation of apparent contours. Int. J. Computer Vision, 9(2):83–112.

    Google Scholar 

  • Collins, R.T. 1996. A space-sweep approach to true multi-image matching. In Proc. Computer Vision and Pattern Recognition Conf., pp. 358–363.

  • Cox, I., Hingorani, S., Rao, S., and Maggs, B. 1996. A maximum likelihood stereo algorithm. CVIU: Image Understanding, 63(3):542–567.

    Google Scholar 

  • Culbertson, W.B., Malzbender, T., and Slabaugh, G. 1999. Generalized voxel coloring. In Workshop on Vision Algorithms: Theory and Practice, Corfu, Greece.

  • Curless, B. and Levoy, M. 1996. A volumetric method for building complex models from range images. In Proc. SIGGRAPH'96, pp. 303–312.

  • Debevec, P.E., Taylor, C.J., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proc. SIGGRAPH'96, pp. 11–20.

  • Epstein, R., Yuille, A.L., and Belhumeur, P.N. 1996. Learning object representations from lighting variations. In Object Representation in Computer Vision II, J. Ponce, A. Zisserman, and M. Hebert (Eds.). Springer-Verlag, pp. 179–199.

  • Faugeras, O. 1995. Stratification of three-dimensional vision: Projective, affine, and metric representations. J. Opt. Soc. Am. A, 12(3):465–484.

    Google Scholar 

  • Faugeras, O.D. 1998. Personal communication.

  • Faugeras, O. and Keriven, R. 1998. Complete dense stereovision using level set methods. In Proc. 5th European Conf. on Computer Vision, pp. 379–393.

  • Faugeras, O.D. and Maybank, S. 1990. Motion from point matches: Multiplicity of solutions. Int. J. Computer Vision, 4:225–246.

    Google Scholar 

  • Foley, J.D., van Dam, A., Feiner, S.K., and Hughes, J.F. 1990. Computer Graphics Principles and Practice. Addison-Wesley Publishing Co.

  • Forsyth, D. and Zisserman, A. 1991. Reflections on shading. IEEE Trans. Pattern Anal. Machine Intell., 13(7):671–679.

    Google Scholar 

  • Freund, J.E. 1992. Mathematical Statistics. Prentice Hall: Englewood Cliffs, NJ.

    Google Scholar 

  • Fua, P. and Leclerc, Y.G. 1995. Object-centered surface reconstruction: Combining multi-image stereo and shading. Int. J. Computer Vision, 16:35–56.

    Google Scholar 

  • Fuchs, H., Kedem, Z., and Naylor, B.F. 1980. On visible surface generation by a priori tree structures. In Proc. SIGGRAPH’ 80, pp. 39–48.

  • Hoff, W. and Ahuja, N. 1989. Surfaces from stereo: Integrating feature matching, disparity estimation, and contour detection. IEEE Trans. Pattern Anal. Machine Intell., 11:121–136.

    Google Scholar 

  • Kakadiaris, I.A. and Metaxas, D. 1995. 3D human body model acquisition from multiple views. In Proc. Int. Conf. on Computer Vision, pp. 618–623.

  • Kanade, T., Narayanan, P.J., and Rander, P.W. 1995. Virtualized reality: Concepts and early results. In Proc. Workshop on Representations of Visual Scenes, pp. 69–76.

  • Kanade, T., Yoshida, A., Oda, K., Kano, H., and Tanaka, M. 1996. A stereo machine for video-rate dense depth mapping and its new applications. In Proc. Computer Vision and Pattern Recognition Conf.

  • Kang, S.B. and Szeliski, R. 1996. 3-D scene data recovery using omnidirectional multibaseline stereo. In Proc. Computer Vision and Pattern Recognition Conf., pp. 364–370.

  • Katayama, A., Tanaka, K., Oshino, T., and Tamura, H. 1995. A viewpoint dependent stereoscopic display using interpolation of multi-viewpoint images. In Proc. SPIE, Vol. 2409A, pp. 21–30.

    Google Scholar 

  • Koenderink, J.J. and van Doorn, A.J. 1991. Affine structure from motion. J. Opt. Soc. Am., A(2):377–385.

    Google Scholar 

  • Kutulakos, K.N. 1997. Shape from the light field boundary. In Proc. Computer Vision and Pattern Recognition, pp. 53–59.

  • Kutulakos, K.N. 2000. Approximate N-view stereo. In Proc. European Conf. on Computer Vision.

  • Kutulakos, K.N. and Dyer, C.R. 1994. Recovering shape by purposive viewpoint adjustment. Int. J. Computer Vision, 12(2):113–136.

    Google Scholar 

  • Kutulakos, K.N. and Dyer, C.R. 1995. Global surface reconstruction by purposive control of observer motion. Artificial Intelligence Journal, 78(1– 2):147–177.

    Google Scholar 

  • Langer, M.S. and Zucker, S.W. 1994. Shape-from-shading on a cloudy day. J. Opt. Soc. Am. A, 11(2):467–478.

    Google Scholar 

  • Laurentini, A. 1994. The visual hull concept for silhouette-based image understanding. IEEE Trans. Pattern Anal. Machine Intell., 16(2):150–162.

    Google Scholar 

  • Marr, D. 1982. Vision. Freeman.

  • Martin, W.N. and Aggarwal, J.K. 1983. Volumetric descriptions of objects from multiple views. IEEE Proc. Pattern Anal. Machine Intell., 5(2):150–158.

    Google Scholar 

  • Moezzi, S., Katkere, A., Kuramura, D.Y., and Jain, R. 1996. Reality modeling and visualization from multiple video sequences. IEEE Computer Graphics and Applications, 16(6):58–63.

    Google Scholar 

  • Mundy, J.L. and Zisserman, A. (Eds.). 1992. Geometric Invariance in Computer Vision. MIT Press.

  • Narayanan, P.J., Rander, P.W., and Kanade, T. 1998. Constructing virtual worlds using dense stereo. In Proc. Int. Conf. on Computer Vision, pp. 3–10.

  • Newell, M.E., Newell, R.G., and Sancha, T.L. 1972. A solution to the hidden surface problem. In Proc. ACM National Conference, pp. 443–450.

  • Okutomi, M. and Kanade, T. 1993. A multiple-baseline stereo. IEEE Trans. Pattern Anal. Machine Intell., 15(4):353–363.

    Google Scholar 

  • Poggio, T., Torre, V., and Koch, C. 1985. Computational vision and regularization theory. Nature, 317(26):314–319.

    Google Scholar 

  • Pollefeys, M., Koch, R., and Gool, L.V. 1998. Self-calibration and metric reconstruction in spite of varying and unknown internal camera parameters. In Proc. 6th Int. Conf. on Computer Vision, pp. 90–95.

  • Pritchett, P. and Zisserman, A. 1998. Wide baseline stereo matching. In Proc. 6th Int. Conf. on Computer Vision, pp. 754–760.

  • Roy, S. and Cox, I.J. 1998. A maximum-flow formulation of the Ncamera stereo correspondence problem. In Proc. 6th Int. Conf. on Computer Vision, pp. 492–499.

  • Sato, Y., Wheeler, M.D., and Ikeuchi, K. 1997. Object shape and reflectance modeling from observation. In Proc. SIGGRAPH'97, pp. 379–3870.

  • Seales, W.B. and Faugeras, O. 1995. Building three-dimensional object models from image sequences. Computer Vision and Image Understanding, 61(3):308–324.

    Google Scholar 

  • Seitz, S.M. and Dyer, C.R. 1995. Complete scene structure from four point correspondences. In Proc. 5th Int. Conf. on Computer Vision, pp. 330–337.

  • Seitz, S.M. and Dyer, C.R. 1999. Photorealistic scene reconstruction by voxel coloring. Int. J. Computer Vision, 35(2):151–173.

    Google Scholar 

  • Seitz, S.M. and Kutulakos, K.N. 1998. Plenoptic image editing. In Proc. 6th Int. Conf. Computer Vision, pp. 17–24.

  • Smith, A.R. and Blinn, J.F. 1996. Blue screen matting. In Proc. SIGGRAPH'96, pp. 259–268.

  • Stewart, C.V. 1995. MINPRAN: A new robust estimator for computer vision. IEEE Trans. Pattern Anal. Machine Intell., 17(10):925–938.

    Google Scholar 

  • Szeliski, R. 1993. Rapid octree construction from image sequences. CVGIP: Image Understanding, 58(1):23–32.

    Google Scholar 

  • Szeliski, R. and Golland, P. 1998. Stereo matching with transparency and matting. In Proc. 6th Int. Conf. on Computer Vision, pp. 517–524.

  • Szeliski, R. and Weiss, R. 1994. Robust shape recovery from occluding contours using a linear smoother. In Real-time Computer Vision, C.M. Brown and D. Terzopoulos (Eds.). Cambridge University Press, pp. 141–165.

  • Tomasi, C. and Kanade, T. 1992. Shape and motion from image streams under orthography: A factorization method. Int. J. Computer Vision, 9(2):137–154.

    Google Scholar 

  • Torrance, K.E. and Sparrow, E.M. 1967. Theory of off-specular reflection from roughened surface. Journal of the Optical Society of America, 57:1105–1114.

    Google Scholar 

  • Turk, G. and Levoy, M. 1994. Zippered polygon meshes from range images. In Proc. SIGGRAPH'94, pp. 311–318.

  • Vaillant, R. and Faugeras, O.D. 1992. Using extremal boundaries for 3-D object modeling. IEEE Trans. Pattern Anal. Machine Intell., 14(2):157–173.

    Google Scholar 

  • van Veen, J.A.J.C. and Werkhoven, P. 1996. Metamerisms in structure-from-motion perception. Vision Research, 36(14):2197–2210.

    Google Scholar 

  • Woodham, R.J., Iwahori, Y., and Barman, R.A. 1991. Photometric stereo: Lambertian reflectance and light sources with unknown direction and strength. Technical Report 91-18, University of British Columbia, Laboratory for Computational Intelligence.

  • Zhang, Z. 1998. Image-based geometrically-correct photorealistic scene/object modeling (IBPhM): A review. In Proc. 3rd Asian Conf. on Computer Vision, pp. 340–349.

  • Zhao, C. and Mohr, R. 1996. Global three-dimensional surface reconstruction from occluding contours. ComputerVision and Image Understanding, 64(1):62–96.

    Google Scholar 

  • Zitnick, C.L. and Webb, J.A. 1996. Multi-baseline stereo using surface extraction. Technical Report CMU-CS-96-196, Carnegie Mellon University, Pittsburgh, PA.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Department of Dermatology, University of Rochester, Rochester, NY, 14627, USA

    Kiriakos N. Kutulakos

  2. The Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, 15213, USA

    Steven M. Seitz

Authors
  1. Kiriakos N. Kutulakos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Steven M. Seitz
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kutulakos, K.N., Seitz, S.M. A Theory of Shape by Space Carving. International Journal of Computer Vision 38, 199–218 (2000). https://doi.org/10.1023/A:1008191222954

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1008191222954

Search

Navigation