Skip to main content
SpringerLink
Log in

A Theory of Refractive and Specular 3D Shape by Light-Path Triangulation

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

We investigate the feasibility of reconstructing an arbitrarily-shaped specular scene (refractive or mirror-like) from one or more viewpoints. By reducing shape recovery to the problem of reconstructing individual 3D light paths that cross the image plane, we obtain three key results. First, we show how to compute the depth map of a specular scene from a single viewpoint, when the scene redirects incoming light just once. Second, for scenes where incoming light undergoes two refractions or reflections, we show that three viewpoints are sufficient to enable reconstruction in the general case. Third, we show that it is impossible to reconstruct individual light paths when light is redirected more than twice. Our analysis assumes that, for every point on the image plane, we know at least one 3D point on its light path. This leads to reconstruction algorithms that rely on an “environment matting” procedure to establish pixel-to-point correspondences along a light path. Preliminary results for a variety of scenes (mirror, glass, etc.) are also presented.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Agarwal, S., Mallick, S. P., Kriegman, D., & Belongie, S. (2004). On refractive optical flow. In Proceedings of the 8th European conference on computer vision (pp. 483–494).

  • Baba, M., Ohtani, K., & Imai, M. (2001). New laser range finder for three-dimensional shape measurement of specular objects. Optical Engineering, 40(1), 53–60.

    Article  Google Scholar 

  • Ben-Ezra, M., & Nayar, S. (2003). What does motion reveal about transparency? In Proceedings of the 9th international conference on computer vision (pp. 1025–1032).

  • Blake, A. (1985). Specular stereo. In Proceedings of the international joint conference on artificial intelligence (pp. 973–976).

  • Bonfort, T., & Sturm, P. (2003). Voxel carving for specular surfaces. In Proceedings of the 9th international conference on computer vision (pp. 591–596).

  • Bonfort, T., Sturm, P., & Gargallo, P. (2006). General specular surface triangulation. In Proceedings of the Asian conference on computer vision (pp. 872–881).

  • Bouguet, J.-Y. (2001) MATLAB camera calibration toolbox. http://www.vision.caltech.edu/bouguetj/calib_doc/.

  • Faugeras, O., & Keriven, R. (1998). Complete dense stereovision using level set methods. In Proceedings of the 5th European conference on computer vision (pp. 379–393).

  • Fischler, M., & Bolles, R. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24, 381–395.

    Article  MathSciNet  Google Scholar 

  • Glassner, A. S. (1995). Principles of digital image synthesis. Los Altos: Kaufmann.

    Google Scholar 

  • Gluckman, J. M., & Nayar, S. K. (1999) Planar catadioptric stereo: geometry and calibration. In Proceedings of the computer vision and pattern recognition conference (pp. 22–28).

  • Grossberg, M. D., & Nayar, S. K. (2001). A general imaging model and a method for finding its parameters. In Proceedings of the 8th international conference on computer vision (Vol. 2, pp. 108–115).

  • Guillemin, V., & Pollack, A. (1974). Differential topology. New York: Prentice-Hall.

    MATH  Google Scholar 

  • Halstead, M. A., Barsky, B. A., Klein, S. A., & Mandell, R. B. (1996). Reconstructing curved surfaces from specular reflection patterns using spline surface fitting of normals. In Proceedings of ACM SIGGRAPH (pp. 335–342).

  • Hasinoff, S., & Kutulakos, K. N. (2003). Photo-consistent 3d fire by flame sheet decomposition. In Proceedings of the 9th international conference on computer vision (pp. 1184–1191).

  • Hertzman, A., & Seitz, S. M. (2003). Shape and materials by example: a photometric stereo approach. In Proceedings of the computer vision and pattern recognition conference (pp. 533–540).

  • Höhle, J. (1971) Reconstruction of the underwater object. In Photogrammetric engineering (pp. 948–954).

  • Ikeuchi, K. (1981). Determining surface orientations of specular surfaces by using the photometric stereo method. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3(6), 661–669.

    Google Scholar 

  • Jähne, B., Klinke, J., & Waas, S. (1994). Imaging of short ocean wind waves: a critical theoretical review. Journal of the Optical Society of America A, 11(8), 2197–2209.

    Google Scholar 

  • Keller, W. C., & Gotwols, B. L. (1983). Two-dimensional optical measurement of wave slope. Applied Optics, 22(22), 3476–3478.

    Article  Google Scholar 

  • Koendering, J. J., & van Doorn, A. J. (1979). The structure of two-dimensional scalar fields with applications to vision. Biological Cybernetics, 33, 151–158.

    Article  Google Scholar 

  • Kutulakos, K. N., & Steger, E. (2005). A theory of refractive and specular 3d shape by light-path triangulation. In Proceedings of the 10th international conference on computer vision (pp. 1448–1455).

  • Maas, H.-G. (1995) New developments in multimedia photogrammetry. In Optical 3D measurement techniques III. Karlsruhe: Wichmann.

    Google Scholar 

  • Matusik, W., Pfister, H., Ziegler, R., Ngan, A., & McMillan, L. (2002). Acquisition and rendering of transparent and refractive objects. In Proceedings of the 13th Eurographics workshop on rendering (pp. 267–278).

  • Miyazaki, D., Kagesawa, M., & Ikeuchi, K. (2004). Transparent surface modeling from a pair of polarization images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 26(1), 73–82.

    Article  Google Scholar 

  • Morris, N. J. W. (2004). Image-based water surface reconstruction with refractive stereo. Master’s thesis, Department of Computer Science, University of Toronto. http://www.dgp.toronto.edu/~nmorris/Water/thesis.pdf.

  • Morris, N., & Kutulakos, K. N. (2005). Dynamic refraction stereo. In Proceedings of the 10th international conference on computer vision (pp. 1573–1580).

  • Murase, H. (1990). Surface shape reconstruction of an undulating transparent object. In Proceedings of the 3rd international conference on computer vision (pp. 313–317).

  • Nayar, S. K., Ikeuchi, K., & Kanade, T. (1991). Surface reflection: physical and geometrical perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(7), 611–634.

    Article  Google Scholar 

  • Nehab, D., Rusinkiewicz, S., Davis, J., & Ramamoorthi, R. (2005). Efficiently combining positions and normals for precise 3d geometry. In Proceedings of ACM SIGGRAPH (pp. 536–543).

  • Oren, M., & Nayar, S. K. (1995). A theory of specular surface geometry. In Proceedings of the 5th international conference on computer vision (pp. 740–747).

  • Pless, R. (2002). Two view discrete and differential constraints for generalized imaging systems. In Proceedings of the IEEE workshop on omnidirectional vision (pp. 53–59).

  • Sanderson, A. C., Weiss, L. E., & Nayar, S. K. (1988). Structured highlight inspection of specular surfaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 10(1), 44–55.

    Article  Google Scholar 

  • Savarese, S., & Perona, P. (2002). Local analysis for 3d reconstruction of specular surfaces, part ii. In Proceedings of the 7th European conference on computer vision (pp. 759–774).

  • Sharpe, J., Ahlgren, U., Perry, P., Hill, B., Ross, A., Hecksher-Sorensen, J., Baldock, R., & Davidson, D. (2002). Optical projection tomography as a tool for 3d microscopy and gene expression studies. Science, 296, 541–545.

    Article  Google Scholar 

  • Steger, E. (2006). Reconstructing transparent objects by refractive light-path triangulation. Master’s thesis, Department of Computer Science, University of Toronto. http://www.cs.toronto.edu/~esteger/thesis.pdf.

  • Tarini, M., Lensch, H. P. A., Goesele, M., & Seidel, H. -P. (2005). 3D acquisition of mirroring objects using striped patterns. Graphical Models, 67(4), 233–259.

    Article  Google Scholar 

  • Trifonov, B., Bradley, D., & Heidrich, W. (2006). Tomographic reconstruction of transparent objects. In Proceedings of the 17th Eurographics symposium on rendering (pp. 51–60).

  • Wang, J., & Dana, K. J. (2003). A novel approach for texture shape recovery. In Proceedings of the 9th international conference on computer vision (pp. 1374–1380).

  • Zhang, X., & Cox, C. S. (1994). Measuring the two-dimensional structure of a wavy water surface optically: a surface gradient detector. Experiments in Fluids, 17, 225–237.

    Article  MATH  Google Scholar 

  • Zheng, J. Y., & Murata, A. (2000). Acquiring a complete 3d model from specular motion under the illumination of circular-shaped light sources. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8), 913–920.

    Article  Google Scholar 

  • Zickler, T., Belhumeur, P. N., & Kriegman, D. J. (2002). Helmholz stereopsis: exploiting reciprocity for surface reconstruction. In Proceedings of the 7th European conference on computer vision (pp. 869–884).

  • Zongker, D. E., Werner, D. M., Curless, B., & Salesin, D. H. (1999). Environment matting and compositing. In Proceedings of ACM SIGGRAPH (pp. 205–214).

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Toronto, Toronto, ON, M5S 3G4, Canada

    Kiriakos N. Kutulakos & Eron Steger

Authors
  1. Kiriakos N. Kutulakos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eron Steger
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kiriakos N. Kutulakos.

Additional information

Part of this research was conducted while K. Kutulakos was serving as a Visiting Scholar at Microsoft Research Asia.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kutulakos, K.N., Steger, E. A Theory of Refractive and Specular 3D Shape by Light-Path Triangulation. Int J Comput Vis 76, 13–29 (2008). https://doi.org/10.1007/s11263-007-0049-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-007-0049-9

Keywords

Search

Navigation