Skip to main content
SpringerLink
Log in

Depth Sensing Using Geometrically Constrained Polarization Normals

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Analyzing the polarimetric properties of reflected light is a potential source of shape information. However, it is well-known that polarimetric information contains fundamental shape ambiguities, leading to an underconstrained problem of recovering 3D geometry. To address this problem, we use additional geometric information, from coarse depth maps, to constrain the shape information from polarization cues. Our main contribution is a framework that combines surface normals from polarization (hereafter polarization normals) with an aligned depth map. The additional geometric constraints are used to mitigate physics-based artifacts, such as azimuthal ambiguity, refractive distortion and fronto-parallel signal degradation. We believe our work may have practical implications for optical engineering, demonstrating a new option for state-of-the-art 3D reconstruction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. For example, (Yu et al. 2013; Han et al. 2013; Wu et al. 2014) using SfS, and (Nehab et al. 2005; Haque et al. 2014) using PS.

  2. Formalized by Augustin-Jean Fresnel (1788–1827) as the famous “Fresnel Equations”.

  3. For this paper, we rotate the polarizer to the desired angle. However, the mechanical process of rotation could lead to small errors, for which Schechner (2015) has devised a self-calibrating solution.

  4. In practice, the degree of polarization is generally an order of magnitude larger for specular dominant reflections.

  5. Refer to Sect. 5 for a detailed analysis of key assumptions.

  6. For all experiments, this distance was set to 20 millimeters. To find the neighborhood, we use the kd-tree search algorithm, which can be implemented by the rangesearch command in MATLAB.

  7. \({\mathbf {N}^{\text {polar}}}\) is obtained through shape from polarization and this normal map will suffer from the physics-based artifacts described previously. This can be seen visually in Fig. 1c.

  8. The 2D-TV implementation parallels the optimization program from prior work (Kadambi and Boufounos 2015).

  9. Calculation of \(2\times 2^K - V\): each facet can have 2 possible normal orientations due to the azimuthal ambiguity, leading to \(2^K\) possible surface configurations due only to the degrees of freedom of the facets. By constraining the rest of the surface using the anchor point as a boundary condition, this leads to \(2\times 2^K\) surface configurations. Since we assume that the facet is continuous with the anchor point, the overall dimensionality is reduced to \(2\times 2^K - V\).

  10. Implementing high-frequency ambiguity correction. First, a difference image is formed of the depth normals and polarization normals. The difference image only contains detailed features, as it would not show up in the former (otherwise the method from Sect. 4.1.1 would have been sufficient). From the difference image, the pixel at an edge from 0 to a non-zero value represents an anchor point. An edge corresponding to a change in sign (e.g., at the V-groove of a corner) is a pivot point. A greedy approach is used to flip surface facets, which enforces a closed surface constraint. Details about the closed surface constraint can be found in Miyazaki et al. (2003).

  11. Note: this notion of a patch operates on the pixel grid and is thus different from our convention of defining a point cloud neighborhood (cf. Sect. 4.1). For our paper, we use a 7x7 patch.

  12. Estimation of the sinusoidal parameters from Eq. 1 becomes unstable when there is little contrast between \(I_{\min }\) and \(I_{\max }\).

  13. A value of \(\lambda = 0.02\) is recommended.

  14. In this paper, we use the term “reflection components” following an optical imaging convention. This is identical to the term “reflectance components”, which may be more familiar to some readers.

  15. Laser Scanner: http://nextengine.com/assets/pdf/scanner-techspecs.pdf.

  16. While there are variants of generalized photometric stereo for non-Lambertian objects and natural environment lighting, they usually require about 50-100 images according to a recent survey in Shi et al. (2016).

  17. Polarization mosaic: http://moxtek.com/optics-product/pixelated-polarizer.

References

  • Ackermann, J., Langguth, F., Fuhrmann, S., & Goesele, M. (2012). Photometric stereo for outdoor webcams. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

  • Agrawal, A., Raskar, R., & Chellappa, R. (2006). What is the range of surface reconstructions from a gradient field. ECCV

  • Atkinson, G. A., & Hancock, E. R. (2005). Multi-view surface reconstruction using polarization. ICCV.

  • Atkinson, G. A., & Hancock, E. R. (2006). Recovery of surface orientation from diffuse polarization. IEEE Transactions on Image Processing, 15, 1653–1664.

  • Basri, R., Jacobs, D., & Kemelmacher, I. (2007). Photometric stereo with general, unknown lighting. IJCV.

  • Chen, T., Lensch, H. P. A., Fuchs, C., & Seidel, H. P. (2007). Polarization and phase-shifting for 3d scanning of translucent objects. CVPR.

  • Cula, O. G., Dana, K. J., Pai, D. K., & Wang, D. (2007). Polarization multiplexing and demultiplexing for appearance-based modeling. IEEE TPAMI.

  • Dickson, C. N., Wallace, A. M., Kitchin, M., & Connor, B. (2015). Long-wave infrared polarimetric cluster-based vehicle detection. JOSA A, 32(12), 2307–2315.

    Article  Google Scholar 

  • Esteban, C., Vogiatzis, G., & Cipolla, R. (2008). Multiview photometric stereo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30, 548–554.

  • Fraile, R., & Hancock, E. R. (2006). Combinatorial surface integration. ICPR.

  • Ghosh, A., Chen, T., Peers, P., Wilson, C. A., & Debevec, P. (2009). Estimating specular roughness and anisotropy from second order spherical gradient illumination. EGSR.

  • Ghosh, A., Chen, T., Peers, P., Wilson, C. A., & Debevec, P. (2010). Circularly polarized spherical illumination reflectometry. SIGGRAPH Asia.

  • Ghosh, A., Fyffe, G., Tunwattanapong, B., Busch, J., Yu, X., & Debevec, P. (2011). Multiview face capture using polarized spherical gradient illumination. SIGGRAPH Asia.

  • Guarnera, G. C., Peers, P., Debevec, P., & Ghosh, A. (2012). Estimating surface normals from spherical stokes reflectance fields. ECCV Workshops.

  • Gupta, M., Nayar, S. K., Hullin, M. B., & Martin, J. (2015). Phasor imaging: A generalization of correlation-based time-of-flight imaging. ACM Transactions on Graphics (TOG). doi:10.1145/2735702

  • Han, Y., Lee, J., & Kweon, I. (2013). High quality shape from a single RGBD image under uncalibrated natural illumination. ICCV.

  • Haque, S. M., Chatterjee, A., & Govindu, V. M. (2014). High quality photometric reconstruction using a depth camera. CVPR.

  • Hecht, E. (2002). Optics (4th ed.). San Francisco: Addison-Wesley.

    Google Scholar 

  • Holroyd, M., Lawrence, J., Humphreys, G., & Zickler, T. (2008). A photometric approach for estimating normals and tangents. ACM Transactions on Graphics (TOG). doi:10.1145/1457515.1409086.

  • Huynh, C. P., Robles-Kelly, A., & Hancock, E. R. (2013). Shape and refractive index from single-view spectro-polarimetric images. International Journal of Computer Vision, 101(1), 64–94.

    Article  MathSciNet  Google Scholar 

  • Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., Newcombe, R., Kohli, P., et al. (2011). Kinectfusion: Real-time 3D reconstruction and interaction using a moving depth camera. ACM UIST.

  • Jayasuriya, S., Sivaramakrishnan, S., Chuang, E., Guruaribam, D., Wang, A., & Molnar, A. (2015). Dual light field and polarization imaging using cmos diffractive image sensors. Optics Letters, 40(10), 2433–2436.

    Article  Google Scholar 

  • Joshi, N., & Kriegman, D. (2007). Shape from varying illumination and viewpoint. ICCV.

  • Kadambi, A., & Boufounos, P. T. (2015). Coded aperture compressive 3-d lidar. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1166–1170). IEEE.

  • Kadambi, A., Taamazyan, V., Shi, B., & Raskar, R. (2015a). Polarized 3d: High-quality depth sensing with polarization cues. ICCV.

  • Kadambi, A., Taamazyan, V., Shi, B., & Raskar, R. (2015b). Polarized 3d: synthesis of polarization and depth cues for enhanced 3d sensing. In: SIGGRAPH 2015: Studio, p. 23. ACM.

  • Klasing, K., Althoff, D., Wollherr, D. & Buss, M. (2011). Comparison of surface normal estimation methods for range sensing applications. ICRA.

  • Lanman, D., Wetzstein, G., Hirsch, M., Heidrich, W., & Raskar, R. (2011). Polarization fields: Dynamic light field display using multi-layer lcds. SIGGRAPH Asia.

  • Ma, W. C., Hawkins, T., Peers, P., Chabert, C. F., Weiss, M., & Debevec, P. (2007). Rapid acquisition of specular and diffuse normal maps from polarized spherical gradient illumination. Eurographics.

  • Manakov, A., Restrepo, J. F., Klehm, O., Hegedüs, R., Eisemann, E., Seidel, H. P., et al. (2013). A reconfigurable camera add-on for high dynamic range, multispectral, polarization, and light-field imaging. SIGGRAPH.

  • Mitra, N. J., & Nguyen, A. (2003). Estimating surface normals in noisy point cloud data. Geom: Eurographics Symp. on Comp.

  • Miyazaki, D., Kagesawa, M., & Ikeuchi, K. (2004). Transparent surface modeling from a pair of polarization images. TPAMI.

  • Miyazaki, D., Shigetomi, T., Baba, M., Furukawa, R., Hiura, S., & Asada, N. (2012). Polarization-based surface normal estimation of black specular objects from multiple viewpoints. 3DIMPVT.

  • Miyazaki, D., Tan, R. T., Hara, K., & Ikeuchi, K. (2003). Polarization-based inverse rendering from a single view. ICCV.

  • Morel, O., Meriaudeau, F., Stolz, C., & Gorria, P. (2005). Polarization imaging applied to 3d reconstruction of specular metallic surfaces. Electronic Imaging.

  • Naik, N., Kadambi, A., Rhemann, C., Izadi, S., Raskar, R., & Kang, S.B. (2015). A light transport model for mitigating multipath interference in tof sensors. CVPR.

  • Nayar, S. K., Fang, X. S., & Boult, T. (1997) Separation of reflection components using color and polarization. IJCV.

  • Nehab, D., Rusinkiewicz, S., Davis, J., & Ramamoorthi, R. (2005). Efficiently combining positions and normals for precise 3d geometry. SIGGRAPH.

  • Ngo Thanh, T., Nagahara, H., & Taniguchi, R. I. (2015). Shape and light directions from shading and polarization.

  • Nießner, M., Zollhöfer, M., Izadi, S., & Stamminger, M. (2013). Real-time 3d reconstruction at scale using voxel hashing. SIGGRAPH Asia.

  • Oliensis, J. (1991). Uniqueness in shape from shading.

  • Or-el, R., Rosman, G., Wetzler, A., Kimmel, R., & Bruckstein, A. (2015). Rgbd-fusion: Real-time high precision depth recovery.

  • O’Toole, M., Heide, F., Xiao, L., Hullin, M.B., Heidrich, W., & Kutulakos, K.N. (2014). Temporal frequency probing for 5d transient analysis of global light transport. SIGGRAPH.

  • Rahmann, S., & Canterakis, N. (2001). Reconstruction of specular surfaces using polarization imaging.

  • Robles-Kelly, A., & Huynh, C. P. (2012). Imaging spectroscopy for scene analysis. Berlin: Springer.

    Google Scholar 

  • Saito, M., Sato, Y., Ikeuchi, K., & Kashiwagi, H. (1999). Measurement of surface orientations of transparent objects using polarization in highlight. CVPR.

  • Schechner, Y. Y. (2015). Self-calibrating imaging polarimetry. IEEE ICCP.

  • Schechner, Y. Y., Narasimhan, S. G., Nayar, & S. K. (2001). Instant dehazing of images using polarization. CVPR.

  • Schechner, Y. Y., & Nayar, S. K. (2005). Generalized mosaicing: Polarization panorama. IEEE TPAMI.

  • Shi, B., Inose, K., Matsushita, Y., Tan, P., Yeung, S. K., & Ikeuchi, K. (2014). Photometric stereo using internet images. IEEE.

  • Shi, B., Tan, P., Matsushita, Y., & Ikeuchi, K. (2014). Bi-polynomial modeling of low-frequency reflectances. TPAMI.

  • Shi, B., Wu, Z., Mo, Z., Duan, D., Yeung, S. K., & Tan, P. (2016). A benchmark dataset and evaluation for non-lambertian and uncalibrated photometric stereo. CVPR.

  • Smith, W. A., Ramamoorthi, R., Tozza, S. (2016). Linear depth estimation from an uncalibrated, monocular polarisation image.

  • Stolz, C., Kechiche, A. Z., & Aubreton, O. (2016) Short review of polarimetric imaging based method for 3d measurements. In: SPIE Photonics Europe, pp. 98,960P–98,960P. International Society for Optics and Photonics.

  • Treibitz, T., & Schechner, Y. Y. (2009). Active polarization descattering. IEEE TPAMI.

  • Wallace, A. M., Liang, B., Trucco, E., & Clark, J. (1999). Improving depth image acquisition using polarized light. International Journal of Computer Vision, 32(2), 87–109.

    Article  Google Scholar 

  • Wolff, L. B. (1997). Polarization vision: a new sensory approach to image understanding. Image and Vision computing, 15(2), 81–93.

    Article  Google Scholar 

  • Woodham, R. J. (1980). Photometric method for determining surface orientation from multiple images. Optical Engineering, 19(1), 139–144.

  • Wu, C., Wilburn, B., Matsushita, Y., & Theobalt, C. (2011). High-quality shape from multi-view stereo and shading under general illumination. CVPR.

  • Wu, C., Zollhöfer, M., Nießner, M., Stamminger, M., Izadi, S., & Theobalt, C. (2014). Real-time shading-based refinement for consumer depth cameras. SIGGRAPH.

  • Yeung, S. K., Wu, T. P., Tang, C. K., Chan, T. F., & Osher, S. J. (2014). Normal estimation of a transparent object using a video. TPAMI.

  • Yu, L. F., Yeung, S. K., Tai, Y. W., & Lin, S. (2013). Shading-based shape refinement of RGB-D images. CVPR.

  • Yuille, A., & Snow, D. (1997). Shape and albedo from multiple images using integrability.

  • Zappa, C. J., Banner, M. L., Schultz, H., Corrada-Emmanuel, A., Wolff, L. B., & Yalcin, J. (2008). Retrieval of short ocean wave slope using polarimetric imaging. Measurement Science and Technology, 19(5), 055503.

    Article  Google Scholar 

  • Zhang, L., Curless, B., Hertzmann, A., & Seitz, S. (2003). Shape and motion under varying illumination: Unifying structure from motion, photometric stereo, and multi-view stereo. ICCV.

  • Zhang, L., & Hancock, E. R. (2012). A comprehensive polarisation model for surface orientation recovery. In: 2012 21st International Conference on Pattern Recognition (ICPR) (pp. 3791–3794). IEEE.

  • Zhang, Q., Ye, M., Yang, R., Matsushita, Y., Wilburn, B., & Yu, H. (2012). Edge-preserving photometric stereo via depth fusion. CVPR.

  • Zhao, Y., Yi, C., Kong, S. G., Pan, Q., & Cheng, Y. (2016). Multi-band polarization imaging and applications. Springer Berlin Heidelberg. doi:10.1007/978-3-662-49373-1.

  • Zickler, T., Ramamoorthi, R., Enrique, S., & Belhumeur, P. N. (2006). Reflectance sharing: Predicting appearance from a sparse set of images of a known shape. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28, 1–16.

  • Zickler, T. E., Belhumeur, P. N., & Kriegman, D. J. (2002). Helmholtz stereopsis: Exploiting reciprocity for surface reconstruction. International Journal of Computer Vision, 49(2–3), 215–227.

    Article  MATH  Google Scholar 

Download references

Acknowledgements

The authors thank G. Atkinson, T. Boult, S. Izadi, D. Miyazaki, G. Satat, N. Naik, I.K. Park, H. Zhao for valuable feedback. Achuta Kadambi is supported by the Charles Draper Doctoral Fellowship and the Qualcomm Innovation Fellowship. Boxin Shi is supported by a project commissioned by the New Energy and Industrial Technology Development Organization (NEDO). The work of the MIT-affiliated coauthors was supported by the Media Lab Consortium members.

Author information

Authors and Affiliations

  1. MIT Media Lab, 75 Amherst Street, Cambridge, MA, 02139, USA

    Achuta Kadambi, Vage Taamazyan, Boxin Shi & Ramesh Raskar

  2. Skoltech-MIT Initiative, Moscow Institute of Physics and Technology, Moscow, Russia

    Vage Taamazyan

  3. Tardis 3D Technologies LLC, Moscow, Russia

    Vage Taamazyan

  4. Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology (AIST), Tokyo, Japan

    Boxin Shi

Authors
  1. Achuta Kadambi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vage Taamazyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Boxin Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ramesh Raskar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Achuta Kadambi or Boxin Shi.

Additional information

Communicated by Rene Vidal, Katsushi Ikeuchi, Josef Sivic and Christoph Schnoerr.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kadambi, A., Taamazyan, V., Shi, B. et al. Depth Sensing Using Geometrically Constrained Polarization Normals. Int J Comput Vis 125, 34–51 (2017). https://doi.org/10.1007/s11263-017-1025-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-017-1025-7

Keywords

Search

Navigation