Skip to main content
SpringerLink
Log in

One-Day Outdoor Photometric Stereo Using Skylight Estimation

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

A Correction to this article was published on 28 February 2019

This article has been updated

Abstract

We present an outdoor photometric stereo method using images captured in a single day. We simulate a sky hemisphere for each image according to its GPS and timestamp and parameterize the obtained sky hemisphere into quadratic skylight and Gaussian sunlight distributions. Our previous work recovered an outdoor scene on a clear day, whereas the current paper shows that cloudy days can provide better illumination conditions for surface orientation recovery, and hence we propose a modified sky model to represent a well-conditioned skylight distribution for outdoor photometric stereo. The proposed method models natural illumination according to a sky model, providing sufficient constraints for shape reconstruction from 1-day images. We tested the proposed method to recover various sized objects and scenes from real-world outdoor daylight images and verified the method using synthetic and real data experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24

Similar content being viewed by others

Change history

  • 28 February 2019

    The original version of this article unfortunately contained mistakes in the figure captions.

References

  • Abrams, A., Hawley, C., & Pless, R. (2012). Heliometric stereo: Shape from sun position. In Proceedings of European conference on computer vision (ECCV).

  • Abrams, A., Hawley, C., Miskell, K., Stoica, A., Jacobs, N., & Pless, R. (2013a). Shadow estimation method for “the episolar constraint: Monocular shape from shadow correspondence”. arXiv:1304.4112.

  • Abrams, A., Miskell, K., & Pless, R. (2013b). The episolar constraint: Monocular shape from shadow correspondence. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).

  • Ackermann, J., Langguth, F., Fuhrmann, S., & Goesele, M. (2012). Photometric stereo for outdoor webcams. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).

  • Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., & Seitz, S. M. (2011). Building Rome in a day. Communications of the ACM, 54(10), 105–112.

    Article  Google Scholar 

  • Furukawa, Y., & Ponce, J. (2010). Accurate, dense, and robust multi-view stereopsis. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 32(8), 1362–1376.

    Article  Google Scholar 

  • Google Earth. (2015). https://earth.google.com.

  • Grossmann, P. (1987). Depth from focus. Pattern Recognition Letters, 5(1), 63–69.

    Article  MathSciNet  Google Scholar 

  • Han, Y., Lee, J. Y., & Kweon, I. S. (2013). High quality shape from a single RGB-D image under uncalibrated natural illumination. In Proceedings of international conference on computer vision (ICCV).

  • Hold-Geoffroy, Y., Zhang, J., Gotardo, P. F. U., & LaLonde, J. F. (2015a). What is a good day for outdoor photometric stereo? In International conference on computational photography.

  • Hold-Geoffroy, Y., Zhang, J., Gotardo, P. F. U., & LaLonde, J. F. (2015b). x-hour outdoor photometric stereo. In International conference on 3D vision.

  • Hold-Geoffroy, Y., Sunkavalli, K., Hadap, S., Gambaretto, E., & LaLonde, J. F. (2017). Deep outdoor illumination estimation. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).

  • Horn, B. K. P. (1989). Obtaining shape from shading information. In Shape from shading (pp. 123–171). Cambridge, MA, USA: MIT Press.

  • Hošek, L., & Wilkie, A. (2013). Adding a solar radiance function to the Hošek–Wilkie skylight model. IEEE Computer Graphics and Applications, 33(3), 44–52.

    Article  Google Scholar 

  • Huang, R., & Smith, W. A. P. (2011). Shape-from-shading under complex natural illumination. In Proceedings of international conference on image processing (ICIP).

  • Hung, C. H., Wu, T. P., Matsushita, Y., Xu, L., Jia, J., & Tang, C. K. (2015). Photometric stereo in the wild. In Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV). Waikoloa, HI: IEEE.

  • Inose, K., Shimizu, S., Kawakami, R., Mukaigawa, Y., & Ikeuchi, K. (2013). Refining outdoor photometric stereo based on sky model. IPSJ Transaction on Computer Vision and Applications, 5(1), 104–108.

    Article  Google Scholar 

  • Jacobs, N., Roman, N., & Pless, R. (2007). Consistent temporal variations in many outdoor scenes. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).

  • Johnson, M. K., & Adelson, E. H. (2011). Shape estimation in natural illumination. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).

  • Jung, J. (2016). Outdoor photometric stereo for 3D scene recovery. Ph.D. Thesis, KAIST, Daejeon, Republic of Korea.

  • Jung, J., Lee, J. Y., Jeong, Y., & Kweon, I. S. (2015a). Time-of-flight sensor calibration for a color and depth camera pair. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 37(7), 1501–1513.

    Article  Google Scholar 

  • Jung, J., Lee, J. Y., & Kweon, I. S. (2015b). One-day outdoor photometric stereo via skylight estimation. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).

  • Kawakami, R., Zhao, H., Tan, R. T., & Ikeuchi, K. (2013). Camera spectral sensitivity and white balance estimation from sky images. International Journal on Computer Vision (IJCV), 105(3), 187–204.

    Article  MATH  Google Scholar 

  • Koenderink, J. J., van Doorn, A. J., & Kappers, A. M. L. (1992). Surface perception in pictures. Perception and Psychophysics, 52(5), 487–496.

    Article  Google Scholar 

  • Kolmogorov, V., & Zabih, R. (2004). What energy functions can be minimized via graph cuts? IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 26(2), 147–159.

    Article  MATH  Google Scholar 

  • Lalonde, J. F., Narasimhan, S. G., & Efros, A. A. (2010). What do the sun and the sky tell us about the camera? International Journal on Computer Vision (IJCV), 88(1), 24–51.

    Article  Google Scholar 

  • Lalonde, J. F., Efros, A. A., & Narasimhan, S. G. (2012). Estimating natural illumination from a single outdoor image. International Journal on Computer Vision (IJCV), 98(2), 123–145.

    Article  MathSciNet  Google Scholar 

  • Lalonde, J. F., Asselin, L. P., Becirovski, J., Hold-Geoffroy, Y., Garon, M., Gardner, M. A., & Zhang, J. (2016). The Laval HDR sky database. http://www.hdrdb.com. Accessed 1 October 2015.

  • Lee, J. Y., Matsushita, Y., Shi, B., & Kweon, I. S. (2013). Radiometric calibration by rank minimization. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 35(1), 114–156.

    Article  Google Scholar 

  • Lombardi, S., & Nishino, K. (2012). Reflectance and illumination from a single image. In Proceedings of European conference on computer vision (ECCV).

  • Lu, F., Matsushita, Y., Sato, I., Okabe, T., & Sato, Y. (2013). Uncalibrated photometric stereo for unknown isotropic reflectances. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).

  • Okutomi, M., & Kanade, T. (1993). A multiple-baseline stereo system. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 15(4), 353–363.

    Article  Google Scholar 

  • Oxholm, G., & Nishino, K. (2012). Shape and reflectance from natural illumination. In Proceedings of European conference on computer vision (ECCV).

  • Preetham, A. J., Shirley, P., & Smits, B. (1999). A practical analytic model for daylight. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques (SIGGRAPH ’99) (pp. 91–100). ACM Press/Addison-Wesley Publishing Co.

  • Ramamoorthi, R., & Hanrahan, P. (2001). On the relationship between radiance and irradiance: Determining the illumination from images of a convex lambertian object. Journal of the Optical Society of America A (JOSA A), 18(10), 2448–2459.

    Article  MathSciNet  Google Scholar 

  • Saff, E., & Kuijlaars, A. B. J. (1997). Distributing many points on a sphere. The Mathematical Intelligencer, 19(1), 5–11.

    Article  MathSciNet  MATH  Google Scholar 

  • Scharstein, D., & Szeliski, R. (2002). A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal on Computer Vision (IJCV), 47(13), 7–42.

    Article  MATH  Google Scholar 

  • Shan, Q., Adams, R., Curless, B., Furukawa, Y., & Seitz, S. M. (2013). The visual turing test for scene reconstruction. In Proceedings of international conference on 3D vision.

  • Shen, F., Sunkavalli, K., Bonneel, N., Rusinkiewicz, S., Pfister, H., & Tong, X. (2014). Time-lapse photometric stereo and applications. Computer Graphics Forum, 33(7), 359–367.

    Article  Google Scholar 

  • Shi, B., Matsushita, Y., Wei, Y., Xu, C., & Tan, P. (2010). Self-calibrating photometric stereo. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).

  • Simchony, T., Chellappa, R., & Shao, M. (1990). Direct analytical methods for solving poisson equations in computer vision problems. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 12(5), 435–446.

    Article  MATH  Google Scholar 

  • Stumpfel, J., Tchou, C., Jones, A., Hawkins, T., Wenger, A., & Debevec, P. (2004). Direct HDR capture of the sun and sky. In Proceedings of the 3rd International Conference on Computer Graphics, Virtual Reality, Visualisation and Interaction in Africa (AFRIGRAPH) (pp. 145–149). ACM.

  • Subbarao, M., & Surya, G. (1994). Depth from defocus: A spatial domain approach. International Journal on Computer Vision (IJCV), 13(3), 271–294.

    Article  Google Scholar 

  • Sunkavalli, K., Matusik, W., Pfister, H., & Rusinkiewicz, S. (2007). Factored time-lapse video. ACM Transaction on Graphics (TOG), 26(3), 101–111.

    Article  Google Scholar 

  • Sunkavalli, K., Romeiro, F., Matusik, W., Zickler, T., & Pfister, H. (2008). What do color changes reveal about an outdoor scene? In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).

  • Turk, G., & Levoy, M. (1994). Zippered polygon meshes from range images. In Proceedings of the 21st annual conference on Computer graphics and interactive techniques (SIGGRAPH ’94) (pp. 311-318). ACM.

  • Woodham, R. (1980). Photometric method for determining surface orientation from multiple images. Optical Engineering, 19(1), 139–144.

    Article  Google Scholar 

  • Yoon, K. J., & Kweon, I. S. (2006). Adaptive support-weight approach for correspondence search. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 28(4), 650–656.

    Article  Google Scholar 

  • Yu, L. F., Yeung, S. K., Tai, Y. W., & Lin, S. (2013). Shading-based shape refinement of RGB-D images. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).

  • Zhang, Q., Ye, M., Yang, R., Matsushita, Y., Wilburn, B., & Yu, H. (2012). Edge-preserving photometric stereo via depth fusion. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR).

Download references

Acknowledgements

This work was supported by a Grant from Kyung Hee University (KHU-20170718) and National Research Foundation of Korea (NRF-2017R1C1B5075945).

Author information

Authors and Affiliations

  1. Department of Software Convergence, Kyung Hee University, Yongin, Republic of Korea

    Jiyoung Jung

  2. Adobe Research, San Jose, CA, USA

    Joon-Young Lee

  3. School of Electrical Engineering, KAIST, Daejeon, Republic of Korea

    In So Kweon

Authors
  1. Jiyoung Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joon-Young Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. In So Kweon
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiyoung Jung.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original version of this article was revised: Figure 15-18 captions was incorrect. It has been corrected.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jung, J., Lee, JY. & Kweon, I.S. One-Day Outdoor Photometric Stereo Using Skylight Estimation. Int J Comput Vis 127, 1126–1142 (2019). https://doi.org/10.1007/s11263-018-01145-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-018-01145-1

Keywords

Search

Navigation