Abstract
Given a single outdoor image, we present a method for estimating the likely illumination conditions of the scene. In particular, we compute the probability distribution over the sun position and visibility. The method relies on a combination of weak cues that can be extracted from different portions of the image: the sky, the vertical surfaces, the ground, and the convex objects in the image. While no single cue can reliably estimate illumination by itself, each one can reinforce the others to yield a more robust estimate. This is combined with a data-driven prior computed over a dataset of 6 million photos. We present quantitative results on a webcam dataset with annotated sun positions, as well as quantitative and qualitative results on consumer-grade photographs downloaded from Internet. Based on the estimated illumination, we show how to realistically insert synthetic 3-D objects into the scene, and how to transfer appearance across images while keeping the illumination consistent.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Basri, R., Jacobs, D., & Kemelmacher, I. (2007). Photometric stereo with general, unknown lighting. International Journal of Computer Vision, 72(3), 239–257.
Bird, R. E. (1984). A simple spectral model for direct normal and diffuse horizontal irradiance. Solar Energy, 32, 461–471.
Bitouk, D., Kumar, N., Dhillon, S., Belhumeur, P. N., & Nayar, S. K. (2008). Face swapping: automatically replacing faces in photographs. ACM Transactions on Graphics (SIGGRAPH 2008), 27(3), 39:1–39:8.
Blinn, J. F., & Newell, M. E. (1976). Texture and reflection in computer generated images. In Proceedings of ACM SIGGRAPH 1976.
Buluswar, S. D., & Draper, B. A. (2002). Color models for outdoor machine vision. Computer Vision and Image Understanding, 85(2), 71–99.
Cavanagh, P. (2005). The artist as neuroscientist. Nature, 434, 301–307.
Chang, C.-C., & Lin, C.-J. (2001). Libsvm: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/cjlin/libsvm.
Chen, H., Belhumeur, P., & Jacobs, D. (2000). In search of illumination invariants. In IEEE conference on computer vision and pattern recognition.
Chong, H. Y., Gortler, S. J., & Zickler, T. (2008). A perception-based color space for illumination-invariant image processing. ACM Transactions on Graphics (SIGGRAPH 2008) (pp. 61:1–61:7).
Collins, M., Shapire, R., & Singer, Y. (2002). Logistic regression, adaboost and Bregman distances. Machine Learning, 48(1), 158–169.
Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In IEEE conference on computer vision and pattern recognition.
Dale, K., Johnson, M. K., Sunkavalli, K., Matusik, W., & Pfister, H. (2009). Image restoration using online photo collections. In International conference on computer vision.
Debevec, P. (1998). Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proceedings of ACM SIGGRAPH 1998.
Debevec, P., & Malik, J. (1997). Recovering high dynamic range radiance maps from photographs. In Proceedings of ACM SIGGRAPH 1997, August.
Debevec, P., Tchou, C., Gardner, A., Hawkins, T., Poullis, C., Stumpfel, J., Jones, A., Yun, N., Einarsson, P., Lundgren, T., Fajardo, M., & Martinez, P. (2004). Estimating surface reflectance properties of a complex scene under captured natural illumination (Technical Report ICT-TR-06.2004, USC ICT).
Dollár, P., Wojek, C., Schiele, B., & Perona, P. (2009). Pedestrian detection: a benchmark. In IEEE conference on computer vision and pattern recognition.
Dror, R. O., Willsky, A. S., & Adelson, E. H. (2004). Statistical characterization of real-world illumination. Journal of Vision, 4, 821–837.
Felzenszwalb, P., Girshick, R. B., McAllester, D., & Ramanan, D. (2010). Object detection with discriminatively trained part based models. IEEE Transactions on Pattern Analysis and Machine Intelligence (pp. 1627–1645).
Finlayson, G., Drew, M., & Funt, B. (1993). Diagonal transforms suffice for color constancy. In International conference on computer vision.
Finlayson, G. D., Drew, M. S., & Lu, C. (2004). Intrinsic images by entropy minimization. In European conference on computer vision.
Finlayson, G. D., Fredembach, C., & Drew, M. S. (2007). Detecting illumination in images. In IEEE international conference on computer vision.
Finlayson, G. D., Hordley, S. D., & Drew, M. S. (2002). Removing shadows from images. In European conference on computer vision.
Hays, J., & Efros, A. A. (2008). im2gps: estimating geographic information from a single image. In IEEE conference on computer vision and pattern recognition.
Healey, G., & Slater, D. (1999). Models and methods for automated material identification in hyperspectral imagery acquired under unknown illumination and atmospheric conditions. IEEE Transactions on Geoscience and Remote Sensing, 37(6), 2706–2717.
Hill, R. (1994). Theory of geolocation by light levels. In B. J. LeBouef & R. M. Laws (Eds.), Elephant seals: population ecology, behavior, and physiology (pp. 227–236). Berkeley: University of California Press. Chapter 12.
Hoiem, D., Efros, A. A., & Hebert, M. (2005). Automatic photo pop-up. ACM Transactions on Graphics (SIGGRAPH 2005), 24(3), 577–584.
Hoiem, D., Efros, A. A., & Hebert, M. (2007a). Recovering surface layout from an image. International Journal of Computer Vision, 75(1), 151–172.
Hoiem, D., Stein, A., Efros, A. A., & Hebert, M. (2007b). Recovering occlusion boundaries from a single image. In IEEE international conference on computer vision.
Jacobs, N., Roman, N., & Pless, R. (2007). Consistent temporal variations in many outdoor scenes. In IEEE conference on computer vision and pattern recognition.
Judd, D. B., Macadam, D. L., Wyszecki, G., Budde, H. W., Condit, H. R., Henderson, S. T., & Simonds, J. L. (1964). Spectral distribution of typical daylight as a function of correlated color temperature. Journal of the Optical Society of America A, 54(8), 1031–1036.
Junejo, I. N., & Foroosh, H. (2008). Estimating geo-temporal location of stationary cameras using shadow trajectories. In European conference on computer vision.
Kasten, F., & Young, A. T. (1989). Revised optical air mass tables and approximation formula. Applied Optics, 28, 4735–4738.
Ke, Y., Tang, X., & Jing, F. (2006). The design of high-level features for photo quality assessment. In IEEE conference on computer vision and pattern recognition.
Khan, E. A., Reinhard, E., Fleming, R., & Büelthoff, H. (2006). Image-based material editing. ACM Transactions on Graphics (ACM SIGGRAPH 2006), August (pp. 654–663).
Kim, T., & Hong, K.-S. (2005). A practical single image based approach for estimating illumination distribution from shadows. In IEEE international conference on computer vision.
Koenderink, J. J., van Doorn, A. J., & Pont, S. C. (2004). Light direction from shad(ow)ed random gaussian surfaces. Perception, 33(12), 1405–1420.
Košecká, J., & Zhang, W. (2002). Video compass. In European conference on computer vision.
Lalonde, J.-F., Efros, A. A., & Narasimhan, S. G. (2009). Estimating natural illumination from a single outdoor image. In IEEE international conference on computer vision.
Lalonde, J.-F., Efros, A. A., & Narasimhan, S. G. (2009). Webcam clip art: Appearance and illuminant transfer from time-lapse sequences. ACM Transactions on Graphics (SIGGRAPH Asia 2009), 28(5), December (pp. 131:1–131:10).
Lalonde, J.-F., Efros, A. A., & Narasimhan, S. G. (2010). Detecting ground shadows in outdoor consumer photographs. In European conference on computer vision.
Lalonde, J.-F., Efros, A. A., & Narasimhan, S. G. (2010). Ground shadow boundary dataset. http://graphics.cs.cmu.edu/projects/shadows, September.
Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., & Criminisi, A. (2007). Photo clip art. ACM Transactions on Graphics (SIGGRAPH 2007).
Lalonde, J.-F., Narasimhan, S. G., & Efros, A. A. (2010). What do the sun and the sky tell us about the camera? International Journal of Computer Vision, 88(1), 24–51.
Langer, M. S., & Büelthoff, H. H. (2001). A prior for global convexity in local shape-from-shading. Perception, 30(4), 403–410.
Li, Y., Lin, S., Lu, H., & Shum, H.-Y. (2003). Multiple-cue illumination estimation in textured scenes. In IEEE international conference on computer vision.
Manduchi, R. (2006). Learning outdoor color classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(11), 1713–1723.
Maxwell, B. A., Friedhoff, R. M., & Smith, C. A. (2008). A bi-illuminant dichromatic reflection model for understanding images. In IEEE conference on computer vision and pattern recognition.
Mills, D. (2004). Advances in solar thermal electricity and technology. Solar Energy, 76, 19–31.
Narasimhan, S. G., Ramesh, V., & Nayar, S. K. (2005). A class of photometric invariants: Separating material from shape and illumination. In IEEE international conference on computer vision.
Park, D., Ramanan, D., & Fowlkes, C. C. (2010). Multiresolution models for object detection. In European conference on computer vision.
Perez, R., Seals, R., & Michalsky, J. (1993). All-weather model for sky luminance distribution—preliminary configuration and validation. Solar Energy, 50(3), 235–245.
Platt, J. C. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers.
Preetham, A. J., Shirley, P., & Smits, B. (1999). A practical analytic model for daylight. In Proceedings of ACM SIGGRAPH 1999, August.
Ramamoorthi, R., & Hanrahan, P. (2001). A signal-processing framework for inverse rendering. In Proceedings of ACM SIGGRAPH 2001.
Reda, I., & Andreas, A. (2005). Solar position algorithm for solar radiation applications (Technical Report NREL/TP-560-34302). National Renewable Energy Laboratory, November.
Reinhart, C. F., Mardaljevic, J., & Rogers, Z. (2006). Dynamic daylight performance metrics for sustainable building design. Leukos, 3(1), 1–25.
Romeiro, F., & Zickler, T. (2010). Blind reflectometry. In European conference on computer vision.
Russell, B. C., Torralba, A., Murphy, K. P., & Freeman, W. T. (2008). LabelMe: a database and web-based tool for image annotation. International Journal of Computer Vision, 77(1–3), 157–173.
Sato, I., Sato, Y., & Ikeuchi, K. (2003). Illumination from shadows. IEEE Transactions on Pattern Matching and Machine Intelligence, 25(3), 290–300.
Sato, Y., & Ikeuchi, K. (1995). Reflectance analysis under solar illumination. In Proceedings of the IEEE workshop on physics-based modeling and computer vision (pp. 180–187).
Slater, D., & Healey, G. (1998). Analyzing the spectral dimensionality of outdoor visible and near-infrared illumination functions. Journal of the Optical Society of America, 15(11), 2913–2920.
Stauffer, C. (1999). Adaptive background mixture models for real-time tracking. In IEEE conference on computer vision and pattern recognition.
Stumpfel, J., Jones, A., Wenger, A., Tchou, C., Hawkins, T., & Debevec, P. (2004). Direct HDR capture of the sun and sky. In Proceedings of AFRIGRAPH.
Sun, M., Schindler, G., Turk, G., & Dellaert, F. (2009). Color matching and illumination estimation for urban scenes. In IEEE international workshop on 3-D digital imaging and modeling.
Sunkavalli, K., Romeiro, F., Matusik, W., Zickler, T., & Pfister, H. (2008). What do color changes reveal about an outdoor scene. In IEEE conference on computer vision and pattern recognition.
Tian, J., Sun, J., & Tang, Y. (2009). Tricolor attenuation model for shadow detection. IEEE Transactions on Image Processing, 18(10), 2355–2363.
Tsin, Y., Collins, R. T., Ramesh, V., & Kanade, T. (2001). Bayesian color constancy for outdoor object recognition. In IEEE conference on computer vision and pattern recognition.
Ward, G. (1994). The RADIANCE lighting simulation and rendering system. In Proceedings of ACM SIGGRAPH 1994.
Weiss, Y. (2001). Deriving intrinsic images from image sequences. In IEEE international conference on computer vision.
Wu, T.-P., & Tang, C.-K. (2005). A Bayesian approach for shadow extraction from a single image. In IEEE international conference on computer vision.
Yu, Y., & Malik, J. (1998). Recovering photometric properties of architectural scenes from photographs. In Proceedings of ACM SIGGRAPH 1998, July.
Zhu, J., Samuel, K. G. G., Masood, S. Z., & Tappen, M. F. (2010). Learning to recognize shadows in monochromatic natural images. In IEEE conference on computer vision and pattern recognition.
Zickler, T., Mallick, S. P., Kriegman, D. J., & Belhumeur, P. N. (2006). Color subspaces as photometric invariants. In IEEE conference on computer vision and pattern recognition.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lalonde, JF., Efros, A.A. & Narasimhan, S.G. Estimating the Natural Illumination Conditions from a Single Outdoor Image. Int J Comput Vis 98, 123–145 (2012). https://doi.org/10.1007/s11263-011-0501-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-011-0501-8