Abstract
We present an end-to-end network for spatially-varying outdoor lighting estimation in urban scenes given a single limited field-of-view LDR image and any assigned 2D pixel position. We use three disentangled latent spaces learned by our network to represent sky light, sun light, and lighting-independent local contents respectively. At inference time, our lighting estimation network can run efficiently in an end-to-end manner by merging the global lighting and the local appearance rendered by the local appearance renderer with the predicted local silhouette. We enhance an existing synthetic dataset with more realistic material models and diverse lighting conditions for more effective training. We also capture the first real dataset with HDR labels for evaluating spatially-varying outdoor lighting estimation. Experiments on both synthetic and real datasets show that our method achieves state-of-the-art performance with more flexible editability.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Errors are calculated on the masked region by the (predicted) local silhouette masks.
- 2.
We calculate rendering errors on pixels belonging to the bunny body.
References
ambientCG, public domain materials for physically based rendering. [licensed under CC0 1.0 universal]. https://ambientcg.com
Blender SceneCity. https://www.cgchan.com/store/scenecity
Poly Haven, The Public 3D Asset Library. [licensed under CC0 1.0 universal]. https://polyhaven.com
The Stanford 3D Scanning Repository. http://graphics.stanford.edu/data/3Dscanrep/
Barron, J.T., Malik, J.: Intrinsic scene properties from a single RGB-D image. In: Proceedings of Computer Vision and Pattern Recognition (2013)
Burley, B.: Physically-based shading at Disney. In: SIGGRAPH Course: Practical Physically Based Shading in Film and Game Production (2012)
Cheng, D., Shi, J., Chen, Y., Deng, X., Zhang, X.: Learning scene illumination by pairwise photos from rear and front mobile cameras. Comput. Graph. Forum, 37(7), 213–221 (2018)
Debevec, P.: Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In: Proceedings of ACM SIGGRAPH (1998)
Gardner, M.A., Hold-Geoffroy, Y., Sunkavalli, K., Gagné, C., Lalonde, J.F.: Deep parametric indoor lighting estimation. In: Proceedings of International Conference on Computer Vision (2019)
Garon, M., Sunkavalli, K., Hadap, S., Carr, N., Lalonde, J.F.: Fast spatially-varying indoor lighting estimation. In: Proceedings of Computer Vision and Pattern Recognition (2019)
Hold-Geoffroy, Y., Athawale, A., Lalonde, J.F.: Deep sky modeling for single image outdoor lighting estimation. In: Proceedings of Computer Vision and Pattern Recognition (2019)
Hold-Geoffroy, Y., Sunkavalli, K., Hadap, S., Gambaretto, E., Lalonde, J.F.: Deep outdoor illumination estimation. In: Proceedings of Computer Vision and Pattern Recognition (2017)
Hosek, L., Wilkie, A.: An analytic model for full spectral sky-dome radiance. ACM Trans. Graph. (TOG) 31(4), 1–9 (2012)
Hosek, L., Wilkie, A.: Adding a solar-radiance function to the Hošek-Wilkie skylight model. IEEE Comput. Graphics Appl. 33(3), 44–52 (2013)
Lalonde, J.F., Efros, A.A., Narasimhan, S.G.: Estimating the natural illumination conditions from a single outdoor image. Int. J. Comput. Vision 98(2), 123–145 (2012)
Lalonde, J.F., et al.: The laval HDR sky database. [free license for academic or government-sponsored researchers] (2016). http://sky.hdrdb.com
Lalonde, J.F., Matthews, I.: Lighting estimation in outdoor image collections. In: Proceedings of International Conference on 3D Vision (2014)
LeGendre, C., et al.: DeepLight: learning illumination for unconstrained mobile mixed reality. In: Proceedings of Computer Vision and Pattern Recognition (2019)
Li, Z., Shafiei, M., Ramamoorthi, R., Sunkavalli, K., Chandraker, M.: Inverse rendering for complex indoor scenes: shape, spatially-varying lighting and SVBRDF from a single image. In: Proceedings of Computer Vision and Pattern Recognition (2020)
Newell, A., Yang, K., Deng, J.: Stacked hourglass networks for human pose estimation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 483–499. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46484-8_29
Ramamoorthi, R., Hanrahan, P.: An efficient representation for irradiance environment maps. In: Proceedings of ACM SIGGRAPH (2001)
Saito, S., Huang, Z., Natsume, R., Morishima, S., Kanazawa, A., Li, H.: PIFu: pixel-aligned implicit function for high-resolution clothed human digitization. In: Proceedings of International Conference on Computer Vision (2019)
Song, S., Funkhouser, T.: Neural illumination: lighting prediction for indoor environments. In: Proceedings of Computer Vision and Pattern Recognition (2019)
Srinivasan, P.P., Mildenhall, B., Tancik, M., Barron, J.T., Tucker, R., Snavely, N.: Lighthouse: predicting lighting volumes for spatially-coherent illumination. In: Proceedings of Computer Vision and Pattern Recognition (2020)
Stumpfel, J., Jones, A., Wenger, A., Tchou, C., Hawkins, T., Debevec, P.: Direct HDR capture of the sun and sky. In: Proceedings of ACM SIGGRAPH (2004)
Yu, P., et al.: Hierarchical disentangled representation learning for outdoor illumination estimation and editing. In: Proceedings of International Conference on Computer Vision (2021)
Zhang, J., Sunkavalli, K., Hold-Geoffroy, Y., Hadap, S., Eisenmann, J., Lalonde, J.F.: All-weather deep outdoor lighting estimation. In: Proceedings of Computer Vision and Pattern Recognition (2019)
Zhu, Y., Zhang, Y., Li, S., Shi, B.: Spatially-varying outdoor lighting estimation from intrinsics. In: Proceedings of Computer Vision and Pattern Recognition (2021)
Acknowledgements
This work is supported by National Natural Science Foundation of China under Grant No. 62136001, 61872012.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Tang, J., Zhu, Y., Wang, H., Chan, J.H., Li, S., Shi, B. (2022). Estimating Spatially-Varying Lighting in Urban Scenes with Disentangled Representation. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13666. Springer, Cham. https://doi.org/10.1007/978-3-031-20068-7_26
Download citation
DOI: https://doi.org/10.1007/978-3-031-20068-7_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20067-0
Online ISBN: 978-3-031-20068-7
eBook Packages: Computer ScienceComputer Science (R0)