Skip to main content
SpringerLink
Log in

Spatiotemporal Outdoor Lighting Aggregation on Image Sequences

  • Conference paper
  • First Online:
Pattern Recognition (DAGM GCPR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 13024))

Included in the following conference series:

Abstract

In this work, we focus on outdoor lighting estimation by aggregating individual noisy estimates from images, exploiting the rich image information from wide-angle cameras and/or temporal image sequences. Photographs inherently encode information about the scene’s lighting in the form of shading and shadows. Whereas computer graphic (CG) methods target accurately reproducing the image formation process knowing the exact lighting in the scene, the inverse rendering is an ill-posed problem attempting to estimate the geometry, material, and lighting behind a recorded 2D picture. Recent work based on deep neural networks has shown promising results for single image lighting estimation despite its difficulty. However, the main challenge remains on the stability of measurements. We tackle this problem by combining lighting estimates from many image views sampled in the angular and temporal domain of an image sequence. Thereby, we make efficient use of the camera calibration and camera ego-motion estimation to globally register the individual estimates and apply outlier removal and filtering algorithms. Our method not only improves the stability for rendering applications like virtual object augmentation but also shows higher accuracy for single image based lighting estimation compared to the state-of-the-art.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Balcı, H., Güdükbay, U.: Sun position estimation and tracking for virtual object placement in time-lapse videos. SIViP 11(5), 817–824 (2017)

    Article  Google Scholar 

  2. Comaniciu, D., Meer, P.: Mean shift: a robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 603–619 (2002)

    Article  Google Scholar 

  3. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361. IEEE (2012)

    Google Scholar 

  4. Godard, C., Mac Aodha, O., Firman, M., Brostow, G.J.: Digging into self-supervised monocular depth estimation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3828–3838 (2019)

    Google Scholar 

  5. Hold-Geoffroy, Y., Athawale, A., Lalonde, J.F.: Deep sky modeling for single image outdoor lighting estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6927–6935 (2019)

    Google Scholar 

  6. Hold-Geoffroy, Y., Sunkavalli, K., Hadap, S., Gambaretto, E., Lalonde, J.F.: Deep outdoor illumination estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7312–7321 (2017)

    Google Scholar 

  7. Hosek, L., Wilkie, A.: An analytic model for full spectral sky-dome radiance. ACM Trans. Graph. (TOG) 31(4), 1–9 (2012)

    Article  Google Scholar 

  8. Jin, X., et al.: Sun-sky model estimation from outdoor images. J. Ambient Intell. Hum. Comput. 1–12 (2020). https://doi.org/10.1007/s12652-020-02367-3

  9. Jin, X., et al.: Sun orientation estimation from a single image using short-cuts in DCNN. Optics Laser Technol. 110, 191–195 (2019)

    Article  Google Scholar 

  10. Kajiya, J.T.: The rendering equation. In: Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques, pp. 143–150 (1986)

    Google Scholar 

  11. Kán, P., Kaufmann, H.: Deeplight: light source estimation for augmented reality using deep learning. Vis. Comput. 35(6–8), 873–883 (2019)

    Article  Google Scholar 

  12. Karsch, K., Hedau, V., Forsyth, D., Hoiem, D.: Rendering synthetic objects into legacy photographs. ACM Trans. Graph. (TOG) 30(6), 1–12 (2011)

    Article  Google Scholar 

  13. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  14. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

    Google Scholar 

  15. Lalonde, J.F., Efros, A.A., Narasimhan, S.G.: Estimating the natural illumination conditions from a single outdoor image. Int. J. Comput. Vis. 98(2), 123–145 (2012)

    Article  MathSciNet  Google Scholar 

  16. Lalonde, J.F., Matthews, I.: Lighting estimation in outdoor image collections. In: 2014 2nd International Conference on 3D Vision, vol. 1, pp. 131–138. IEEE (2014)

    Google Scholar 

  17. Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013)

  18. Liu, F.T., Ting, K.M., Zhou, Z.H.: Isolation forest. In: 2008 Eighth IEEE International Conference on Data Mining, pp. 413–422. IEEE (2008)

    Google Scholar 

  19. Liu, Y., Granier, X.: Online tracking of outdoor lighting variations for augmented reality with moving cameras. IEEE Trans. Visual Comput. Graph. 18(4), 573–580 (2012)

    Article  Google Scholar 

  20. Lu, B.V., Kakuta, T., Kawakami, R., Oishi, T., Ikeuchi, K.: Foreground and shadow occlusion handling for outdoor augmented reality. In: 2010 IEEE International Symposium on Mixed and Augmented Reality, pp. 109–118. IEEE (2010)

    Google Scholar 

  21. Ma, W.C., Wang, S., Brubaker, M.A., Fidler, S., Urtasun, R.: Find your way by observing the sun and other semantic cues. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 6292–6299. IEEE (2017)

    Google Scholar 

  22. Madsen, C.B., Lal, B.B.: Outdoor illumination estimation in image sequences for augmented reality. GRAPP 11, 129–139 (2011)

    Google Scholar 

  23. Madsen, C.B., Störring, M., Jensen, T., Andersen, M.S., Christensen, M.F.: Real-time illumination estimation from image sequences. In: Proceedings: 14th Danish Conference on Pattern Recognition and Image Analysis, Copenhagen, Denmark, pp. 1–9 (2005)

    Google Scholar 

  24. Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)

    MathSciNet  MATH  Google Scholar 

  25. Reda, I., Andreas, A.: Solar position algorithm for solar radiation applications. Sol. Energy 76(5), 577–589 (2004)

    Article  Google Scholar 

  26. Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4104–4113 (2016)

    Google Scholar 

  27. Van Dijk, T., de Croon, G.C.H.E.: How do neural networks see depth in single images? In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2183–2191 (2019)

    Google Scholar 

  28. Wei, H., Liu, Y., Xing, G., Zhang, Y., Huang, W.: Simulating shadow interactions for outdoor augmented reality with RGBD data. IEEE Access 7, 75292–75304 (2019)

    Article  Google Scholar 

  29. Whelan, T., Salas-Moreno, R.F., Glocker, B., Davison, A.J., Leutenegger, S.: Elasticfusion: real-time dense slam and light source estimation. Int. J. Robot. Res. 35(14), 1697–1716 (2016)

    Article  Google Scholar 

  30. Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: CBAM: convolutional block attention module. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 3–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_1

    Chapter  Google Scholar 

  31. Xiao, J., Ehinger, K.A., Oliva, A., Torralba, A.: Recognizing scene viewpoint using panoramic place representation. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2695–2702. IEEE (2012)

    Google Scholar 

  32. Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. arXiv preprint arXiv:1611.05431 (2016)

  33. Xiong, Y., Chen, H., Wang, J., Zhu, Z., Zhou, Z.: DSNet: deep shadow network for illumination estimation. In: 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pp. 179–187. IEEE (2021)

    Google Scholar 

  34. Zhang, J., Sunkavalli, K., Hold-Geoffroy, Y., Hadap, S., Eisenman, J., Lalonde, J.F.: All-weather deep outdoor lighting estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10158–10166 (2019)

    Google Scholar 

  35. Zhang, K., Li, X., Jin, X., Liu, B., Li, X., Sun, H.: Outdoor illumination estimation via all convolutional neural networks. Comput. Electr. Eng. 90, 106987 (2021)

    Article  Google Scholar 

  36. Zhu, Y., Zhang, Y., Li, S., Shi, B.: Spatially-varying outdoor lighting estimation from intrinsics. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12834–12842 (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Vision Research Lab, Robert Bosch GmbH, Hildesheim, Germany

    Haebom Lee & Robert Herzog

  2. Computer Vision and Learning Lab, Heidelberg University, Heidelberg, Germany

    Haebom Lee & Carsten Rother

  3. Campus Minden, Bielefeld University of Applied Sciences, Minden, Germany

    Jan Rexilius

Authors
  1. Haebom Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert Herzog
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jan Rexilius
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Carsten Rother
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Fraunhofer IAIS, Sankt Augustin, Germany

    Christian Bauckhage

  2. University of Bonn, Bonn, Germany

    Juergen Gall

  3. University of Illinois at Urbana-Champaign, Urbana, IL, USA

    Alexander Schwing

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lee, H., Herzog, R., Rexilius, J., Rother, C. (2021). Spatiotemporal Outdoor Lighting Aggregation on Image Sequences. In: Bauckhage, C., Gall, J., Schwing, A. (eds) Pattern Recognition. DAGM GCPR 2021. Lecture Notes in Computer Science(), vol 13024. Springer, Cham. https://doi.org/10.1007/978-3-030-92659-5_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-92659-5_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-92658-8

  • Online ISBN: 978-3-030-92659-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation