Skip to main content
SpringerLink
Log in

Inverse Lighting with Differentiable Physically-Based Model

  • Conference paper
  • First Online:
Learning and Intelligent Optimization (LION 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14286))

Included in the following conference series:

  • 465 Accesses

Abstract

The design of scene lighting in video games and computer graphics can be a challenging and time-consuming task for lighting artists. Automating the lighting in problems such as stadium lighting design in sports games would help the artists by making this tedious process more efficient. In this work, we explore several practical solutions to this problem via optimization and data-driven models. First, we evaluate evolutionary and swarm intelligence gradient-free algorithms with black-box Physically-Based Rendering (PBR) models. Next, by implementing a differentiable PBR model, we leverage gradients to apply gradient-descent optimization to find an optimal solution. We exploit this differentiable model to develop a data-driven framework to learn the mapping from the illumination field to the lighting parameters via minimizing the loss between the illumination and its reconstructed field using a differentiable PBR decoder. Having the learned model, we directly predict the lighting configuration given a user-defined target illumination. In general, we show that all the mentioned methods can reach acceptable solutions, however, based on the conditions, one method can be preferred among others.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Chen, W., et al.: DIB-R++: learning to predict lighting and material with a hybrid differentiable renderer. In: Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems (2021). https://openreview.net/forum?id=gRqHB07GGz3

  2. Gardner, M.A., Hold-Geoffroy, Y., Sunkavalli, K., Gagne, C., Lalonde, J.F.: Deep parametric indoor lighting estimation. In: The IEEE International Conference on Computer Vision (ICCV) (2019)

    Google Scholar 

  3. Gardner, M.A., et al.: Learning to predict indoor illumination from a single image. ACM Trans. Graph. 36(6), 1–14 (2017). https://doi.org/10.1145/3130800.3130891

    Article  Google Scholar 

  4. Garon, M., Sunkavalli, K., Hadap, S., Carr, N., Lalonde, J.F.: Fast spatially-varying indoor lighting estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  5. Gkaravelis, A., Papaioannou, G.: Inverse lighting design using a coverage optimization strategy. Vis. Comput. 32, 771–780 (2016). https://doi.org/10.1007/s00371-016-1237-9

    Article  Google Scholar 

  6. Gkaravelis, A., Papaioannou, G.: Light optimization for detail highlighting. In: Computer Graphics Forum, vol. 37 (2018)

    Google Scholar 

  7. Hansen, N.: The CMA evolution strategy: a tutorial (2016). https://doi.org/10.48550/ARXIV.1604.00772. https://arxiv.org/abs/1604.00772

  8. Hold-Geoffroy, Y., Athawale, A., Lalonde, J.F.: Deep sky modeling for single image outdoor lighting estimation. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6920–6928 (2019)

    Google Scholar 

  9. Hold-Geoffroy, Y., Sunkavalli, K., Hadap, S., Gambaretto, E., Lalonde, J.F.: Deep outdoor illumination estimation. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2373–2382 (2017). https://doi.org/10.1109/CVPR.2017.255

  10. Kato, H., et al.: Differentiable rendering: a survey (2020). https://doi.org/10.48550/ARXIV.2006.12057. https://arxiv.org/abs/2006.12057

  11. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2015)

    Google Scholar 

  12. Lagarde, S.: SIGGRAPH 2014: moving frostbite to physically based rendering V3 (2014). https://seblagarde.wordpress.com/2015/07/14/siggraph-2014-moving-frostbite-to-physically-based-rendering/. Accessed 09 Mar 2022

  13. Li, Z., Shafiei, M., Ramamoorthi, R., Sunkavalli, K., Chandraker, M.: Inverse rendering for complex indoor scenes: shape, spatially-varying lighting and SVBRDF from a single image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2475–2484 (2020)

    Google Scholar 

  14. Lima, G.F.M., Tavares, J., Peretta, I.S., Yamanaka, K., Cardoso, A., Lamounier, E.: Optimization of lighting design usign genetic algorithms. In: 2010 9th IEEE/IAS International Conference on Industry Applications - INDUSCON 2010, pp. 1–6 (2010). https://doi.org/10.1109/INDUSCON.2010.5740021

  15. Madias, E.N.D., Kontaxis, P.A., Topalis, F.V.: Application of multi-objective genetic algorithms to interior lighting optimization. Energy Build. 125, 66–74 (2016). https://doi.org/10.1016/j.enbuild.2016.04.078. https://www.sciencedirect.com/science/article/pii/S0378778816303553

  16. Meidani, K., Hemmasian, A., Mirjalili, S., Barati Farimani, A.: Adaptive grey wolf optimizer. Neural Comput. Appl. 34(10), 7711–7731 (2022). https://doi.org/10.1007/s00521-021-06885-9

    Article  Google Scholar 

  17. Mirjalili, S., Mirjalili, S.M., Lewis, A.: Grey wolf optimizer. Adv. Eng. Softw. 69, 46–61 (2014). https://doi.org/10.1016/j.advengsoft.2013.12.007. https://www.sciencedirect.com/science/article/pii/S0965997813001853

  18. Nieto, G., Jiddi, S., Robert, P.: Robust point light source estimation using differentiable rendering (2018). https://doi.org/10.48550/ARXIV.1812.04857. https://arxiv.org/abs/1812.04857

  19. Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 32, pp. 8024–8035. Curran Associates, Inc. (2019). http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

  20. Petranović, D.: Stadium reflector aiming using genetic algorithms. In: 2012 Proceedings of the 35th International Convention MIPRO, pp. 1070–1075 (2012)

    Google Scholar 

  21. Salimans, T., Ho, J., Chen, X., Sutskever, I.: Evolution strategies as a scalable alternative to reinforcement learning. ArXiv ArXiv:1703.03864 (2017)

  22. Sengupta, S., Gu, J., Kim, K., Liu, G., Jacobs, D., Kautz, J.: Neural inverse rendering of an indoor scene from a single image. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8597–8606 (2019). https://doi.org/10.1109/ICCV.2019.00869

  23. Song, S., Funkhouser, T.: Neural illumination: lighting prediction for indoor environments. In: Proceedings of 33th IEEE Conference on Computer Vision and Pattern Recognition (2019)

    Google Scholar 

  24. Srinivasan, P.P., Mildenhall, B., Tancik, M., Barron, J.T., Tucker, R., Snavely, N.: Lighthouse: predicting lighting volumes for spatially-coherent illumination. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Google Scholar 

  25. Wang, L.-W., Siu, W.-C., Liu, Z.-S., Li, C.-T., Lun, D.P.K.: Deep relighting networks for image light source manipulation. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020, Part III. LNCS, vol. 12537, pp. 550–567. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-67070-2_33

    Chapter  Google Scholar 

  26. Wang, Z., Philion, J., Fidler, S., Kautz, J.: Learning indoor inverse rendering with 3D spatially-varying lighting. In: Proceedings of International Conference on Computer Vision (ICCV) (2021)

    Google Scholar 

  27. Xiao, H., Fang, J., Zhu, P., Yin, W., Kang, Q.: Energy-saving optimization of football field lighting via genetic algorithm. Sens. Lett. 12, 264–269 (2014). https://doi.org/10.1166/sl.2014.3265

    Article  Google Scholar 

  28. Yu, Y., Smith, W.: InverseRenderNet: learning single image inverse rendering. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3150–3159 (2019)

    Google Scholar 

  29. Zhan, F., et al.: GMLight: lighting estimation via geometric distribution approximation. IEEE Trans. Image Process. 31, 2268–2278 (2022). https://doi.org/10.1109/TIP.2022.3151997

    Article  Google Scholar 

  30. Zhang, E., Cohen, M.F., Curless, B.: Discovering point lights with intensity distance fields. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  31. Zhang, J., Sunkavalli, K., Hold-Geoffroy, Y., Hadap, S., Eisenmann, J., Lalonde, J.F.: All-weather deep outdoor lighting estimation. In: IEEE International Conference on Computer Vision and Pattern Recognition (2019)

    Google Scholar 

  32. Zhang, K., Luan, F., Wang, Q., Bala, K., Snavely, N.: PhySG: inverse rendering with spherical gaussians for physics-based material editing and relighting. In: The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

    Google Scholar 

  33. Zhao, S.Y., Xie, Y.P., Li, W.J.: On the convergence and improvement of stochastic normalized gradient descent. Sci. China Inf. Sci. 64(3), 132103 (2021). https://doi.org/10.1007/s11432-020-3023-7

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Electronic Arts, Redwood City, CA, USA

    Kazem Meidani, Igor Borovikov & Harold Chaput

  2. Department of Mechanical Engineering, Carnegie Mellon University, Pittsburgh, PA, USA

    Kazem Meidani & Amir Barati Farimani

  3. Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA, USA

    Amir Barati Farimani

Authors
  1. Kazem Meidani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Igor Borovikov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amir Barati Farimani
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Harold Chaput
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kazem Meidani .

Editor information

Editors and Affiliations

  1. InsideOpt, Dover, DE, USA

    Meinolf Sellmann

  2. Bielefeld University, Bielefeld, Germany

    Kevin Tierney

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Meidani, K., Borovikov, I., Barati Farimani, A., Chaput, H. (2023). Inverse Lighting with Differentiable Physically-Based Model. In: Sellmann, M., Tierney, K. (eds) Learning and Intelligent Optimization. LION 2023. Lecture Notes in Computer Science, vol 14286. Springer, Cham. https://doi.org/10.1007/978-3-031-44505-7_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-44505-7_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-44504-0

  • Online ISBN: 978-3-031-44505-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation