Abstract
Physically based differentiable rendering has recently evolved into a powerful tool for solving inverse problems involving light. Methods in this area perform a differentiable simulation of the physical process of light transport and scattering to estimate partial derivatives relating scene parameters to pixels in the rendered image. Together with gradient-based optimization, such algorithms have interesting applications in diverse disciplines, e.g., to improve the reconstruction of 3D scenes, while accounting for interreflection and transparency, or to design meta-materials with specified optical properties.
The most versatile differentiable rendering algorithms rely on reverse-mode differentiation to compute all requested derivatives at once, enabling optimization of scene descriptions with millions of free parameters. However, a severe limitation of the reverse-mode approach is that it requires a detailed transcript of the computation that is subsequently replayed to back-propagate derivatives to the scene parameters. The transcript of typical renderings is extremely large, exceeding the available system memory by many orders of magnitude, hence current methods are limited to simple scenes rendered at low resolutions and sample counts.
We introduce radiative backpropagation, a fundamentally different approach to differentiable rendering that does not require a transcript, greatly improving its scalability and efficiency. Our main insight is that reverse-mode propagation through a rendering algorithm can be interpreted as the solution of a continuous transport problem involving the partial derivative of radiance with respect to the optimization objective. This quantity is "emitted" by sensors, "scattered" by the scene, and eventually "received" by objects with differentiable parameters. Differentiable rendering then decomposes into two separate primal and adjoint simulation steps that scale to complex scenes rendered at high resolutions. We also investigated biased variants of this algorithm and find that they considerably improve both runtime and convergence speed. We showcase an efficient GPU implementation of radiative backpropagation and compare its performance and the quality of its gradients to prior work.
Supplemental Material
Available for Download
- Joel Andersson. 2013. A general-purpose software framework for dynamic optimization. Ph.D. Dissertation. KU Leuven.Google Scholar
- James Arvo. 1995. Analytic methods for simulated light transport. Ph.D. Dissertation. Yale University.Google Scholar
- Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner. 2019. Inverse Path Tracing for Joint Material and Lighting Estimation. In Proceedings of Computer Vision and Pattern Recognition (CVPR), IEEE.Google ScholarCross Ref
- Lukas Balles and Philipp Hennig. 2018. Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research), Jennifer Dy and Andreas Krause (Eds.), Vol. 80. PMLR, 404--413.Google Scholar
- Christian Bischof, Alan Carle, George Corliss, Andreas Griewank, and Paul Hovland. 1992. ADIFOR-generating derivative codes from Fortran programs. Scientific Programming 1, 1 (1992), 11--29.Google ScholarDigital Library
- Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2018. Inverse Transport Networks. arXiv preprint arXiv:1809.10820 (2018).Google Scholar
- Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. 2018. Neural ordinary differential equations. In Advances in neural information processing systems. 6571--6583.Google Scholar
- Oskar Elek, Denis Sumin, Ran Zhang, Tim Weyrich, Karol Myszkowski, Bernd Bickel, Alexander Wilkie, and Jaroslav Křivánek. 2017. Scattering-aware Texture Reproduction for 3D Printing. ACM Transactions on Graphics 36, 6 (Nov. 2017).Google ScholarDigital Library
- Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An evaluation of computational imaging techniques for heterogeneous inverse scattering. In European Conference on Computer Vision. Springer, 685--701.Google ScholarCross Ref
- Ioannis Gkioulekas, Shuang Zhao, Kavita Bala, Todd Zickler, and Anat Levin. 2013. Inverse Volume Rendering with Material Dictionaries. ACM Transactions on Graphics 32, 6, Article 162 (Nov. 2013).Google ScholarDigital Library
- Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives: principles and techniques of algorithmic differentiation. Vol. 105. SIAM.Google Scholar
- Laurent Hascoet and Valérie Pascual. 2013. The Tapenade automatic differentiation tool: Principles, model, and specification. ACM Transactions on Mathematical Software (TOMS) 39, 3 (2013), 20.Google ScholarDigital Library
- Shayan Hoshyari, Hongyi Xu, Espen Knoop, Stelian Coros, and Moritz Bächer. 2019. Vibration-minimizing motion retargeting for robotic characters. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--14.Google ScholarDigital Library
- Wenzel Jakob. 2013. Light Transport On Path-Space Manifolds. Ph.D. Dissertation. Cornell University.Google Scholar
- Wenzel Jakob. 2019. Enoki: structured vectorization and differentiation on modern processor architectures. https://github.com/mitsuba-renderer/enoki. (Date accessed: 2020-01-05).Google Scholar
- Kevin G Jamieson, Robert Nowak, and Ben Recht. 2012. Query Complexity of Derivative-Free Optimization. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 2672--2680.Google Scholar
- James T Kajiya. 1986. The rendering equation. In ACM Siggraph Computer Graphics, Vol. 20. ACM.Google ScholarDigital Library
- Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2017. Neural 3D Mesh Renderer. CoRR abs/1711.07566 (2017). arXiv:1711.07566 http://arxiv.org/abs/1711.07566Google Scholar
- Pramook Khungurn, Daniel Schroeder, Shuang Zhao, Kavita Bala, and Steve Marschner. 2015. Matching Real Fabrics with Micro-Appearance Models. ACM Transactions on Graphics 35, 1 (2015).Google ScholarDigital Library
- Samuli Laine, Tero Karras, and Timo Aila. 2013. Megakernels Considered Harmful: Wavefront Path Tracing on GPUs. In Proceedings of the 5th High-Performance Graphics Conference (HPG 13). Association for Computing Machinery, New York, NY, USA, 137143.Google ScholarDigital Library
- Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo Ray Tracing Through Edge Sampling. ACM Transactions on Graphics 37, 6, Article 222 (Dec. 2018).Google ScholarDigital Library
- Shichen Liu, Weikai Chen, Tianye Li, and Hao Li. 2019. Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction. CoRR abs/1901.05567 (2019). arXiv:1901.05567 http://arxiv.org/abs/1901.05567Google Scholar
- Matthew M Loper and Michael J Black. 2014. OpenDR: An approximate differentiable renderer. In European Conference on Computer Vision. Springer.Google ScholarCross Ref
- Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing Discontinuous Integrands for Differentiable Rendering. ACM Transactions on Graphics 38, 6 (Dec. 2019).Google ScholarDigital Library
- Antoine McNamara, Adrien Treuille, Zoran Popović, and Jos Stam. 2004. Fluid control using the adjoint method. In ACM Transactions On Graphics (TOG), Vol. 23. ACM, 449--456.Google ScholarDigital Library
- Thomas Müller, Markus Gross, and Jan Novák. 2017. Practical Path Guiding for Efficient Light-Transport Simulation. Computer Graphics Forum 36, 4 (June 2017).Google ScholarDigital Library
- Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A Retargetable Forward and Inverse Renderer. ACM Transactions on Graphics 38, 6 (Dec. 2019).Google ScholarDigital Library
- Steven G. Parker, James Bigler, Andreas Dietrich, Heiko Friedrich, Jared Hoberock, David Luebke, David McAllister, Morgan McGuire, Keith Morley, Austin Robison, and Martin Stich. 2010. OptiX: A General Purpose Ray Tracing Engine. ACM Transactions on Graphics 29, 4, Article 66 (July 2010).Google ScholarDigital Library
- Felix Petersen, Amit H. Bermano, Oliver Deussen, and Daniel Cohen-Or. 2019. Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer. CoRR abs/1903.11149 (2019). arXiv:1903.11149 http://arxiv.org/abs/1903.11149Google Scholar
- Alain Petrowski, Gerard Dreyfus, and Claude Girault. 1993. Performance analysis of a pipelined backpropagation parallel algorithm. IEEE Transactions on Neural Networks 4, 6 (1993), 970--981.Google ScholarDigital Library
- Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically based rendering: From theory to implementation (third ed.). Morgan Kaufmann.Google Scholar
- Lev Semenovich Pontryagin. 1962. Mathematical theory of optimal processes. CRC Press.Google Scholar
- Ravi Ramamoorthi, Dhruv Mahajan, and Peter Belhumeur. 2007. A first-order analysis of lighting, shading, and shadows. ACM Transactions on Graphics (TOG) 26, 1 (2007), 2.Google ScholarDigital Library
- Helge Rhodin, Nadia Robertini, Christian Richardt, Hans-Peter Seidel, and Christian Theobalt. 2015. A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation. In Proceedings of ICCV 2015.Google ScholarDigital Library
- Denis Sumin, Tobias Rittig, Vahid Babaei, Thomas Nindel, Alexander Wilkie, Piotr Didyk, Bernd Bickel, Jaroslav Křivánek, Karol Myszkowski, and Tim Weyrich. 2019. Geometry-Aware Scattering Compensation for 3D Printing. ACM Transactions on Graphics (2019).Google Scholar
- Eric Veach. 1997. Robust Monte Carlo methods for light transport simulation. Ph.D. Dissertation. Stanford University.Google ScholarDigital Library
- Eric Veach and Leonidas Guibas. 1995. Bidirectional estimators for light transport. In Photorealistic Rendering Techniques. Springer, 145--167.Google Scholar
- Zdravko Velinov, Marios Papas, Derek Bradley, Paulo Gotardo, Parsa Mirdehghan, Steve Marschner, Jan Novák, and Thabo Beeler. 2018. Appearance Capture and Modeling of Human Teeth. ACM Transactions on Graphics 37, 6, Article 207 (Dec. 2018), 13 pages.Google ScholarDigital Library
- Yu M Volin and GM Ostrovskii. 1985. Automatic computation of derivatives with the use of the multilevel differentiating technique1. Algorithmic basis. Computers & mathematics with applications 11, 11 (1985).Google Scholar
- R. E. Wengert. 1964. A Simple Automatic Derivative Evaluation Program. Commun. ACM 7, 8 (Aug. 1964), 463464. Google ScholarDigital Library
- Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A Differential Theory of Radiative Transfer. ACM Trans. Graph. 38, 6 (2019).Google ScholarDigital Library
- Shaung Zhao, Lifan Wu, Frédo Durand, and Ravi Ramamoorthi. 2016. Downsampling Scattering Parameters for Rendering Anisotropic Media. ACM Transactions on Graphics 35, 6 (2016).Google ScholarDigital Library
Index Terms
- Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering
Recommendations
Ray tracing-based interactive diffuse indirect illumination
Despite great efforts in recent years to accelerate global illumination computation, the real-time ray tracing of fully dynamic scenes to support photorealistic indirect illumination effects has yet to be achieved in computer graphics. In this paper, we ...
Complex Luminaires: Illumination and Appearance Rendering
Simulating a complex luminaire such as a chandelier is expensive and slow, even using state-of-the-art algorithms. A more practical alternative is to use precomputation to accelerate rendering. Prior approaches cached information on an aperture surface ...
RenderMan: An Advanced Path-Tracing Architecture for Movie Rendering
Special Issue On Production Rendering and Regular PapersPixar’s RenderMan renderer is used to render all of Pixar’s films and by many film studios to render visual effects for live-action movies. RenderMan started as a scanline renderer based on the Reyes algorithm, and it was extended over the years with ...
Comments