ABSTRACT
Physics-based rendering algorithms generate photorealistic images by simulating the flow of light through a detailed mathematical representation of a virtual scene. In contrast, physics-based differentiable rendering algorithms focus on computing derivative of images exhibiting complex light transport effects (e.g., soft shadows, interreflection, and caustics) with respect to arbitrary scene parameters such as camera pose, object geometry (e.g., vertex positions) as well as spatially varying material properties expressed as 2D textures and 3D volumes. This new level of generality has made physics-based differentiable rendering a key ingredient for solving many challenging inverse-rendering problems, that is, the search of scene configurations optimizing user-specified objective functions, using gradient-based methods (as illustrated in the figure below). Further, these techniques can be incorporated into probabilistic inference and machine learning pipelines. For instance, differentiable renderers allow "rendering losses" to be computed with complex light transport effects captured. Additionally, they can be used as generative models that synthesize photorealistic images.
- M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.Google Scholar
- James Arvo. 1994. The Irradiance Jacobian for partially occluded polyhedral sources. In SIGGRAPH '94. 343--350.Google ScholarDigital Library
- Edwin Catmull. 1978. A Hidden-Surface Algorithm with Anti-Aliasing. Comput. Graph. (Proc. SIGGRAPH) 12, 3 (1978), 6--11.Google ScholarDigital Library
- Subrahmanyan Chandrasekhar. 1960. Radiative Transfer. Courier Corporation.Google Scholar
- Robert L. Cook, Thomas Porter, and Loren Carpenter. 1984. Distributed Ray Tracing. Comput. Graph. (Proc. SIGGRAPH) 18, 3 (1984), 137--145.Google ScholarDigital Library
- Franklin C Crow. 1977. The aliasing problem in computer-generated shaded images. Commun. ACM 20, 11 (1977), 799--805.Google ScholarDigital Library
- Tomáš Davidović, Thomas Engelhardt, Iliyan Georgiev, Philipp Slusallek, and Carsten Dachsbacher. 2012. 3D rasterization: a bridge between rasterization and ray casting. In Graphics Interface. 201--208.Google Scholar
- Martin de La Gorce, David J Fleet, and Nikos Paragios. 2011. Model-based 3D hand pose estimation from monocular video. IEEE Trans. Pattern Anal. Mach. Intell. 33, 9 (2011), 1793--1805.Google ScholarDigital Library
- Harley Flanders. 1973. Differentiation under the integral sign. The American Mathematical Monthly 80, 6 (1973), 615--627.Google ScholarCross Ref
- Ned Greene and Michael Kass. 1994. Error-bounded antialiased rendering of complex environments. In SIGGRAPH. 59--66.Google Scholar
- Ned Greene, Michael Kass, and Gavin Miller. 1993. Hierarchical Z-buffer visibility. In SIGGRAPH. ACM, 231--238.Google Scholar
- Andreas Griewank. 1992. Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiation. 1, 1 (1992), 35--54.Google ScholarCross Ref
- Andreas Griewank and Shawn Reese. 1991. On the Calculation of Jacobian Matrices by the Markowitz Rule. In Automatic Differentiation of Algorithms: Theory, Implementation, and Application. SIAM, 126--135.Google Scholar
- Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives: principles and techniques of algorithmic differentiation. Vol. 105. SIAM.Google Scholar
- Warren Hunt, Michael Mara, and Alex Nankervis. 2018. Hierarchical Visibility for Virtual Reality. ACM Comput. Graph. Interact. Tech. (Proc. I3D) 1, 1 (2018), 1--18.Google ScholarDigital Library
- Wolfram Research, Inc. [n.d.]. Mathematica, Version 11.3. Champaign, IL, 2018.Google Scholar
- Wenzel Jakob. 2019. Enoki: structured vectorization and differentiation on modern processor architectures. https://github.com/mitsuba-renderer/enoki.Google Scholar
- Henrik Wann Jensen. 1995. Importance driven path tracing using the photon map. In Rendering Techniques (Proc. EGWR). Springer, 326--335.Google Scholar
- James T. Kajiya. 1986. The Rendering Equation. In SIGGRAPH '86. 143--150.Google Scholar
- Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D mesh renderer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3907--3916.Google ScholarCross Ref
- L. Gary Leal. 2007. Advanced Transport Phenomena: fluid mechanics and convective transport processes. Vol. 7. Cambridge University Press.Google Scholar
- Tzu-Mao Li. 2019. Differentiable visual computing. MIT PhD Dissertation.Google Scholar
- Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo ray tracing through edge sampling. ACM Trans. Graph. 37, 6 (2018), 222:1--222:11.Google ScholarDigital Library
- Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019. Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning. International Conference on Computer Vision (2019).Google ScholarCross Ref
- Matthew M Loper and Michael J Black. 2014. OpenDR: An approximate differentiable renderer. In European Conference on Computer Vision. Springer, 154--169.Google ScholarCross Ref
- Morgan McGuire. 2004. Observations on silhouette sizes. J. Graph. Tools 9, 1 (2004), 1--12.Google ScholarCross Ref
- Morgan McGuire and Louis Bavoil. 2013. Weighted blended order-independent transparency. Journal of Computer Graphics Techniques 2, 4 (2013).Google Scholar
- Don P Mitchell and Arun N Netravali. 1988. Reconstruction filters in computer-graphics. Comput. Graph. (Proc. SIGGRAPH) 22, 4 (1988), 221--228.Google ScholarDigital Library
- Uwe Naumann. 2008. Optimal Jacobian accumulation is NP-complete. Mathematical Programming 112, 2 (2008), 427--441.Google ScholarDigital Library
- Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A retargetable forward and inverse renderer. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (2019), 1--17.Google ScholarDigital Library
- Matt Olson and Hao Zhang. 2006. Silhouette extraction in Hough space. Comput. Graph. Forum (Proc. Eurographics) 25, 3 (2006), 273--282.Google ScholarCross Ref
- Eric Paquette, Pierre Poulin, and George Drettakis. 1998. A Light Hierarchy for Fast Rendering of Scenes with Many Lights. Comput. Graph. Forum (Proc. Eurographics) (1998), 63--74.Google Scholar
- Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. (2017).Google Scholar
- Thomas Porter and Tom Duff. 1984. Compositing digital images. (1984), 253--259.Google Scholar
- Ravi Ramamoorthi, Dhruv Mahajan, and Peter Belhumeur. 2007. A First-order Analysis of Lighting, Shading, and Shadows. ACM Trans. Graph. 26, 1 (2007), 2:1--2:21.Google ScholarDigital Library
- Pedro V. Sander, Hugues Hoppe, John Snyder, and Steven J. Gortler. 2001. Discontinuity edge overdraw. In Symposium on Interactive 3D Graphics and Games. ACM, 167--174.Google Scholar
- Michelle Mills Strout and Paul Hovland. 2006. Linearity analysis for automatic differentiation. In International Conference on Computational Science. Springer, 574--581.Google ScholarDigital Library
- Eric Veach. 1998. Robust Monte Carlo Methods for Light Transport Simulation. Ph.D. Dissertation. Stanford University. Advisor(s) Guibas, Leonidas J.Google ScholarDigital Library
- Dominique Villard and Michael B Monagan. 1999. ADrien: an implementation of automatic differentiation in Maple. In International Symposium on Symbolic and Algebraic Computation. 221--228.Google ScholarDigital Library
- Bruce Walter, Sebastian Fernandez, Adam Arbree, Kavita Bala, Michael Donikian, and Donald P. Greenberg. 2005. Lightcuts: A Scalable Approach to Illumination. ACM Trans. Graph. (Proc. SIGGRAPH) 24, 3 (2005), 1098--1107.Google ScholarDigital Library
- Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-Space Differentiable Rendering. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 6 (2020), 143:1--143:19.Google ScholarDigital Library
- Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shaung Zhao. 2019. A Differential Theory of Radiative Transfer. ACM Trans. Graph. 38, 6 (2019), 227:1--227:16.Google ScholarDigital Library
- Physics-based differentiable rendering: from theory to implementation
Recommendations
Reparameterizing discontinuous integrands for differentiable rendering
Differentiable rendering has recently opened the door to a number of challenging inverse problems involving photorealistic images, such as computational material design and scattering-aware reconstruction of geometry and materials from photographs. ...
Reconstructing Translucent Objects using Differentiable Rendering
SIGGRAPH '22: ACM SIGGRAPH 2022 Conference ProceedingsInverse rendering is a powerful approach to modeling objects from photographs, and we extend previous techniques to handle translucent materials that exhibit subsurface scattering. Representing translucency using a heterogeneous bidirectional ...
Comments