skip to main content
10.1145/3588432.3591493acmconferencesArticle/Chapter ViewAbstractPublication PagessiggraphConference Proceedingsconference-collections
research-article

Deep Real-time Volumetric Rendering Using Multi-feature Fusion

Published: 23 July 2023 Publication History

Abstract

We present Multi-feature Radiance-Predicting Neural Networks (MRPNN), a practical framework with a lightweight feature fusion neural network for rendering high-order scattered radiance of participating media in real time. By reformulating the Radiative Transfer Equation (RTE) through theoretical examination, we propose transmittance fields, generated at a low cost, as auxiliary information to help the network better approximate the RTE, drastically reducing the size of the neural network. The light weight network efficiently estimates the difficult-to-solve in-scattering term and allows for configurable shading parameters while improving prediction accuracy. In addition, we propose a frequency-sensitive stencil design in order to handle non-cloud shapes, resulting in accurate shadow boundaries. Results show that our MRPNN is able to synthesize indistinguishable output compared to the ground truth. Most importantly, MRPNN achieves a speedup of two orders of magnitude compared to the state-of-the-art, and is able to render high-quality participating material in real time.

Supplemental Material

MP4 File
Appendix, demo video
MP4 File
presentation
PDF File
Appendix, demo video

References

[1]
James Arvo. 1993. Transfer equations in global illumination. Global Illumination, SIGGRAPH ‘93 Course Notes 2 (1993).
[2]
Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony DeRose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graph. 36, 4 (2017), 97:1–97:14.
[3]
George EP Box and David R Cox. 1964. An analysis of transformations. Journal of the Royal Statistical Society: Series B (Methodological) 26, 2 (1964), 211–243.
[4]
Xi Deng, Shaojie Jiao, Benedikt Bitterli, and Wojciech Jarosz. 2019. Photon surfaces for robust, unbiased volumetric density estimation. ACM Trans. Graph. 38, 4 (2019), 46:1–46:12.
[5]
Robert A Drebin, Loren Carpenter, and Pat Hanrahan. 1988. Volume rendering. ACM Siggraph Computer Graphics 22, 4 (1988), 65–74.
[6]
Sebastian Herholz, Yangyang Zhao, Oskar Elek, Derek Nowrouzezahrai, Hendrik P. A. Lensch, and Jaroslav Krivánek. 2019. Volume path guiding based on zero-variance random walk theory. ACM Trans. Graph. 38, 3 (2019), 25:1–25:19.
[7]
Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7132–7141.
[8]
Wojciech Jarosz, Derek Nowrouzezahrai, Robert Thomas, Peter-Pike J. Sloan, and Matthias Zwicker. 2011. Progressive photon beams. ACM Trans. Graph. 30, 6 (2011), 181:1–181:12.
[9]
Wojciech Jarosz, Matthias Zwicker, and Henrik Wann Jensen. 2008. The beam radiance estimate for volumetric photon mapping. In ACM SIGGRAPH 2008 classes. 1–112.
[10]
Henrik Wann Jensen, Stephen R. Marschner, Marc Levoy, and Pat Hanrahan. 2001. A practical model for subsurface light transport. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2001, Los Angeles, California, USA, August 12-17, 2001. ACM, 511–518.
[11]
James T. Kajiya and Brian Von Herzen. 1984. Ray tracing volume densities. In Proceedings of the 11th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1984, Minneapolis, Minnesota, USA, July 23-27, 1984. ACM, 165–174.
[12]
Simon Kallweit, Thomas Müller, Brian McWilliams, Markus H. Gross, and Jan Novák. 2017. Deep scattering: rendering atmospheric clouds with radiance-predicting neural networks. ACM Trans. Graph. 36, 6 (2017), 231:1–231:11.
[13]
Anton Kaplanyan and Carsten Dachsbacher. 2010. Cascaded light propagation volumes for real-time indirect illumination. In Proceedings of the 2010 ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games. 99–107.
[14]
David Koerner, Jamie Portsmouth, Filip Sadlo, Thomas Ertl, and Bernd Eberhardt. 2014. Flux-limited diffusion for multiple scattering in participating media. Computer Graphics Forum 33, 6 (2014), 178–189.
[15]
Jaroslav Krivánek, Iliyan Georgiev, Toshiya Hachisuka, Petr Vévoda, Martin Sik, Derek Nowrouzezahrai, and Wojciech Jarosz. 2014. Unifying points, beams, and paths in volumetric light transport simulation. ACM Trans. Graph. 33, 4 (2014), 103:1–103:13.
[16]
Eric P Lafortune and Yves D Willems. 1996. Rendering participating media with bidirectional path tracing. In Eurographics Workshop on Rendering Techniques. Springer, 91–100.
[17]
Ludwig Leonard, Kevin Höhlein, and Rüdiger Westermann. 2021. Learning multiple‐scattering solutions for sphere‐tracing of volumetric subsurface effects. Computer Graphics Forum 40, 2 (2021), 165–178.
[18]
Liangchen Luo, Yuanhao Xiong, Yan Liu, and Xu Sun. 2019. Adaptive gradient methods with dynamic bound of learning rate. In Proceedings of the 7th International Conference on Learning Representations. New Orleans, Louisiana.
[19]
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: representing scenes as neural radiance fields for view synthesis. In Computer Vision - ECCV 2020 - 16th European Conference(Lecture Notes in Computer Science, Vol. 12346). Springer, 405–421.
[20]
Thomas Müller, Fabrice Rousselle, Jan Novák, and Alexander Keller. 2021. Real-time neural radiance caching for path tracing. ACM Trans. Graph. 40, 4 (2021), 36:1–36:16.
[21]
Mikhail Panin and Sergey I. Nikolenko. 2019. Faster RPNN: rendering clouds with latent space light probes. In SIGGRAPH Asia 2019 Technical Briefs, SA 2019, Brisbane, QLD, Australia, November 17-20, 2019. ACM, 21–24.
[22]
Mark Pauly, Thomas Kollig, and Alexander Keller. 2000. Metropolis light transport for participating media. In Eurographics Workshop on Rendering Techniques. Springer, 11–22.
[23]
Jos Stam. 1995. Multiple scattering as a diffusion process. In Eurographics Workshop on Rendering Techniques. Springer, 41–50.
[24]
Eric Veach. 1998. Robust Monte Carlo Methods for Light Transport Simulation. Ph. D. Dissertation. Stanford University, Stanford, CA, USA. Advisor(s) Guibas, Leonidas J. AAI9837162.
[25]
D. Vicini, V. Koltun, and W. Jakob. 2019. A learned shape-adaptive subsurface scattering model. ACM Trans. Graph. 38, 4 (2019), 127:1–127:15.
[26]
E Woodcock, T Murphy, P Hemmings, and S Longworth. 1965. Techniques used in the GEM code for Monte Carlo neutronics calculations in reactors and other systems of complex geometry. In Proc. Conf. Applications of Computing Methods to Reactor Problems, Vol. 557. 557–579.
[27]
M. Wrenninge, N. B. Zafar, A. Bouthors, J. Tessendorf, and G. Graham. 2011. Production Volume Rendering: Systems. In ACM SIGGRAPH (Courses).
[28]
In-Kwon Yeo and Richard A Johnson. 2000. A new family of power transformations to improve normality or symmetry. Biometrika 87, 4 (2000), 954–959.
[29]
Arno Zinke, Cem Yuksel, Andreas Weber, and John Keyser. 2008. Dual scattering approximation for fast multiple scattering in hair. ACM Trans. Graph. 27, 3 (2008), 32:1–32:10.

Cited By

View all
  • (2024)GPU-Enabled Volume Renderer for Use with MATLABDigital10.3390/digital40400494:4(990-1007)Online publication date: 30-Nov-2024
  • (2024)NeuralTO: Neural Reconstruction and View Synthesis of Translucent ObjectsACM Transactions on Graphics10.1145/365818643:4(1-14)Online publication date: 19-Jul-2024
  • (2024)Improved Perceptual Representation of Isosurfaces From Volume Data Using Curvature-based Features2024 28th International Conference on System Theory, Control and Computing (ICSTCC)10.1109/ICSTCC62912.2024.10744636(139-144)Online publication date: 10-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGGRAPH '23: ACM SIGGRAPH 2023 Conference Proceedings
July 2023
911 pages
ISBN:9798400701597
DOI:10.1145/3588432
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 July 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Participating media
  2. radiative transfer equation
  3. real time
  4. variable phase function
  5. volumetric rendering

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • Intel
  • Adobe
  • the National Natural Science Foundation of China
  • Meta
  • Key and the R&D Program of Zhejiang

Conference

SIGGRAPH '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)179
  • Downloads (Last 6 weeks)11
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)GPU-Enabled Volume Renderer for Use with MATLABDigital10.3390/digital40400494:4(990-1007)Online publication date: 30-Nov-2024
  • (2024)NeuralTO: Neural Reconstruction and View Synthesis of Translucent ObjectsACM Transactions on Graphics10.1145/365818643:4(1-14)Online publication date: 19-Jul-2024
  • (2024)Improved Perceptual Representation of Isosurfaces From Volume Data Using Curvature-based Features2024 28th International Conference on System Theory, Control and Computing (ICSTCC)10.1109/ICSTCC62912.2024.10744636(139-144)Online publication date: 10-Oct-2024
  • (2024)NeRFDeformer: NeRF Transformation from a Single View via 3D Scene Flows2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00980(10293-10303)Online publication date: 16-Jun-2024
  • (2023)Seal-3D: Interactive Pixel-Level Editing for Neural Radiance Fields2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01621(17637-17647)Online publication date: 1-Oct-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media