skip to main content
research-article

Interactive Monte Carlo denoising using affinity of neural features

Published: 19 July 2021 Publication History

Abstract

High-quality denoising of Monte Carlo low-sample renderings remains a critical challenge for practical interactive ray tracing. We present a new learning-based denoiser that achieves state-of-the-art quality and runs at interactive rates. Our model processes individual path-traced samples with a lightweight neural network to extract per-pixel feature vectors. The rest of our pipeline operates in pixel space. We define a novel pairwise affinity over the features in a pixel neighborhood, from which we assemble dilated spatial kernels to filter the noisy radiance. Our denoiser is temporally stable thanks to two mechanisms. First, we keep a running average of the noisy radiance and intermediate features, using a per-pixel recursive filter with learned weights. Second, we use a small temporal kernel based on the pairwise affinity between features of consecutive frames. Our experiments show our new affinities lead to higher quality outputs than techniques with comparable computational costs, and better high-frequency details than kernel-predicting approaches. Our model matches or outperfoms state-of-the-art offline denoisers in the low-sample count regime (2--8 samples per pixel), and runs at interactive frame rates at 1080p resolution.

Supplementary Material

VTT File (3450626.3459793.vtt)
ZIP File (a37-isik.zip)
a37-isik.zip
MP4 File (a37-isik.mp4)
MP4 File (3450626.3459793.mp4)
Presentation.

References

[1]
Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graph. 36, 4 (2017), 97--1.
[2]
Pablo Bauszat, Martin Eisemann, S John, and M Magnor. 2015. Sample-based manifold filtering for interactive global illumination and depth of field. Computer Graphics Forum (2015).
[3]
Pablo Bauszat, Martin Eisemann, and Marcus Magnor. 2011. Guided Image Filtering for Interactive High-quality Global Illumination. Computer Graphics Forum (Proc. EGSR) (2011).
[4]
Laurent Belcour, Cyril Soler, Kartic Subr, Nicolas Holzschuch, and Fredo Durand. 2013. 5D covariance tracing for efficient defocus and motion blur. ACM TOG (2013).
[5]
Benedikt Bitterli. 2016. Rendering resources. https://benedikt-bitterli.me/resources.
[6]
Benedikt Bitterli, Fabrice Rousselle, Bochang Moon, José A. Iglesias-Guitiàn, David Adler, Kenny Mitchell, Wojciech Jarosz, and Jan Novák. 2016. Nonlinearly Weighted First-order Regression for Denoising Monte Carlo Renderings. Computer Graphics Forum (Proc. EGSR) (2016).
[7]
Chakravarty R Alla Chaitanya, Anton S Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1--12.
[8]
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR]. Stanford University --- Princeton University --- Toyota Technological Institute at Chicago.
[9]
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. 2014. Describing Textures in the Wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR).
[10]
Holger Dammertz, Daniel Sewtz, Johannes Hanika, and Hendrik PA Lensch. 2010. Edge-avoiding À-Trous wavelet transform for fast global illumination filtering. In Proceedings of the Conference on High Performance Graphics. Citeseer, 67--75.
[11]
Mauricio Delbracio, Pablo Musé, Antoni Buades, Julien Chauvier, Nicholas Phelps, and Jean-Michel Morel. 2014. Boosting monte carlo rendering by ray histogram fusion. ACM Transactions on Graphics (TOG) 33, 1 (2014), 1--15.
[12]
Frédo Durand, Nicolas Holzschuch, Cyril Soler, Eric Chan, and François X Sillion. 2005. A frequency analysis of light transport. ACM SIGGRAPH (2005).
[13]
Kevin Egan, Florian Hecht, Frédo Durand, and Ravi Ramamoorthi. 2011. Frequency analysis and sheared filtering for shadow light fields of complex occluders. ACM TOG (2011).
[14]
Kevin Egan, Yu-Ting Tseng, Nicolas Holzschuch, Frédo Durand, and Ravi Ramamoorthi. 2009. Frequency analysis and sheared reconstruction for rendering motion blur. ACM SIGGRAPH (2009).
[15]
Michaël Gharbi, Jiawen Chen, Jonathan T Barron, Samuel W Hasinoff, and Frédo Durand. 2017. Deep bilateral learning for real-time image enhancement. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1--12.
[16]
Michaël Gharbi, Tzu-Mao Li, Miika Aittala, Jaakko Lehtinen, and Frédo Durand. 2019. Sample-based Monte Carlo denoising using a kernel-splatting network. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--12.
[17]
Toshiya Hachisuka, Wojciech Jarosz, Richard Peter Weistroffer, Kevin Dale, Greg Humphreys, Matthias Zwicker, and Henrik Wann Jensen. 2008. Multidimensional adaptive sampling and reconstruction for ray tracing. ACM SIGGRAPH (2008).
[18]
J. Hasselgren, J. Munkberg, M. Salvi, A. Patney, and A. Lefohn. 2020. Neural Temporal Adaptive Sampling and Denoising. Computer Graphics Forum 39 (2020).
[19]
Kaiming He, Jian Sun, and Xiaoou Tang. 2010. Guided image filtering. ECCV (2010).
[20]
Matthias Holschneider, Richard Kronland-Martinet, Jean Morlet, and Ph Tchamitchian. 1990. A real-time algorithm for signal analysis with the help of the wavelet transform. In Wavelets. Springer, 286--297.
[21]
James T. Kajiya. 1986. The Rendering Equation. ACM SIGGRAPH (1986).
[22]
Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A machine learning approach for filtering Monte Carlo noise. ACM Trans. Graph. 34, 4 (2015), 122--1.
[23]
Nima Khademi Kalantari and Pradeep Sen. 2013. Removing the noise in Monte Carlo rendering with general image denoising algorithms. Computer Graphics Forum (Proc. EG) (2013).
[24]
Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. 2019. Deep convolutional reconstruction for gradient-domain rendering. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--12.
[25]
Khronos. 2020. Vulkan 1.2.167 Specification. https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/vkspec.html
[26]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[27]
Matias Koskela, Kalle Immonen, Markku Mäkitalo, Alessandro Foi, Timo Viitanen,
[28]
Pekka Jääskeläinen, Heikki Kultala, and Jarmo Takala. 2019. Blockwise multi-order feature regression for real-time path-tracing reconstruction. ACM Transactions on Graphics (TOG) 38, 5 (2019), 1--14.
[29]
Alexandr Kuznetsov, Nima Khademi Kalantari, and Ravi Ramamoorthi. 2018. Deep Adaptive Sampling for Low Sample Count Rendering. In Computer Graphics Forum, Vol. 37. Wiley Online Library, 35--44.
[30]
Eric P Lafortune and Yves D Willems. 1993. Bi-directional path tracing. (1993).
[31]
Jaakko Lehtinen, Timo Aila, Jiawen Chen, Samuli Laine, and Frédo Durand. 2011. Temporal light field reconstruction for rendering distribution effects. ACM SIGGRAPH (2011).
[32]
Jaakko Lehtinen, Timo Aila, Samuli Laine, and Frédo Durand. 2012. Reconstructing the indirect light field for global illumination. ACM SIGGRAPH (2012).
[33]
Tzu-Mao Li, Yu-Ting Wu, and Yung-Yu Chuang. 2012. SURE-based optimization for adaptive sampling and reconstruction. ACM SIGGRAPH Asia (2012).
[34]
Michael Mara, Morgan McGuire, Benedikt Bitterli, and Wojciech Jarosz. 2017. An efficient denoising algorithm for global illumination. High Performance Graphics 10 (2017), 3105762--3105774.
[35]
Xiaoxu Meng, Quan Zheng, Amitabh Varshney, Gurprit Singh, and Matthias Zwicker. 2020. Real-time Monte Carlo Denoising with the Neural Bilateral Grid. In Eurographics Symposium on Rendering - DL-only Track, Carsten Dachsbacher and Matt Pharr (Eds.). The Eurographics Association.
[36]
Microsoft. 2021. DirectX Raytracing (DXR) Functional Spec. https://microsoft.github.io/DirectX-Specs/d3d/Raytracing.html
[37]
Bochang Moon, Nathan Carr, and Sung-Eui Yoon. 2014. Adaptive rendering based on weighted local regression. ACM TOG (2014).
[38]
Bochang Moon, Steven McDonagh, Kenny Mitchell, and Markus Gross. 2016. Adaptive polynomial rendering. ACM Transactions on Graphics (TOG) 35, 4 (2016), 1--10.
[39]
Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. 2019. Neural importance sampling. ACM Transactions on Graphics (TOG) 38, 5 (2019), 1--19.
[40]
Jacob Munkberg and Jon Hasselgren. 2020. Neural Denoising with Layer Embeddings. In Computer Graphics Forum, Vol. 39. Wiley Online Library, 1--12.
[41]
Ryan S Overbeck, Craig Donner, and Ravi Ramamoorthi. 2009. Adaptive wavelet rendering. ACM SIGGRAPH Asia (2009).
[42]
Steven G. Parker, Heiko Friedrich, David Luebke, Keith Morley, James Bigler, Jared Hoberock, David McAllister, Austin Robison, Andreas Dietrich, Greg Humphreys, Morgan McGuire, and Martin Stich. 2013. GPU Ray Tracing. Commun. ACM 56, 5 (May 2013), 93--101.
[43]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024--8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
[44]
Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically based rendering: From theory to implementation. Morgan Kaufmann.
[45]
Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda. 2002. Photographic tone reproduction for digital images. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques. 267--276.
[46]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.
[47]
Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2011. Adaptive Sampling and Reconstruction Using Greedy Error Minimization. ACM Trans. Graph. 30, 6 (Dec. 2011), 1--12.
[48]
Fabrice Rousselle, Claude Knaus, and Matthias Zwicker. 2012. Adaptive rendering with non-local means filtering. ACM SIGGRAPH Asia (2012).
[49]
Fabrice Rousselle, Marco Manzi, and Matthias Zwicker. 2013. Robust denoising using feature and color information. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 121--130.
[50]
Christoph Schied, Anton Kaplanyan, Chris Wyman, Anjul Patney, Chakravarty R. Alla Chaitanya, John Burgess, Shiqiu Liu, Carsten Dachsbacher, Aaron Lefohn, and Marco Salvi. 2017. Spatiotemporal Variance-guided Filtering: Real-time Reconstruction for Path-traced Global Illumination. High Performance Graphics (2017).
[51]
Christoph Schied, Christoph Peters, and Carsten Dachsbacher. 2018. Gradient estimation for real-time adaptive temporal filtering. Proceedings of the ACM on Computer Graphics and Interactive Techniques 1, 2 (2018), 1--16.
[52]
Pradeep Sen and Soheil Darabi. 2012. On filtering the noise from the random parameters in Monte Carlo rendering. ACM TOG (2012).
[53]
Carlo Tomasi and Roberto Manduchi. 1998. Bilateral filtering for gray and color images. In Sixth international conference on computer vision (IEEE Cat. No. 98CH36271). IEEE, 839--846.
[54]
Eric Veach and Leonidas Guibas. 1995. Bidirectional estimators for light transport. In Photorealistic Rendering Techniques. Springer, 145--167.
[55]
Thijs Vogels, Fabrice Rousselle, Brian McWilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with kernel prediction and asymmetric loss functions. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--15.
[56]
Yue Wu, Brian Tracey, Premkumar Natarajan, and Joseph P Noonan. 2013. James-Stein type center pixel weights for non-local means image denoising. IEEE Signal Processing Letters 20, 4 (2013), 411--414.
[57]
Bing Xu, Junfei Zhang, Rui Wang, Kun Xu, Yong-Liang Yang, Chuan Li, and Rui Tang. 2019. Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation. ACM Trans. Graph. 38, 6 (2019), 224--1.
[58]
Fisher Yu and Vladlen Koltun. 2016. Multi-Scale Context Aggregation by Dilated Convolutions. In ICLR.
[59]
M. Zwicker, W. Jarosz, J. Lehtinen, B. Moon, R. Ramamoorthi, F. Rousselle, P. Sen, C. Soler, and S.-E. Yoon. 2015. Recent Advances in Adaptive Sampling and Reconstruction for Monte Carlo Rendering. Computer Graphics Forum (Proc. EG) (2015).

Cited By

View all
  • (2024)Neural Kernel Regression for Consistent Monte Carlo DenoisingACM Transactions on Graphics10.1145/368794943:6(1-14)Online publication date: 19-Dec-2024
  • (2024)Online Neural Denoising with Cross-Regression for Interactive RenderingACM Transactions on Graphics10.1145/368793843:6(1-12)Online publication date: 19-Dec-2024
  • (2024)Denoising Monte Carlo Renders with Diffusion ModelsACM SIGGRAPH 2024 Posters10.1145/3641234.3671026(1-2)Online publication date: 25-Jul-2024
  • Show More Cited By

Index Terms

  1. Interactive Monte Carlo denoising using affinity of neural features

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Transactions on Graphics
      ACM Transactions on Graphics  Volume 40, Issue 4
      August 2021
      2170 pages
      ISSN:0730-0301
      EISSN:1557-7368
      DOI:10.1145/3450626
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 19 July 2021
      Published in TOG Volume 40, Issue 4

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Monte Carlo denoising
      2. deep learning

      Qualifiers

      • Research-article

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)184
      • Downloads (Last 6 weeks)25
      Reflects downloads up to 17 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Neural Kernel Regression for Consistent Monte Carlo DenoisingACM Transactions on Graphics10.1145/368794943:6(1-14)Online publication date: 19-Dec-2024
      • (2024)Online Neural Denoising with Cross-Regression for Interactive RenderingACM Transactions on Graphics10.1145/368793843:6(1-12)Online publication date: 19-Dec-2024
      • (2024)Denoising Monte Carlo Renders with Diffusion ModelsACM SIGGRAPH 2024 Posters10.1145/3641234.3671026(1-2)Online publication date: 25-Jul-2024
      • (2024)Neural Denoising for Deep‐Z Monte Carlo RenderingsComputer Graphics Forum10.1111/cgf.1505043:2Online publication date: 15-May-2024
      • (2024)Real-Time Monte Carlo Denoising With Adaptive Fusion NetworkIEEE Access10.1109/ACCESS.2024.336958812(29154-29165)Online publication date: 2024
      • (2024)All-optical image denoising using a diffractive visual processorLight: Science & Applications10.1038/s41377-024-01385-613:1Online publication date: 4-Feb-2024
      • (2024)Joint self-attention for denoising Monte Carlo renderingThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-024-03446-840:7(4623-4634)Online publication date: 1-Jul-2024
      • (2023)Optimal MIS weights in case of mixing 3 strategies for bidirectional MCRT with photon mapsMathematica Montisnigri10.20948/mathmontis-2023-57-557(65-83)Online publication date: 2023
      • (2023)MCNeRF: Monte Carlo Rendering and Denoising for Real-Time NeRFsSIGGRAPH Asia 2023 Conference Papers10.1145/3610548.3618221(1-11)Online publication date: 10-Dec-2023
      • (2023)Input-Dependent Uncorrelated Weighting for Monte Carlo DenoisingSIGGRAPH Asia 2023 Conference Papers10.1145/3610548.3618177(1-10)Online publication date: 10-Dec-2023
      • Show More Cited By

      View Options

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media