Skip to main content
SpringerLink
Log in

Denoising Monte Carlo renderings via a multi-scale featured dual-residual GAN

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Monte Carlo (MC) path tracing causes a lot of noise on the rendered image at a low samples per pixel. Recently, with the help of inexpensive auxiliary buffers and the generative adversarial network (GAN), deep learning-based denoising MC rendering methods have been able to generate noise-free images with high perceptual quality in seconds. In this paper, we propose a novel GAN structure for denoising Monte Carlo renderings, called dual residual connection GAN. Our key insight is that the dual residual connections can improve the chance of the optimal feature selection and implicitly increase the number of potential interactions between modules. We also propose a multi-scale auxiliary features extraction method, aiming to make full use of the rich geometry and texture information of auxiliary buffers. Moreover, we adopt a spatial-adaptive block with the deformable convolution to help the network adapt to the variance in spatial texture and edge features. Compared with the state-of-the-art methods, our network has fewer parameters and less inference time, and the results surpass the previous in terms of visual effects and quantitative metrics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein GAN. arXiv:1701.07875 (2017)

  2. Bako, S., Vogels, T., McWilliams, B., Meyer, M., Novák, J., Harvill, A., Sen, P., Derose, T., Rousselle, F.: Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graph. 36(4), 97–1 (2017)

    Article  Google Scholar 

  3. Bitterli, B.: Rendering resources (2016). https://benedikt-bitterli.me/resources/

  4. Bitterli, B., Rousselle, F., Moon, B., Iglesias-Guitián, J.A., Adler, D., Mitchell, K., Jarosz, W., Novák, J.: Nonlinearly weighted first-order regression for denoising Monte Carlo renderings. In: Computer Graphics Forum, vol. 35, pp. 107–117. Wiley Online Library (2016)

  5. Chaitanya, C.R.A., Kaplanyan, A.S., Schied, C., Salvi, M., Lefohn, A., Nowrouzezahrai, D., Aila, T.: Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Trans. Graph. (TOG) 36(4), 1–12 (2017)

    Article  Google Scholar 

  6. Chang, M., Li, Q., Feng, H., Xu, Z.: Spatial-adaptive network for single image denoising. arXiv:2001.10291 (2020)

  7. Gharbi, M., Chen, J., Barron, J.T., Hasinoff, S.W., Durand, F.: Deep bilateral learning for real-time image enhancement. ACM Trans. Graph. (TOG) 36(4), 1–12 (2017)

    Article  Google Scholar 

  8. Gharbi, M., Li, T.M., Aittala, M., Lehtinen, J., Durand, F.: Sample-based Monte Carlo denoising using a kernel-splatting network. ACM Trans. Graph. (TOG) 38(4), 1–12 (2019)

    Article  Google Scholar 

  9. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. Adv. Neural Inf. Process. Syst. 27, 2672–2680 (2014)

    Google Scholar 

  10. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of Wasserstein GANs. In: Advances in Neural Information Processing Systems, pp. 5767–5777 (2017)

  11. Guo, J., Li, M., Li, Q., Qiang, Y., Hu, B., Guo, Y., Yan, L.Q.: Gradnet: unsupervised deep screened Poisson reconstruction for gradient-domain rendering. ACM Trans. Graph. (TOG) 38(6), 1–13 (2019)

    Google Scholar 

  12. Hasselgren, J., Munkberg, J., Salvi, M., Patney, A., Lefohn, A.: Neural temporal adaptive sampling and denoising. In: Computer Graphics Forum, vol. 39, pp. 147–155. Wiley Online Library (2020)

  13. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  14. He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: European Conference on Computer Vision, pp. 630–645. Springer (2016)

  15. Intel open image denoise. https://www.openimagedenoise.org/documentation.html

  16. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)

  17. Jo, Y., Park, J.: Sc-fegan: Face editing generative adversarial network with user’s sketch and color. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1745–1753 (2019)

  18. Kajiya, J.T.: The rendering equation. In: Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques, pp. 143–150 (1986)

  19. Kalantari, N.K., Bako, S., Sen, P.: A machine learning approach for filtering Monte Carlo noise. ACM Trans. Graph. 34(4), 122–1 (2015)

    Article  Google Scholar 

  20. Keller, A., Fascione, L., Fajardo, M., Georgiev, I., Christensen, P., Hanika, J., Eisenacher, C., Nichols, G.: The path tracing revolution in the movie industry. In: ACM SIGGRAPH 2015 Courses, pp. 1–7 (2015)

  21. Kettunen, M., Härkönen, E., Lehtinen, J.: Deep convolutional reconstruction for gradient-domain rendering. ACM Trans. Graph. (TOG) 38(4), 1–12 (2019)

    Article  Google Scholar 

  22. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv:1412.6980 (2014)

  23. Kuznetsov, A., Kalantari, N.K., Ramamoorthi, R.: Deep adaptive sampling for low sample count rendering. In: Computer Graphics Forum, vol. 37, pp. 35–44. Wiley Online Library (2018)

  24. Lehtinen, J., Munkberg, J., Hasselgren, J., Laine, S., Karras, T., Aittala, M., Aila, T.: Noise2noise: learning image restoration without clean data. arXiv:1803.04189 (2018)

  25. Liu, X., Suganuma, M., Sun, Z., Okatani, T.: Dual residual networks leveraging the potential of paired operations for image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7007–7016 (2019)

  26. Lu, Y., Xie, N., Shen, H.T.: DMCR-GAN: Adversarial denoising for Monte Carlo renderings with residual attention networks and hierarchical features modulation of auxiliary buffers. In: SIGGRAPH Asia 2020 Technical Communications, pp. 1–4 (2020)

  27. Meng, X., Zheng, Q., Varshney, A., Singh, G., Zwicker, M.: Real-time Monte Carlo denoising with the neural bilateral grid (2020)

  28. Munkberg, J., Hasselgren, J.: Neural denoising with layer embeddings. In: Computer Graphics Forum, vol. 39, pp. 1–12. Wiley Online Library (2020)

  29. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014)

  30. Vicini, D., Adler, D., Novák, J., Rousselle, F., Burley, B.: Denoising deep Monte Carlo renderings. In: Computer Graphics Forum, vol. 38, pp. 316–327. Wiley Online Library (2019)

  31. Vogels, T., Rousselle, F., McWilliams, B., Röthlin, G., Harvill, A., Adler, D., Meyer, M., Novák, J.: Denoising with kernel prediction and asymmetric loss functions. ACM Trans. Graph. (TOG) 37(4), 1–15 (2018)

    Article  Google Scholar 

  32. Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional GANs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8798–8807 (2018)

  33. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Change Loy, C.: Esrgan: enhanced super-resolution generative adversarial networks. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)

  34. Wong, K.M., Wong, T.T.: Deep residual learning for denoising Monte Carlo renderings. Comput. Vis. Med. 5(3), 239–255 (2019)

    Article  Google Scholar 

  35. Xu, B., Zhang, J., Wang, R., Xu, K., Yang, Y.L., Li, C., Tang, R.: Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation. ACM Trans. Graph. 38(6), 224–1 (2019)

    Google Scholar 

  36. Yang, X., Wang, D., Hu, W., Zhao, L.J., Yin, B.C., Zhang, Q., Wei, X.P., Fu, H.: Demc: A deep dual-encoder network for denoising Monte Carlo rendering. J. Comput. Sci. Technol. 34(5), 1123–1135 (2019)

    Article  Google Scholar 

  37. Zamir, S.W., Arora, A., Khan, S., Hayat, M., Khan, F.S., Yang, M.H., Shao, L.: Learning enriched features for real image restoration and enhancement. arXiv:2003.06792 (2020)

  38. Zhu, X., Hu, H., Lin, S., Dai, J.: Deformable convnets v2: More deformable, better results. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9308–9316 (2019)

  39. Zwicker, M., Jarosz, W., Lehtinen, J., Moon, B., Ramamoorthi, R., Rousselle, F., Sen, P., Soler, C., Yoon, S.E.: Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. In: Computer Graphics Forum, vol. 34, pp. 667–681. Wiley Online Library (2015)

Download references

Funding

This work is part of the research supported by the National Nature Science Foundation of China under Grant No. 61602088, No. This work is part of the research supported by the Sichuan Provincial NSFC (No. 2018JY0528), the Fundamental Research Funds for the Central Universities No. Y03019023601008011, the interactive Technology Research Fund of the Research Center for Interactive Technology Industry, School of Economics and Management, Tsinghua University (No. RCITI2021T006) and sponsored by TiMi L1 Studio of Tencent corporation.

Author information

Authors and Affiliations

  1. Center for Future Media, Department of Computer Science and Engineering, University of Electronic Science and Technology of China (UESTC), Chengdu, 611731, China

    Yifan Lu, Siyuan Fu & Ning Xie

  2. Hiroshima Institute of Technology, Hiroshima, Japan

    Xiao Hua Zhang

Authors
  1. Yifan Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Siyuan Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiao Hua Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ning Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ning Xie.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lu, Y., Fu, S., Zhang, X.H. et al. Denoising Monte Carlo renderings via a multi-scale featured dual-residual GAN. Vis Comput 37, 2513–2525 (2021). https://doi.org/10.1007/s00371-021-02204-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02204-4

Keywords

Search

Navigation