skip to main content
research-article

Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder

Published: 20 July 2017 Publication History

Abstract

We describe a machine learning technique for reconstructing image sequences rendered using Monte Carlo methods. Our primary focus is on reconstruction of global illumination with extremely low sampling budgets at interactive rates. Motivated by recent advances in image restoration with deep convolutional networks, we propose a variant of these networks better suited to the class of noise present in Monte Carlo rendering. We allow for much larger pixel neighborhoods to be taken into account, while also improving execution speed by an order of magnitude. Our primary contribution is the addition of recurrent connections to the network in order to drastically improve temporal stability for sequences of sparsely sampled input images. Our method also has the desirable property of automatically modeling relationships based on auxiliary per-pixel input channels, such as depth and normals. We show significantly higher quality results compared to existing methods that run at comparable speeds, and furthermore argue a clear path for making our method run at realtime rates in the near future.

Supplementary Material

ZIP File (a98-alla_chaitanya.zip)
Supplemental files.
MP4 File (papers-0088.mp4)

References

[1]
Bako, S., Vogels, T., McWilliams, B., Meyer, M., Novák, J., Harvill, A., Sen, P., DeRose, T., and Rousselle, F. 2017. Kernel-Predicting Convolutional Networks for Denoising Monte Carlo Renderings. ACM Trans. Graph. (2017).
[2]
Bauszat, P., Eisemann, M., John, S., and Magnor, M. 2015. Sample-Based Manifold Filtering for Interactive Global Illumination and Depth of Field. Comp. Graph. Forum 34, 1 (2015).
[3]
Bauszat, P., Eisemann, M., and Magnor, M. 2011. Guided Image Filtering for Interactive High-quality Global Illumination (Comp. Graph. Forum). 8.
[4]
Bikker, J. and van Schijndel, J. 2013. The brigade renderer: A path tracer for real-time games. J. Comp. Games Tech. 8 (2013).
[5]
Bitterli, B., Rousselle, F., Moon, B., Iglesias-Guitián, J. A., Adler, D., Mitchell, K., Jarosz, W., and Novák, J. 2016. Nonlinearly Weighted First-order Regression for Denoising Monte Carlo Renderings. Comp. Graph. Forum 35, 4 (2016).
[6]
Dammertz, H., Sewtz, D., Hanika, J., and Lensch, H. 2010. Edge-avoiding À-Trous wavelet transform for fast global illumination filtering. In Proc. High Perf. Graph.
[7]
Deering, M., Winner, S., Schediwy, B., Duffy, C., and Hunt, N. 1988. The Triangle Processor and Normal Vector Shader: A VLSI System for High Performance Graphics. Proc. SIGGRAPH 22, 4 (1988).
[8]
Dieleman, S., Schlüter, J., Raffel, C., Olson, E., Sønderby, S. K., and others. 2015. Lasagne: First release. (2015).
[9]
Dosovitskiy, A., Springenberg, J. T., and Brox, T. 2015. Learning to generate chairs with convolutional neural networks. In Proc. CVPR. 1538--1546.
[10]
Egan, K., Durand, F., and Ramamoorthi, R. 2011a. Practical Filtering for Efficient Ray-traced Directional Occlusion. ACM Trans. Graph. 30, 6 (2011).
[11]
Egan, K., Hecht, F., Durand, F., and Ramamoorthi, R. 2011b. Frequency Analysis and Sheared Filtering for Shadow Light Fields of Complex Occluders. ACM Trans. Graph. 30, 2 (2011).
[12]
Egan, K., Tseng, Y.-T., Holzschuch, N., Durand, F., and Ramamoorthi, R. 2009. Frequency Analysis and Sheared Reconstruction for Rendering Motion Blur. ACM Trans. Graph. 28, 3 (2009).
[13]
Fischer, P., Dosovitskiy, A., Ilg, E., Häusser, P., Hazirbas, C., Golkov, V., van der Smagt, P., Cremers, D., and Brox, T. 2015. FlowNet: Learning Optical Flow with Convolutional Networks. ArXiv e-prints (2015). arXiv:1504.06852.
[14]
Flynn, J., Neulander, I., Philbin, J., and Snavely, N. 2016. DeepStereo: Learning to Predict New Views From the World's Imagery. In Proc. CVPR.
[15]
Gastal, E. S. and Oliveira, M. M. 2012. Adaptive manifolds for real-time high-dimensional filtering. ACM Trans. Graph. 31, 4 (2012).
[16]
Girshick, R., Donahue, J., Darrell, T., and Malik, J. 2014. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In Proc. CVPR. 580--587.
[17]
Halton, J. H. 1964. Algorithm 247: Radical-inverse Quasi-random Point Sequence. Comm. ACM 7, 12 (1964).
[18]
Han, S., Liu, X., Mao, H., Pu, J., Pedram, A., Horowitz, M. A., and Dally, W. J. 2016. EIE: Efficient Inference Engine on Compressed Deep Neural Network. In Proc. Comp. Arch.
[19]
Hanika, J., Dammertz, H., and Lensch, H. 2011. Edge-Optimized À-Trous Wavelets for Local Contrast Enhancement with Robust Denoising. Comp. Graph. Forum 30, 7 (2011).
[20]
He, K., Zhang, X., Ren, S., and Sun, J. 2015a. Deep Residual Learning for Image Recognition. ArXiv e-prints (2015). arXiv:1512.03385.
[21]
He, K., Zhang, X., Ren, S., and Sun, J. 2015b. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In Proc. IEEE Conf. Comp. Vis.
[22]
Hill, S., McAuley, S., Burley, B., Chan, D., Fascione, L., Iwanicki, M., Hoffman, N., Jakob, W., Neubelt, D., Pesce, A., and Pettineo, M. 2015. Physically Based Shading in Theory and Practice. In SIGGRAPH Courses. Article 22.
[23]
Hill, S., McAuley, S., Dupuy, J., Gotanda, Y., Heitz, E., Hoffman, N., Lagarde, S., Langlands, A., Megibben, I., Rayani, F., and de Rousiers, C. 2014. Physically Based Shading in Theory and Practice. In SIGGRAPH Courses. Article 23.
[24]
Huang, Y., Wang, W., and Wang, L. 2015. Bidirectional recurrent convolutional networks for multi-frame super-resolution. In Proc. NIPS.
[25]
Iizuka, S., Simo-Serra, E., and Ishikawa, H. 2016. Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM Trans. Graph. 35, 4 (2016).
[26]
Jensen, H. W. and Christensen, N. J. 1995. Optimizing Path Tracing using Noise Reduction Filters. In Proc. WSCG.
[27]
Kajiya, J. T 1986. The rendering equation. In Proc. SIGGRAPH.
[28]
Kalantari, N. K., Bako, S., and Sen, P. 2015. A machine learning approach for filtering Monte Carlo noise. ACM Trans. Graph. 34, 4 (2015).
[29]
Kaplanyan, A. S. and Dachsbacher, C. 2013. Path Space Regularization for Holistic and Robust Light Transport. Comp. Graph. Forum 32, 2 (2013).
[30]
Karis, B. 2014. High-quality temporal supersampling. In Advances in Real-Time Rendering in Games, SIGGRAPH Courses.
[31]
Keller, A., Fascione, L., Fajardo, M., Georgiev, I., Christensen, P., Hanika, J., Eisenacher, C., and Nichols, G. 2015. The path tracing revolution in the movie industry. SIGGRAPH Courses (2015).
[32]
Kingma, D. and Ba, J. 2014. Adam: A Method for Stochastic Optimization. ArXiv e-prints (2014). arXiv:1412.6980.
[33]
Krizhevsky, A., Sutskever, I., and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. In Proc. NIPS.
[34]
Larsson, G., Maire, M., and Shakhnarovich, G. 2016. Learning Representations for Automatic Colorization. In Proc. ECCV.
[35]
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. 1998. Gradient-based learning applied to document recognition. In Proc. IEEE.
[36]
Ledig, C., Theis, L., Huszar, F., Caballero, J., Aitken, A. P., Tejani, A., Totz, J., Wang, Z., and Shi, W. 2016. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. ArXiv e-prints (2016). arXiv:1609.04802.
[37]
Lee, M. E. and Redner, R. A. 1990. A note on the use of nonlinear filtering in computer graphics. IEEE Comp. Graph. App. 10, 3 (1990).
[38]
Lehtinen, J., Aila, T., Laine, S., and Durand, F. 2012. Reconstructing the indirect light field for global illumination. ACM Trans. Graph. 31, 4 (2012).
[39]
Li, T.-M., Wu, Y.-T., and Chuang, Y.-Y. 2012. SURE-based Optimization for Adaptive Sampling and Reconstruction. ACM Trans. Graph. 31, 6 (2012).
[40]
Maas, A. L., Hannun, A. Y., and Ng, A. Y. 2013. Rectifier nonlinearities improve neural network acoustic models. In International Conference on Machine Learning, Vol. 30.
[41]
Mao, X., Shen, C., and Yang, Y. 2016. Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections. In Proc. NIPS. arXiv:1606.08921.
[42]
McCool, M. D. 1999. Anisotropic diffusion for Monte Carlo noise reduction. ACM Trans. Graph. 18, 2 (1999).
[43]
Mehta, S. U., Wang, B., and Ramamoorthi, R. 2012. Axis-aligned Filtering for Interactive Sampled Soft Shadows. ACM Trans. Graph. 31, 6 (2012).
[44]
Mehta, S. U., Wang, B., Ramamoorthi, R., and Durand, F. 2013. Axis-aligned Filtering for Interactive Physically-based Diffuse Indirect Lighting. ACM Trans. Graph. 32, 4 (2013).
[45]
Mehta, S. U., Yao, J., Ramamoorthi, R., and Durand, F. 2014. Factored Axis-aligned Filtering for Rendering Multiple Distribution Effects. ACM Trans. Graph. 33, 4 (2014).
[46]
Mishkin, D., Sergievskiy, N., and Matas, J. 2016. Systematic evaluation of CNN advances on the ImageNet. ArXiv e-prints (2016). arXiv:1606.02228.
[47]
Moon, B., Iglesias-Guitian, J. A., Yoon, S.-E., and Mitchell, K. 2015. Adaptive rendering with linear predictions. ACM Trans. Graph. 34, 4 (2015).
[48]
Moon, B., McDonagh, S., Mitchell, K., and Gross, M. 2016. Adaptive polynomial rendering. ACM Trans. Graph. 35, 4 (2016).
[49]
Munkberg, J., Hasselgren, J., Clarberg, P., Andersson, M., and Akenine-Möller, T 2016. Texture space caching and reconstruction for ray tracing. ACM Trans. Graph. 35, 6 (2016).
[50]
Parker, S. G., Bigler, J., Dietrich, A., Friedrich, H., Hoberock, J., Luebke, D., McAllister, D., McGuire, M., Morley, K., Robison, A., and Stich, M. 2010. OptiX: A General Purpose Ray Tracing Engine. ACM Trans. Graph. 29, 4 (2010).
[51]
Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., and Efros, A. 2016. Context Encoders: Feature Learning by Inpainting. In Proc. CVPR.
[52]
Patraucean, V., Handa, A., and Cipolla, R. 2015. Spatio-temporal video autoencoder with differentiable memory. ArXiv e-prints (2015). arXiv:1511.06309.
[53]
Raiko, T., Valpola, H., and Lecun, Y. 2012. Deep Learning Made Easier by Linear Transformations in Perceptrons, In Proc. Artificial Intelligence and Statistics. J. of Machine Learning Research 22.
[54]
Ravishankar, S. and Bresler, Y. 2011. MR Image Reconstruction From Highly Undersampled k-Space Data by Dictionary Learning. IEEE Trans. Med. Imaging 30, 5 (2011), 1028--1041.
[55]
Robison, A. and Shirley, P. 2009. Image space gathering. In Proc. High Perf Graph.
[56]
Ronneberger, O., Fischer, P., and Brox, T. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. 234--241.
[57]
Rousselle, F., Manzi, M., and Zwicker, M. 2013. Robust Denoising using Feature and Color Information. Comp. Graph. Forum 32, 7 (2013).
[58]
Schmidhuber, J. 2015. Deep Learning in Neural Networks: An Overview. Neural Networks 61 (2015).
[59]
Schwenk, K. 2013. Filtering techniques for low-noise previews of interactive stochastic ray tracing. Ph.D. Dissertation. Darmstadt University.
[60]
Sen, P. and Darabi, S. 2012. On filtering the noise from the random parameters in Monte Carlo rendering. ACM Trans. Graph. 31, 3 (2012).
[61]
Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A. P., Bishop, R., Rueckert, D., and Wang, Z. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proc. CVPR.
[62]
Stein, C. M. 1981. Estimation of the mean of a multivariate normal distribution. In Annals of Stat., Vol. 9.
[63]
Theano Development Team. 2016. Theano: A Python framework for fast computation of mathematical expressions. ArXiv e-prints (2016). arXiv:1605.02688.
[64]
Veach, E. 1998. Robust Monte Carlo Methods for Light Transport Simulation. Ph.D. Dissertation. Advisor(s) Guibas, Leonidas J.
[65]
Vincent, P., Larochelle, H., Bengio, Y, and Manzagol, P.-A. 2008. Extracting and Composing Robust Features with Denoising Autoencoders. Proc. Machine Learning (2008).
[66]
Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. 2004. Image Quality Assessment: From Error Visibility to Structural Similarity. Trans. Img. Proc. 13, 4 (2004), 600--612.
[67]
Werbos, P. J. 1988. Generalization of backpropagation with application to a recurrent gas market model. Neural Networks 1, 4 (1988).
[68]
Yan, L.-Q., Mehta, S. U., Ramamoorthi, R., and Durand, F. 2015. Fast 4D Sheared Filtering for Interactive Rendering of Distribution Effects. ACM Trans. Graph. 35, 1 (2015).
[69]
Zhao, H., Gallo, O., Frosio, I., and Kautz, J. 2016. Loss Functions for Image Restoration with Neural Networks. IEEE Transactions on Computational Imaging (2016).
[70]
Zimmer, H., Rousselle, F., Jakob, W., Wang, O., Adler, D., Jarosz, W., Sorkine-Hornung, O., and Sorkine-Hornung, A. 2015. Path-space Motion Estimation and Decomposition for Robust Animation Filtering. Comp. Graph. Forum 34, 4 (2015).
[71]
Zwicker, M., Jarosz, W., Lehtinen, J., Moon, B., Ramamoorthi, R., Rousselle, F., Sen, P., Soler, C., and Yoon, S.-E. 2015. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. Comp. Graph. Forum 34, 2 (2015).

Cited By

View all
  • (2024)High-quality real-time rendering using subpixel sampling reconstructionProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i7.28527(7006-7014)Online publication date: 20-Feb-2024
  • (2024)Online Neural Denoising with Cross-Regression for Interactive RenderingACM Transactions on Graphics10.1145/368793843:6(1-12)Online publication date: 19-Dec-2024
  • (2024)A Statistical Approach to Monte Carlo DenoisingSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687591(1-11)Online publication date: 3-Dec-2024
  • Show More Cited By

Index Terms

  1. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Transactions on Graphics
        ACM Transactions on Graphics  Volume 36, Issue 4
        August 2017
        2155 pages
        ISSN:0730-0301
        EISSN:1557-7368
        DOI:10.1145/3072959
        Issue’s Table of Contents
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 20 July 2017
        Published in TOG Volume 36, Issue 4

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. Monte Carlo denoising
        2. image reconstruction
        3. interactive global illumination
        4. machine learning

        Qualifiers

        • Research-article

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)211
        • Downloads (Last 6 weeks)12
        Reflects downloads up to 17 Jan 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)High-quality real-time rendering using subpixel sampling reconstructionProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i7.28527(7006-7014)Online publication date: 20-Feb-2024
        • (2024)Online Neural Denoising with Cross-Regression for Interactive RenderingACM Transactions on Graphics10.1145/368793843:6(1-12)Online publication date: 19-Dec-2024
        • (2024)A Statistical Approach to Monte Carlo DenoisingSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687591(1-11)Online publication date: 3-Dec-2024
        • (2024)Radiance Caching with On-Surface Caches for Real-Time Global IlluminationProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36753827:3(1-17)Online publication date: 9-Aug-2024
        • (2024)Optimizing Path Termination for Radiance Caching Through Explicit Variance TradingProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36753817:3(1-19)Online publication date: 9-Aug-2024
        • (2024)A Fast GPU Schedule For À-Trous Wavelet-Based DenoisersProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/36512997:1(1-18)Online publication date: 13-May-2024
        • (2024)Online Neural Path Guiding with Normalized Anisotropic Spherical GaussiansACM Transactions on Graphics10.1145/364931043:3(1-18)Online publication date: 9-Apr-2024
        • (2024)Dual Channel Residual Learning for Denoising Path TracingInternational Journal of Image and Graphics10.1142/S021946782550003225:01Online publication date: 15-Jan-2024
        • (2024)CrystalNet: Texture‐Aware Neural Refraction Baking for Global IlluminationComputer Graphics Forum10.1111/cgf.1522743:7Online publication date: 24-Oct-2024
        • (2024)Residual path integrals for re‐renderingComputer Graphics Forum10.1111/cgf.1515243:4Online publication date: 24-Jul-2024
        • Show More Cited By

        View Options

        Login options

        Full Access

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media