research-article

Weakly-supervised contrastive learning in path manifold for Monte Carlo image reconstruction

Authors:

Sung-Eui YoonAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 40, Issue 4

Article No.: 38, Pages 1 - 14

https://doi.org/10.1145/3450626.3459876

Published: 19 July 2021 Publication History

Abstract

Image-space auxiliary features such as surface normal have significantly contributed to the recent success of Monte Carlo (MC) reconstruction networks. However, path-space features, another essential piece of light propagation, have not yet been sufficiently explored. Due to the curse of dimensionality, information flow between a regression loss and high-dimensional path-space features is sparse, leading to difficult training and inefficient usage of path-space features in a typical reconstruction framework. This paper introduces a contrastive manifold learning framework to utilize path-space features effectively. The proposed framework employs weakly-supervised learning that converts reference pixel colors to dense pseudo labels for light paths. A convolutional path-embedding network then induces a low-dimensional manifold of paths by iteratively clustering intra-class embeddings, while discriminating inter-class embeddings using gradient descent. The proposed framework facilitates path-space exploration of reconstruction networks by extracting low-dimensional yet meaningful embeddings within the features. We apply our framework to the recent image- and sample-space models and demonstrate considerable improvements, especially on the sample space. The source code is available at https://github.com/Mephisto405/WCMC.

Supplementary Material

VTT File (3450626.3459876.vtt)

Download
15.02 KB

ZIP File (a38-cho.zip)

a38-cho.zip

Download
480.10 MB

MP4 File (3450626.3459876.mp4)

Presentation.

Download
166.47 MB

References

[1]

Martín Abadi and et al. 2016. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467 (2016).

[2]

Jonghee Back, Sung-Eui Yoon, and Bochang Moon. 2018. Feature Generation for Adaptive Gradient-Domain Path Tracing. Computer Graphics Forum 37, 7 (2018), 65--74.

[3]

Steve Bako, Mark Meyer, Tony DeRose, and Pradeep Sen. 2019. Offline deep importance sampling for Monte Carlo path tracing. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 527--542.

[4]

Steve Bako, Thijs Vogels, Brian Mcwilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Transactions on Graphics (TOG) 36, 4 (2017), 97.

Digital Library

[5]

Elena Balashova, Amit H. Bermano, Vladimir G. Kim, Stephen DiVerdi, Aaron Hertzmann, and Thomas Funkhouser. 2019. Learning A Stroke-Based Representation for Fonts. Computer Graphics Forum 38, 1 (2019), 429--442.

[6]

Benedikt Bitterli. 2016. Rendering resources. https://benedikt-bitterli.me/resources/.

[7]

Leo Breiman. 2001. Random forests. Machine learning 45, 1 (2001), 5--32.

Digital Library

[8]

Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. 2005. A review of image denoising algorithms, with a new one. Multiscale Modeling & Simulation 4, 2 (2005), 490--530.

[9]

Chakravarty R Alla Chaitanya, Anton S Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Transactions on Graphics (TOG) 36, 4 (2017), 98.

[10]

Shixing Chen, Caojin Zhang, Ming Dong, Jialiang Le, and Mike Rao. 2017. Using Ranking-CNN for Age Estimation. In CVPR.

[11]

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 1597--1607.

[12]

Mauricio Delbracio, Pablo Musé, Antoni Buades, Julien Chauvier, Nicholas Phelps, and Jean-Michel Morel. 2014. Boosting monte carlo rendering by ray histogram fusion. ACM Transactions on Graphics (TOG) 33, 1 (2014), 1--15.

Digital Library

[13]

Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019. ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In CVPR.

[14]

Elmar Eisemann and Frédo Durand. 2004. Flash photography enhancement via intrinsic relighting. ACM transactions on graphics (TOG) 23, 3 (2004), 673--678.

[15]

Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand. 2016. Deep joint demosaicking and denoising. ACM Transactions on Graphics (TOG) 35, 6 (2016), 1--12.

Digital Library

[16]

Michaël Gharbi, Tzu-Mao Li, Miika Aittala, Jaakko Lehtinen, and Frédo Durand. 2019. Sample-based Monte Carlo denoising using a kernel-splatting network. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--12.

Digital Library

[17]

Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics. 249--256.

[18]

Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality Reduction by Learning an Invariant Mapping. In CVPR.

[19]

Johannes Hanika, Marc Droske, and Luca Fascione. 2015a. Manifold next event estimation. Computer Graphics Forum 34, 4 (2015), 87--97.

Digital Library

[20]

Johannes Hanika, Anton Kaplanyan, and Carsten Dachsbacher. 2015b. Improved half vector space light transport. Computer Graphics Forum 34, 4 (2015), 65--74.

Digital Library

[21]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In CVPR.

[22]

Paul S Heckbert. 1990. Adaptive radiosity textures for bidirectional ray tracing. In Computer graphics and interactive techniques.

[23]

Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. 2007. Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments. Technical Report 07--49. University of Massachusetts, Amherst.

[24]

Yuchi Huo, Rui Wang, Ruzahng Zheng, Hualin Xu, Hujun Bao, and Sung-Eui Yoon. 2020. Adaptive Incident Radiance Field Sampling and Reconstruction Using Deep Reinforcement Learning. ACM Transactions on Graphics (TOG) 39, 1 (2020), 1--17.

Digital Library

[25]

Woobin Im, Sungeun Hong, Sung-Eui Yoon, and Hyun S Yang. 2018. Scale-Varying Triplet Ranking with Classification Loss for Facial Age Estimation. In ACCV. 247--259.

[26]

Wenzel Jakob and Steve Marschner. 2012. Manifold exploration: a Markov Chain Monte Carlo technique for rendering scenes with difficult specular transport. ACM Transactions on Graphics (TOG) 31, 4 (2012), 1--13.

Digital Library

[27]

James T Kajiya. 1986. The rendering equation. In Proceedings of the 13th annual conference on Computer graphics and interactive techniques. 143--150.

Digital Library

[28]

Nima Khademi Kalantari, Steve Bako, and Pradeep Sen. 2015. A machine learning approach for filtering Monte Carlo noise. ACM Transactions on Graphics (TOG) 34, 4 (2015), 122.

Digital Library

[29]

Anton S Kaplanyan, Johannes Hanika, and Carsten Dachsbacher. 2014. The natural-constraint representation of the path space for efficient light transport simulation. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1--13.

Digital Library

[30]

Mahmut Kaya and Hasan şakir Bilge. 2019. Deep metric learning: A survey. Symmetry 11, 9 (2019), 1066.

[31]

Markus Kettunen, Erik Härkönen, and Jaakko Lehtinen. 2019. Deep convolutional reconstruction for gradient-domain rendering. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--12.

Digital Library

[32]

Markus Kettunen, Marco Manzi, Miika Aittala, Jaakko Lehtinen, Frédo Durand, and Matthias Zwicker. 2015. Gradient-domain path tracing. ACM Transactions on Graphics (TOG) 34, 4 (2015), 1--13.

Digital Library

[33]

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. arXiv preprint arXiv:2004.11362 (2020).

[34]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[35]

Weiheng Lin, Beibei Wang, Jian Yang, Lu Wang, and Ling-Qi Yan. 2021. Path-based Monte Carlo Denoising Using a Three-Scale Neural Network. Computer Graphics Forum (2021).

[36]

Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, Nov (2008), 2579--2605.

[37]

Bochang Moon, Nathan Carr, and Sung-Eui Yoon. 2014. Adaptive rendering based on weighted local regression. ACM Transactions on Graphics (TOG) 33, 5 (2014), 1--14.

Digital Library

[38]

Bochang Moon, Jong Yun Jun, JongHyeob Lee, Kunho Kim, Toshiya Hachisuka, and Sung-Eui Yoon. 2013. Robust Image Denoising Using a Virtual Flash Image for Monte Carlo Ray Tracing. Computer Graphics Forum 32, 1 (2013), 139--151.

[39]

Bochang Moon, Steven McDonagh, Kenny Mitchell, and Markus Gross. 2016. Adaptive polynomial rendering. ACM Transactions on Graphics (TOG) 35, 4 (2016), 40.

Digital Library

[40]

Thomas Müller, Markus Gross, and Jan Novák. 2017. Practical path guiding for efficient light-transport simulation. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 91--100.

[41]

Thomas Müller, Brian McWilliams, Fabrice Rousselle, Markus Gross, and Jan Novák. 2019. Neural importance sampling. ACM Transactions on Graphics (TOG) 38, 5 (2019), 1--19.

Digital Library

[42]

Jacob Munkberg and Jon Hasselgren. 2020. Neural Denoising with Layer Embeddings. Computer Graphics Forum 39, 4 (2020), 1--12.

[43]

Steven G. Parker, Heiko Friedrich, David Luebke, Keith Morley, James Bigler, Jared Hoberock, David McAllister, Austin Robison, Andreas Dietrich, Greg Humphreys, Morgan McGuire, and Martin Stich. 2013. GPU Ray Tracing. Commun. ACM 56, 5 (May 2013), 93--101.

Digital Library

[44]

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. (2017).

[45]

Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda. 2002. Photographic tone reproduction for digital images. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques. 267--276.

Digital Library

[46]

Fabrice Rousselle, Marco Manzi, and Matthias Zwicker. 2013. Robust denoising using feature and color information. In Computer Graphics Forum, Vol. 32. Wiley Online Library, 121--130.

[47]

David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1986. Learning representations by back-propagating errors. nature 323, 6088 (1986), 533--536.

[48]

Yi Sun, Yuheng Chen, Xiaogang Wang, and Xiaoou Tang. 2014. Deep Learning Face Representation by Joint Identification-Verification. In Annual Conference on Neural Information Processing Systems 2014. Montreal, Quebec, Canada, 1988--1996.

[49]

Michael Tschannen, Josip Djolonga, Marvin Ritter, Aravindh Mahendran, Neil Houlsby, Sylvain Gelly, and Mario Lucic. 2020. Self-supervised learning of video-induced visual invariances. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13806--13815.

[50]

Berk Ulker, Sander Stuijk, Henk Corporaal, and Rob Wijnhoven. 2020. Reviewing inference performance of state-of-the-art deep learning frameworks. In Proceedings of the 23th International Workshop on Software and Compilers for Embedded Systems. 48--53.

Digital Library

[51]

Thijs Vogels, Fabrice Rousselle, Brian Mcwilliams, Gerhard Röthlin, Alex Harvill, David Adler, Mark Meyer, and Jan Novák. 2018. Denoising with kernel prediction and asymmetric loss functions. ACM Transactions on Graphics (TOG) 37, 4 (2018), 124.

Digital Library

[52]

Jiří Vorba, Ondřej Karlík, Martin Šik, Tobias Ritschel, and Jaroslav Křivánek. 2014. On-line learning of parametric mixture models for light transport simulation. ACM Transactions on Graphics (TOG) 33, 4 (2014), 1--11.

Digital Library

[53]

Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin. 2017. Deep Metric Learning with Angular Loss. In IEEE International Conference on Computer Vision, ICCV 2017. IEEE Computer Society, Venice, Italy, 2612--2620.

[54]

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600--612.

Digital Library

[55]

Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl. 2017. Sampling matters in deep embedding learning. In Proceedings of the IEEE International Conference on Computer Vision. 2840--2848.

[56]

Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018. Unsupervised feature learning via non-parametric instance discrimination. In CVPR.

[57]

Bing Xu, Junfei Zhang, Rui Wang, Kun Xu, Yong-Liang Yang, Chuan Li, and Rui Tang. 2019. Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation. ACM Transactions on Graphics (TOG) 38, 6 (2019), 224--1.

Digital Library

[58]

Tizian Zeltner, Iliyan Georgiev, and Wenzel Jakob. 2020. Specular manifold sampling for rendering high-frequency caustics and glints. ACM Transactions on Graphics (TOG) 39, 4 (2020), 149--1.

Digital Library

[59]

Quan Zheng and Matthias Zwicker. 2019. Learning to importance sample in primary sample space. In Computer Graphics Forum, Vol. 38. Wiley Online Library, 169--179.

[60]

Zhi-Hua Zhou. 2018. A brief introduction to weakly supervised learning. National science review 5, 1 (2018), 44--53.

[61]

Henning Zimmer, Fabrice Rousselle, Wenzel Jakob, Oliver Wang, David Adler, Wojciech Jarosz, Olga Sorkine-Hornung, and Alexander Sorkine-Hornung. 2015. Path-space motion estimation and decomposition for robust animation filtering. In Computer Graphics Forum, Vol. 34. Wiley Online Library, 131--142.

[62]

Károly Zsolnai-Fehér, Peter Wonka, and Michael Wimmer. 2018. Gaussian Material Synthesis. ACM Transactions on Graphics (TOG) 37, 4, Article 76 (July 2018), 14 pages.

Digital Library

Cited By

Chen RShi MHuang STan PKomura TChen X(2024)Taming Diffusion Probabilistic Models for Character ControlSpecial Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers '2410.1145/3641519.3657440(1-10)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657440
Oh GMoon B(2024)Joint self-attention for denoising Monte Carlo renderingThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-024-03446-840:7(4623-4634)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s00371-024-03446-8
Zhu QZhang HLan MHan L(2023)Neural Categorical Priors for Physics-Based Character ControlACM Transactions on Graphics10.1145/361839742:6(1-16)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618397
Show More Cited By

Index Terms

Weakly-supervised contrastive learning in path manifold for Monte Carlo image reconstruction
1. Computing methodologies
  1. Computer graphics
    1. Rendering
      1. Ray tracing
  2. Machine learning
    1. Learning paradigms
      1. Unsupervised learning
        Dimensionality reduction and manifold learning
    2. Machine learning approaches
      1. Neural networks

Recommendations

Riemannian Manifold Learning

Recently, manifold learning has been widely exploited in pattern recognition, data analysis, and machine learning. This paper presents a novel framework, called Riemannian manifold learning (RML), based on the assumption that the input high-dimensional ...
Global-guided weakly-supervised learning for multi-label image classification
Abstract
Multi-label classification with region-free labels is attracting increasing attention compared to that with region-based labels due to the time-consuming manual region-labeling process. Existing methods usually employ attention-based technology ...
Highlights
- The proposed methods designs a global-guided feature disentanglement architecture to improve the feature representation ability.
- The global correlations of multi-level features are used to guide the feature disentanglement of each ...
Deep semi-supervised learning with contrastive learning and partial label propagation for image data
Abstract
Deep semi-supervised learning is becoming an active research topic because it jointly utilizes labeled and unlabeled samples in training deep neural networks. Recent advances are mainly focused on inductive semi-supervised learning ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 40, Issue 4

August 2021

2170 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3450626

Editor:
Sylvain Paris
Adobe Inc.

Issue’s Table of Contents

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 July 2021

Published in TOG Volume 40, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
626
Total Downloads

Downloads (Last 12 months)62
Downloads (Last 6 weeks)4

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen RShi MHuang STan PKomura TChen X(2024)Taming Diffusion Probabilistic Models for Character ControlSpecial Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers '2410.1145/3641519.3657440(1-10)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657440
Oh GMoon B(2024)Joint self-attention for denoising Monte Carlo renderingThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-024-03446-840:7(4623-4634)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s00371-024-03446-8
Zhu QZhang HLan MHan L(2023)Neural Categorical Priors for Physics-Based Character ControlACM Transactions on Graphics10.1145/361839742:6(1-16)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618397
Zheng CHuo YMo SZhong ZWu ZHua WWang RBao H(2023)NeLT: Object-Oriented Neural Light TransferACM Transactions on Graphics10.1145/359649142:5(1-16)Online publication date: 29-Aug-2023
https://dl.acm.org/doi/10.1145/3596491
Balint MWolski KMyszkowski KSeidel HMantiuk R(2023)Neural Partitioning Pyramids for Denoising Monte Carlo RenderingsACM SIGGRAPH 2023 Conference Proceedings10.1145/3588432.3591562(1-11)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.1145/3588432.3591562
Han KOdenthal OKim WYoon S(2023)Pixel-wise Guidance for Utilizing Auxiliary Features in Monte Carlo DenoisingProceedings of the ACM on Computer Graphics and Interactive Techniques10.1145/35855056:1(1-19)Online publication date: 16-May-2023
https://dl.acm.org/doi/10.1145/3585505
Miao JCao FLi MYang BYe H(2023)Triplet teaching graph contrastive networks with self-evolving adaptive augmentationPattern Recognition10.1016/j.patcog.2023.109687142:COnline publication date: 1-Oct-2023
https://dl.acm.org/doi/10.1016/j.patcog.2023.109687
Starke SMason IKomura T(2022)DeepPhaseACM Transactions on Graphics10.1145/3528223.353017841:4(1-13)Online publication date: 22-Jul-2022
https://dl.acm.org/doi/10.1145/3528223.3530178
Zhang XOtt MManzi MGross MPapas M(2022)Automatic Feature Selection for Denoising Volumetric RenderingsComputer Graphics Forum10.1111/cgf.1458741:4(63-77)Online publication date: 30-Jul-2022
https://doi.org/10.1111/cgf.14587
Gopakumar MKim JChoi SPeng YWetzstein G(2021)Unfiltered holography: optimizing high diffraction orders without optical filtering for compact holographic displaysOptics Letters10.1364/OL.44285146:23(5822)Online publication date: 19-Nov-2021
https://doi.org/10.1364/OL.442851

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents