Skip to main content
Springer Nature Link
Log in

Unsupervised Reconstruction for Gradient-Domain Rendering with Illumination Separation

  • Regular Paper
  • Artificial Intelligence and Pattern Recognition
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

Gradient-domain rendering methods can render higher-quality images at the same time cost compared with traditional ray tracing rendering methods, and, combined with the neural network, achieve better rendering quality than conventional screened Poisson reconstruction. However, it is still challenging for these methods to keep detailed information, especially in areas with complex indirect illumination and shadows. We propose an unsupervised reconstruction method that separates the direct rendering from the indirect, and feeds them into our unsupervised network with some corresponding auxiliary channels as two separated tasks. In addition, we introduce attention modules into our network which can further improve details. We finally combine the results of the direct and indirect illumination tasks to form the rendering results. Experiments show that our method significantly improves image quality details, especially in scenes with complex conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Manzi M, Kettunen M, Aittala M, Lehtinen J, Durand F, Zwicker M. Gradient-domain bidirectional path tracing. In Proc. the Eurographics Symposium on Rendering 2015, Jun. 2015, DOI: https://doi.org/10.2312/sre.20151168.

    MATH  Google Scholar 

  2. Guo J, Li M, Li Q, Qiang Y, Hu B, Guo Y, Yan L Q. Gradnet: Unsupervised deep screened Poisson reconstruction for gradient-domain rendering. ACM Trans. Graphics, 2019, 38(6): 223. DOI: https://doi.org/10.1145/3355089.3356538.

    Article  MATH  Google Scholar 

  3. Kettunen M, Härkönen E, Lehtinen J. Deep convolutional reconstruction for gradient-domain rendering. ACM Trans. Graphics, 2019, 38 (4): Article No. 126. DOI: https://doi.org/10.1145/3306346.3323038.

  4. Xu Z, Sun Q, Wang L, Xu Y, Wang B. Unsupervised image reconstruction for gradient-domain volumetric rendering. Computer Graphics Forum, 2020, 39(7): 193–203. DOI: https://doi.org/10.1111/cgf.14137.

    Article  MATH  Google Scholar 

  5. Schied C, Kaplanyan A, Wyman C, Patney A, Chaitanya C R A, Burgess J, Liu S, Dachsbacher C, Lefohn A, Salvi M. Spatiotemporal variance-guided filtering: Real-time reconstruction for path-traced global illumination. In Proc. the 2017 High Performance Graphics, Jul. 2017, Article No. 2. DOI: https://doi.org/10.1145/3105762.3105770.

    Google Scholar 

  6. Lehtinen J, Karras T, Laine S, Aittala M, Durand F, Aila T. Gradient-domain metropolis light transport. ACM Trans. Graphics, 2013, 32 (4): Article No. 95. DOI: https://doi.org/10.1145/2461912.2461943.

  7. Kettunen M, Manzi M, Aittala M, Lehtinen J, Durand F, Zwicker M. Gradient-domain path tracing. ACM Trans. Graphics, 2015, 34 (4): Article No. 123. DOI: https://doi.org/10.1145/2766997.

  8. Bhat P, Curless B, Cohen M, Zitnick C L. Fourier analysis of the 2D screened Poisson equation for gradient domain problems. In Computer Vision-ECCV 2008, Forsyth D, Torr P, Zisserman A (eds.), Springer, 2008, pp.114–128. DOI: https://doi.org/10.1007/978-3-540-88688-4_9.

    Chapter  Google Scholar 

  9. Rousselle F, Jarosz W, Novak J. Image-space control variates for rendering. ACM Trans. Graphics, 2016, 35 (6): Article No. 169. DOI: https://doi.org/10.1145/2980179.2982443.

  10. Manzi M, Vicini D, Zwicker M. Regularizing image reconstruction for gradient-domain rendering with feature patches. Computer Graphics Forum, 2016, 35(2): 263–273. DOI: https://doi.org/10.1111/cgf.12829.

    Article  MATH  Google Scholar 

  11. Back J, Yoon S E, Moon B. Feature generation for adaptive gradient-domain path tracing. Computer Graphics Forum, 2018, 37(7): 65–74. DOI: https://doi.org/10.1111/cgf.13548.

    Article  MATH  Google Scholar 

  12. Ha S, Oh S, Back J, Yoon S E, Moon B. Gradient outlier removal for gradient-domain path tracing. Computer Graphics Forum, 2019, 38(2): 245–253. DOI: https://doi.org/10.1111/cgf.13634.

    Article  MATH  Google Scholar 

  13. He H, Zhao Y, Liu H. Gradient-domain path tracing with gain control strategy. Journal of Computer-Aided Design & Computer Graphics, 2022, 34(2): 294–304. DOI: https://doi.org/10.3724/SP.J.1089.2022.18837. (in Chinese)

    Article  MATH  Google Scholar 

  14. Bako S, Vogels T, Mcwilliams B, Meyer M, NovaK J, Harvill A, Sen P, Derose T, Rousselle F. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Trans. Graphics, 2017, 36 (4): Article No. 97. DOI: https://doi.org/10.1145/3072959.3073708.

  15. Chaitanya C R A, Kaplanyan A S, Schied C, Salvi M, Lefohn A, Nowrouzezahrai D, Aila T. Interactive reconstruction of Monte Carlo image sequences using a recurrent denoising autoencoder. ACM Trans. Graphics, 2017, 36 (4): Article No. 98. DOI: https://doi.org/10.1145/3072959.3073601.

  16. Vogels T, Rousselle F, Mcwilliams B, Röthlin G, Harvill A, Adler D, Meyer M, Novák J. Denoising with kernel predictionand asymmetric loss functions. ACM Trans. Graph., 2018, 37 (4): Article No. 124. DOI: https://doi.org/10.1145/3197517.3201388.

  17. Suppan C, Chalmers A, Zhao J, Doronin A, Rhee T. Neural screen space rendering of direct illumination. In Proc. the 29th Pacific Conference on Computer Graphics and Applications, Oct. 2021, pp.37–42. DOI: https://doi.org/10.2312/pg.20211385.

    MATH  Google Scholar 

  18. Han K B, Odenthal O G, Kim W J, Yoon S E. Pixel-wise guidance for utilizing auxiliary features in Monte Carlo Denoising. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 2023, 6 (1): Article No. 11. DOI: https://doi.org/10.1145/3585505.

  19. Alpay K C, Akyuz A O. Denoising and guided upsampling of Monte Carlo path traced low resolution renderings. In Proc. the 2022 ACM SIGGRAPH Posters, Aug. 2022, Article No. 37. DOI: https://doi.org/10.1145/3532719.3543250.

    MATH  Google Scholar 

  20. Zhang X, Ott M, Manzi M, Gross M, Papas M. Automatic feature selection for denoising volumetric renderings. Computer Graphics Forum, 2022, 41(4): 63–77. DOI: https://doi.org/10.1111/cgf.14587.

    Article  Google Scholar 

  21. Firmino A, Frisvad J R, Jensen H W. Progressive denoising of Monte Carlo rendered images. Computer Graphics Forum, 2022, 41(2): 1–11. DOI: https://doi.org/10.1111/cgf.14454.

    Article  MATH  Google Scholar 

  22. Currius R R, Assarsson U, Sintorn E. Real-time hair filtering with convolutional neural networks. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 2022, 5 (1): Article No. 15. DOI: https://doi.org/10.1145/3522606.

  23. Manzi M, Kettunen M, Durand F, Zwicker M, Lehtinen J. Temporal gradient-domain path tracing. ACM Trans. Graphics, 2016, 35 (6): Article No. 246. DOI: https://doi.org/10.1145/2980179.2980256.

  24. Hua B S, Gruson A, Nowrouzezahrai D, Hachisuka T. Gradient-domain photon density estimation. Computer Graphics Forum, 2017, 36(2): 31–38. DOI: https://doi.org/10.1111/cgf.13104.

    Article  Google Scholar 

  25. Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015, Navab N, Hornegger J, Wells W M, Frangi A F (eds.), Springer, 2015, pp.234–241. DOI: https://doi.org/10.1007/978-3-319-24574-4_28.

    MATH  Google Scholar 

  26. Woo S, Park J, Lee J Y, Kweon I S. CBAM: Convolutional block attention module. In Computer Vision—ECCV 2018, Ferrari V, Hebert M, Sminchisescu C, Weiss Y (eds.), Springer, 2018, pp.3–19. DOI: https://doi.org/10.1007/978-3-030-01234-2_1.

    Chapter  Google Scholar 

  27. Kingma D P, Ba J. Adam: A method for stochastic optimization. In Proc. the 3rd International Conference on Learning Representations, May 2017.

    MATH  Google Scholar 

  28. He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proc. the 2015 IEEE International Conference on Computer Vision (ICCV), Dec. 2015, pp.1026–1034. DOI: https://doi.org/10.1109/ICCV.2015.123.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Software, Shandong University, Jinan, 250101, China

    Ming-Cong Ma  (马鸣聪), Lu Wang  (王 璐), Yan-Ning Xu  (徐延宁) & Xiang-Xu Meng  (孟祥旭)

Authors
  1. Ming-Cong Ma  (马鸣聪)
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lu Wang  (王 璐)
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yan-Ning Xu  (徐延宁)
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiang-Xu Meng  (孟祥旭)
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lu Wang  (王 璐).

Ethics declarations

Conflict of Interest The authors declare that they have no conflict of interest.

Additional information

This work has been partially supported by the National Key Research and Development Program of China under Grant No. 2020YFB1708900 and the National Natural Science Foundation of China under Grant No. 62272275.

Ming-Cong Ma received his M.S. degree in computer science and technology from Shandong University, Jinan, in 2020. He currently is a Ph.D. candidate in the School of Software, Shandong University, Jinan. His research interests include machine learning and image denoising.

Lu Wang received her Ph.D. degree in computer science from Shandong University, Jinan, in 2009. She is a professor in the School of Software, Shandon University, Jinan. Her research interests include photorealistic rendering and high performance rendering.

Yan-Ning Xu received his Ph.D. degree in computer science from Shandong University, Jinan, in 2006. He is an associate professor in the School of Software, Shandong University, Jinan. His research interests include photorealistic rendering and computer aided design.

Xiang-Xu Meng received his Ph.D. degree in computer science from the Institute of Computing Technology, Chinese Academy of Sciences, Beijing, in 1998. He is a professor in the School of Software, Shandong University, Jinan. His current research interests include human-computer interaction, virtual reality, computer graphics, and visualization.

Electronic Supplementary Material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ma, MC., Wang, L., Xu, YN. et al. Unsupervised Reconstruction for Gradient-Domain Rendering with Illumination Separation. J. Comput. Sci. Technol. 39, 1281–1291 (2024). https://doi.org/10.1007/s11390-024-3142-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-024-3142-4

Keywords