Skip to main content
SpringerLink
Log in

State of the Art on Deep Learning-enhanced Rendering Methods

  • Review
  • Published:
Machine Intelligence Research Aims and scope Submit manuscript

Abstract

Photorealistic rendering of the virtual world is an important and classic problem in the field of computer graphics. With the development of GPU hardware and continuous research on computer graphics, representing and rendering virtual scenes has become easier and more efficient. However, there are still unresolved challenges in efficiently rendering global illumination effects. At the same time, machine learning and computer vision provide real-world image analysis and synthesis methods, which can be exploited by computer graphics rendering pipelines. Deep learning-enhanced rendering combines techniques from deep learning and computer vision into the traditional graphics rendering pipeline to enhance existing rasterization or Monte Carlo integration renderers. This state-of-the-art report summarizes recent studies of deep learning-enhanced rendering in the computer graphics community. Specifically, we focus on works of renderers represented using neural networks, whether the scene is represented by neural networks or traditional scene files. These works are either for general scenes or specific scenes, which are differentiated by the need to retrain the network for new scenes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. C. Donner, H. W. Jensen. A spectral BSSRDF for shading human skin. In Proceedings of the 17th Eurographics conference on Rendering Techniques, Eurographics Association, Nicosia, Cyprus, pp. 409–417, 2006.

    Google Scholar 

  2. L. Q. Yan, M. Hašan, 13. Walter, S. Maschhner, R. Ramamoorthi. Rendering specular microgeometry with wave optics. ACM Transactions on Graphics, vol. 37, no. 4, Article number 75, 2018. DOI: https://doi.org/10.1145/3197517.3201351.

  3. L. Q. Yan, W. L. Sun, H. W. Jensen, R. Ramamoorthi. A BSSRDF model for efficient rendering of fur with global illumination. ACM Transactions on Graphics, vol. 36, no. 6, Article number 208, 2017. DOI: https://doi.org/10.1145/3130800.3130802.

  4. E. Veach, L. J. Guibas. Metropolis light transport. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, ACM, Los Angeles, USA, pp. 65–76, 1997. DOI: https://doi.org/10.1145/258734.258775.

    Google Scholar 

  5. M. Pauly, T. Kollig, A. Keller. Metropolis light transport for participating media. In Proceedings of Eurographics Workshop on Rendering Techniques, Springer, Brno, Czech Rppublic, pp 11–22, 2000. DOI: https://doi.org/10.1007/978-3-7091-6303-0_2.

    Google Scholar 

  6. Y. Ouyang, S. Liu, M. Kettunen, M. Pharr, J. Pantaleoni. ReSTIR GI: Path resampling for real-time path tracing. Computer Graphics Forum, vol. 40, no. 8, pp. 17–29, 2021. DOI: https://doi.org/10.1111/cgf.14378.

    Article  Google Scholar 

  7. M. McGuire, M Mara, D. Nowrouzezahrai, D. Luebke. Real-time global illumination using precomputed light field probes. In Proceedings of the 21st ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, San Francisco, USA, pp.2, 2017. DOI: https://doi.org/10.1145/30233683023378

  8. D. P. Fan, Z. L. Huang, P. Zheng, H. Liu, X. B. Qin, L. Van Gool. Facial-sketch synthesis: A new challenge. Machine Intelligence Research, vol. 19, no. 4, pp. 257–287, 2022. DOI: https://doi.org/10.1007/s11633-022-1349-9.

    Article  Google Scholar 

  9. I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, Canada, pp. 2672–2680, 2014.

  10. M. Arjovsky, S. Chintala, L. Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, pp. 214–223, 2017.

  11. J. Y. Zhu, T. Park, P. Isola, A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of IEEE International Conference on Computer Vision, Venice, Italy, pp. 2242–2251, 2017. DOI: https://doi.org/10.1109/ICCV.2017.244.

  12. D. P. Kingma, M. Welling. Auto-encoding variational Bayes. In Proceedings of the 2nd International Conference on Learning Representations, Banff, Canada, 2013. DOI: https://doi.org/10.48550/arXiv.1312.6114.

  13. Y. C. Pu, Z. Gan, R. Henao, X. Yuan, C. Y. Li, A. Stevens, L. Carin. Variational autoencoder for deep learning of images, labels and captions. In Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, Spain, pp. 2360–2368, 2016.

  14. A. Vahdat, J. Kautz. NVAE: A deep hierarchical variational autoencoder. Advances in neural information processing systems, vol. 33, pp. 19667–19679, 2020.

    Google Scholar 

  15. P. Isola, J. Y. Zhu, T. H. Zhou, A. A. Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, pp. 5967–5976, 2017. DOI: https://doi.org/10.1109/CVPR.2017.632.

  16. S. M. A. Eslami, D. Jimenez Rezende, F. Besse, F. Viola, A. S. Morcos, M. Garnelo, A. Ruderman, A. A. Rusu, I. Danihelka, K. Gregor, D. P. Reichert, L. Buesing, T. Weber, O. Vinyals, D. Rosenbaum, N. Rabinowitz, H. King, C. Hillier, M. Botvinick, D. Wierstra, K. Kavukcuoglu, D. Hassabis. Neural scene representation and rendering. Science, vol. 360, no. 6394, pp. 1204–1210, 2018. DOI: https://doi.org/10.1126/science.aar6170.

    Article  Google Scholar 

  17. A. Tewari, O. Fried, J. Thies, V. Sitzmann, S. Lombardi, K. Sunkavalli, R. Martin-brualla, T. Simon, J. Saragih, M. Nießner, R. Pandey, S. Fanello, G Wetzstein, J. Y. Zhu, C. Theobalt, M. Agrawala, E. Shechtman, D. B. Goldman, M. Zollhöfer. State of the art on neural rendering. Computer Graphics Forum, vol. 39, no. 2, pp. 701–727, 2020. DOI: https://doi.org/10.1111/cgf.14022.

    Article  Google Scholar 

  18. C. Zhang, T. Chen. A survey on image-based rendering–representation, samphng and compression. Signal Processing: Image Communication, vol. 19, no. 1, pp. 1–28, 2004. DOI: https://doi.org/10.1016/j.image.2003.07.001.

    Google Scholar 

  19. J. Y. Zhu, Z. T. Zhang, C. K. Zhang, J. J. Wu, A. Torralba, J. B. Tenenbaum, W. T. Freeman. Visual object networks: Image generation with disentangled 3D representation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, Canada, pp. 118–129, 2018.

  20. T. H. Nguyen-Phuoc, C. Li, S. Balaban, Y. L. Yang. RenderNet: A deep convolutional network for differentiable rendering from 3D shapes. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, Montreal, Canada, pp. 7902–7912, 2018.

  21. K. Rematas, V. Ferrari. Neural voxel renderer: Learning an accurate and controllable rendering tool. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Seattle, USA, pp. 5416–5426, 2020. DOI: https://doi.org/10.1109/CVPR42600.2020.00546.

    Google Scholar 

  22. V. Sitzmann, J. Thies, F. Heide, M Nießner, G. Wetzstein, M. Zollhöfer. Deepvoxels: Learning persistent 3D feature embeddings. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Long Beach, USA, pp. 2432–2441, 2019. DOI: https://doi.org/10.1109/CVPR.2019.00254.

    Google Scholar 

  23. Y. Liao, K. Schwarz, L. Mescheder, A. Geiger. Towards unsupervised learning of generative models for 3D controllable image synthesis. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Seattle, USA, pp. 5870–5879, 2020. DOI: https://doi.org/10.1109/CVPR42600.2020.00591.

    Google Scholar 

  24. Z. Chen, A. P. Chen, G. L. Zhang, C. Y. Wang, Y. Ji, K. N. Kutulakos, J. Y. Yu. A neural rendering framework for free-viewpoint relighting. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Seattle, USA, pp. 5598–5609, 2020. DOI: https://doi.org/10.1109/CVPR42600.2020.00564.

    Google Scholar 

  25. J. Granskog, F. Rousselle, M. Papas, J. Novák. Compositional neural scene representations for shading inference. ACM Transactions on Graphics, vol. 39, no. 4, Article number 135, 2020. DOI: https://doi.org/10.1145/3386569.3392475.

  26. H. Kato, Y. Ushiku, T. Harada. Neural 3D mesh renderer. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 3907–3916, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00411.

    Google Scholar 

  27. A. P. Chen, M. Y. Wu, Y. L. Zhang, N. Y. Li, J. Lu, S. H. Gao, J. Y. Yu. Deep surface light fields. Proceedings of ACM on Computer Graphics and Interactive Techniques, vol. 1, no. 1, Article number 14, 0088. DOI: https://doi.org/10.1145/3203192.

  28. J. Thies, M Zollhöfer, M. Nießner. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics, vol. 38, no. 4, Article number 66, 2019. DOI: https://doi.org/10.1145/3306346.3323035.

  29. D. Gao, G. J. Chen, Y. Dong, P. Peers, K. Xu, X. Tong. Deferred neural lighting: Free-viewpoint relighting from unstructured photographs. ACM Transactions on Graphics, vol. 39, no. 6, Article number 258, 2020. DOI: https://doi.org/10.1145/3414685.3417767.

  30. M. Meshry, D. B. Goldman, S. Kharms, H. Hoppe, R. Pandey, N. Snavely, R. Martin-Brualla. Neural rerendering in the wild. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Long Beach, USA, pp. 6871–6880, 2019. DOI: https://doi.org/10.1109/CVPR.2019.00704.

    Google Scholar 

  31. K. A. Aliev, A. Sevastopolsky, M. Kolos, D. Ulyanov, V. Lempitsky. Neural point-based graphics. In Proceedings of the 16th European Conference on Computer Vision, Springer, Glasgow, UK, pp. 696–712, 2020. DOI: https://doi.org/10.1007/978-3-030-58542-6_42.

    Google Scholar 

  32. P. Dai, Y. D. Zhang, Z. W. Li, S. C. Liu, B. Zeng. Neural point cloud rendering via multi-plane projection. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Seattle, USA, pp. 7827–7836, 2020. DOI: https://doi.org/10.1109/CVPR42600.2020.00785.

    Google Scholar 

  33. P. Sanzenbacher, L. Mescheder, A. Geiger. Learning neural light transport, [Online], Available: https://arxiv.org/abs/2006.03427, 2020.

  34. M. Oechsle, M. Niemeyer, C. Reiser, L. Mescheder, T. Strauss, A. Geiger. Learning implicit surface light fields. In Proceedings of International Conference on 3D Vision, IEEE, Fukuoka, Japan, pp. 452–462, 2020. DOI: https://doi.org/10.1109/3DV50981.2020.00055.

    Google Scholar 

  35. Q. Q. Wang, Z. C. Wang, K. Genova, P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla, N. Snavely, T. Funkhouser. IBRNet: Learning multi-view image-based rendering. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Nashville, USA pp. 4688–4697, 2021. DOI: https://doi.org/10.1109/CVPR46437.2021.00466

    Google Scholar 

  36. L. Yariv, J. T. Gu, Y. Kasten, Y. Lipman. Volume rendering of neural implicit surfaces. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp 4805–4815, 2021

  37. V. Sitzmann, M. Zollhöfer, G. Wetzstein. Scene representation networks: Continuous 3D-structure-aware neural scene representations. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, NeurIPS, Vancouver, Canada, pp. 101, 2019.

    Google Scholar 

  38. M. Niemeyer, L. Mescheder, M. Oechsle, A. Geiger. Differentiate volumetric eendering: Learning implicit 3D representations without 3D supervision. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Seattle, USA, pp. 3501–3512, 2020. DOI: https://doi.org/10.1109/CVPR42600.2020.00356.

    Google Scholar 

  39. P. Kellnhofer, L. C. Jebe, A. Jones, R. Spicer, K. Pulli, G. Wetzstein. Neural lumigraph rendering. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Nashville, USA, pp. 4285–4295, 2021. DOI: https://doi.org/10.1109/CVPR46437.2021.00427.

    Google Scholar 

  40. O. Nalbach, E. Arabadzhiyska, D. Mehta, H. P. Seidel, T. Ritschel. Deep shading: Convolutional neural networks for screen space shading. Computer Graphics Forum, vol. 36, no. 4, pp. 65–78, 2017. DOI: https://doi.org/10.1111/cgf.13225.

    Article  Google Scholar 

  41. U. Erra, N. Capece, R. Agatiello. Ambient occlusion baking via a feed-forward neural network. In Proceedings of European Association for Computer Graphics: Short Papers, Lyon, France, pp. 13–16, 2017. DOI: https://doi.org/10.2312/egsh.20171003.

  42. D. J. Zhang, C. H. Xian, G. L. Luo, Y. H. Xiong, C. Han. DeepAO: Efficient screen space ambient occlusion generation via deep network. IEEE Access, vol. 8, pp. 64434–64441, 2020. DOI: https://doi.org/10.1109/ACCESS.2020.2984771.

    Article  Google Scholar 

  43. L. Ren, Y. Song. AOGAN: A generative adversarial network for screen space ambient occlusion. Computational Visual Media, vol. 8, no. 8, pp. 483–494, 2022. DOI: https://doi.org/10.1007/s41095-021-0248-2.

    Article  Google Scholar 

  44. C. Suppan, A. Chalmers, J. Zhao, A. Doronin, T. Rhee. Neural screen space rendering of direct illumination. In Proceedings of the 29th Pacific Conference on Computer Graphics and Applications, Pacific Graphics, Wellington, New Zealand, pp. 37–42, 2021.

    Google Scholar 

  45. M. Mirza, S. Osindero. Conditional generative adversarial nets, [Online], Available: https://arxiv.org/abs/1411.1784, 2014.

  46. M. M. Thomas, A. G. Forbes. Deep illumination: Approximating dynamic global illumination with generative adversarial network, [Online], Available: https://arxiv.org/abs/1710.09834, 2017.

  47. T. Müller, F. Rousselle, A. Keller, J. Novák. Neural control variates. ACM Transactions on Graphics, vol. 39, no. 6, Article number 243, 2020. DOI: https://doi.org/10.1145/3414685.3417804.

  48. P. R. Ren, J. P. Wang, M. M. Gong, S. Lin, X. Tong, B. N. Guo. Global illumination with radiance regression functions. ACM Transactions on Graphics, vol. 32, no. 4, Article number 130, 2013. DOI: https://doi.org/10.1145/2461912.2462009.

  49. S. Hadadan, S. H. Chen, M. Zwicker. Neural radiosity. ACM Transactions on Graphics, vol. 40, no. 6, Article number 236, 2021. DOI: https://doi.org/10.1145/3478513.3480569.

  50. S. Diolatzis, J. Philip, G. Drettakis. Active exploration for neural global illumination of variable scenes. ACM Transactions on Graphics, vol. 41, no. 5, Article number 171, 2022. DOI: https://doi.org/10.1145/3522735.

  51. S. Kallweit, T. Müller, B. Mcwilliams, M. Gross, J. Novák. Deep scattering: Rendering atmospheric clouds with radiance-predicting neural networks. ACM Transactions on Graphics, vol. 36, no. 6, pp. 1–11, 2017. DOI: https://doi.org/10.1145/3130800.3130880.

    Article  Google Scholar 

  52. M. Panin, S. Nikolenko. Faster RPNN: Rendering clouds with latent space light probes. In Proceedings of SIGGRAPH Asia Technical Briefs, ACM, Brisbane, Australia, pp. 21–24, 2019. DOI: https://doi.org/10.1145/3355088.3365150.

    Google Scholar 

  53. F. Abbas, M. C. Babahenini. Forest fog rendering using generative adversarial networks. The Visual Computer, vol. 39, no. 3, pp. 943–952, 2023.

    Article  Google Scholar 

  54. Q. Zheng, G. Singh, H. P. Seidel. Neural relightable participating media rendering. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 15203–15215, 2021.

  55. P. Hermosilla, S. Maisch, T. Ritschel, T. Ropinski. Deep-learning the latent space of light transport. Computer Graphics Forum, vol. 38, no. 4, pp. 207–217, 2019. DOI: https://doi.org/10.1111/cgf.13783.

    Article  Google Scholar 

  56. D. Vicini, V. Koltun, W. Jakob. A learned shape-adaptive subsurface scattering model. ACM Transactions on Graphics, vol. 38, no. 4, Article number 127, 2019. DOI: https://doi.org/10.1145/3306346.3322974.

  57. L. Y. Wei, L. W. Hu, V. Kim, E. Yumer, H. Li. Real-time hair rendering using sequential adversarial networks. In Proceedings of the 15th European Conference on Computer Vision, Springer, Munich, Germany, pp. 105–122, 2018. DOI: https://doi.org/10.1007/978-3-030-01225-07.

    Google Scholar 

  58. R. Martin-Brualla, R. Pandey, S. R. Yang, P. Pidlypenskyi, J. Taylor, J. Valentin, S. Khamis, P. Davidson, A. Tkach, P. Lincoln, A. Kowdle, C. Rhemann, D. B. Goldman, C. Keskin, S. Seitz, S. Izadi, S. Fanello. LookinGood: Enhancing performance capture with real-time neural re-rendering. ACM Transactions on Graphics, vol. 37, no. 6, Article number 255, 2018. DOI: https://doi.org/10.1145/3272127.3275099.

  59. A. Meka, R. Pandey, C. Häne, S. Orts-Escolano, P. Barnum, P. David-Son, D. Erickson, Y. D. Zhang, J. Taylor, S. Bouaziz, C. Legendre, W. C. Ma, R. Overbeck, T. Beeler, P. Debevec, S. Izadi, C. Theobalt, C. Rhemann, S. Fanello. Deep relightable textures: Volumetric performance capture with neural rendering. ACM Transactions on Graphics, vol. 39, no. 6, Article number 259, 2020. DOI: https://doi.org/10.1145/3414685.3417814.

  60. P. Chandran, S. Winberg, G. Zoss, J. Riviere, M. Gross, P. Gotardo, D. Bradley. Rendering with style: Combining traditional and neural approaches for high-quailty ace rendering. ACM Transactions on Graphics, vol. 40, no. 6, Article number 223, 2021. DOI: https://doi.org/10.1145/3478513.3480509.

  61. S. Lombardi, J. Saragih, T. Simon, Y. Sheikh. Deep appearance models or ace rendering. ACM Transactions on Graphics, vol. 37, no. 4, Article number 68, 2018. DOI: https://doi.org/10.1145/3197517.3201401.

  62. L. J. Liu, W. P. Xu, M. Zollhö er, H. Kim, F. Bernard, M. Habermann, W. P. Wang, C. Theobalt. Neural rendering and reenactment o human actor videos. ACM Transactions on Graphics, vol. 38, no. 5, Article number 139, 2019. DOI: https://doi.org/10.1145/3333002.

  63. M. Y. Wu, Y. H. Wang, Q. Hu, J. Y. Yu. Multi-view neural human rendering. In Proceedings o IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Seattle, USA, pp. 1679–1688, 2020. DOI: https://doi.org/10.1109/CVPR42600.2020.00175.

    Google Scholar 

  64. X. M. Zhang, S. Fanello, Y. T. Tsai, T. C. Sun, T. F. Xue, R. Pandey, S. Orts-Escolano, P. Davidson, C. Rhemann, P. Debevec, J. T. Barron, R. Ramamoorthi, W. T. Freeman. Neural light transport or relighting and view synthesis. ACM Transactions on Graphics, vol. 40, no. 1, Article number 9, 2021. DOI: https://doi.org/10.1145/3446328.

  65. A. Edelsten, P. Jukarainen, A. Patney. Truly Next-GEN: Adding Deep Learning to Games & Graphics. In Proceedings of Game Developers Con erence Recording (GDC Vault), USA, 2019.

  66. L. Xiao, S. Nouri, M. Chapman, A. Fix, D. Lanman, A. Kaplanyan. Neural supersampling or real-time rendering. ACM Transactions on Graphics, vol. 39, no. 4, Article number 142, 2020. DOI: https://doi.org/10.1145/3386569.3392376.

  67. J. Guo, X. H. Fu, L. Q. Lin, H. J. Ma, Y. W. Guo, S. Q. Liu, L. Q. Yan. ExtraNet: Real-time extrapolated rendering or low-latency temporal supersampling. ACM Transactions on Graphics, vol. 40, no. 6, Article number 278, 2021. DOI: https://doi.org/10.1145/3478513.3480531.

  68. K. M. Briedis, A. Djelouah, M. Meyer, I. McGonigal, M. Gross, C. Schroers. Neural rame interpolation or rendered content. ACM Transactions on Graphics, vol. 40, no. 6, Article number 239, 2021. DOI: https://doi.org/10.1145/3478513.3480553.

  69. S. Bi, K. Sunkavalli, F. Perazzi, E. Shechtman, V. Kim, R. Ramamoorthi. Deep CG2Real: Synthetic-to-real translation via image disentanglement. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Seoul, Republic o Korea, pp. 2730–2739, 2019. DOI: https://doi.org/10.1109/ICCV.2019.00282.

    Google Scholar 

  70. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, R. Ng. NeRF: Representing scenes as neural radiance ields or view synthesis. In Proceedings of the 16th European Con erence on Computer Vision, Springer, Glasgow, UK, pp. 405–421, 2020. DOI: https://doi.org/10.1007/978-3-030-58452-8_24.

    Google Scholar 

  71. J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, P. P. Srinivasan. Mip-NeRF: A multiscale representation or anti-aliasing neural radiance ields. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Montreal, Canada, pp. 5835–5844, 2021. DOI: https://doi.org/10.1109/ICCV48922.2021.00580.

    Google Scholar 

  72. R. Martin-Brualla, N. Radwan, M. S. M. Sajjadi, J. T. Barron, A. Dosovitskiy, D. Duckworth. NeRF in the wild: Neural radiance fields or unconstrained photo collections. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Nashville, USA, pp. 7206–7215, 2021. DOI: https://doi.org/10.1109/CVPR46437.2021.00713.

    Google Scholar 

  73. K. Park, U. Sinha, P. Hedman, J. T. Barron, S. Bouaziz, D. B. Goldman, R. Martin-Brualla, S. M. Seitz. HyperNeRF: A higher-dimensional representation for topologically varying neural radiance fields. ACM Transactions on Graphics, vol. 40, no. 6, Article number 238, 2021. DOI: https://doi.org/10.1145/3478513.3480487.

  74. A. Pumarola, E. Corona, G. Pons-Moll, F. Moreno-Noguer. D-NeRF: Neural radiance fields for dynamic scenes. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Nashville, USA, pp. 10313–10322, 2021. DOI: https://doi.org/10.1109/CVPR46437.2021.01018

    Google Scholar 

  75. S. Y. Su, F. Yu, M. Zollhöfer, H. Rhodin. A-NeRF: Articulated neural radiance fields for learning human shape, appearance, and pose. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 12278–12291, 2021.

  76. G. Gafni, J. Thies, M. Zollhöfer, M. Nießner. Dynamic neural radiance fields for monocular 4D facial avatar reconstruction. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Nashville, USA, pp. 8645–8654, 2021. DOI: https://doi.org/10.1109/CVPR46437.2021.00854.

    Google Scholar 

  77. C. Gao, Y. Shih, W. S. Lai, C. K. Liang, J. B. Huang. Portrait neural radiance fields from a single image, [Online], Available: https://arxiv.org/abs/2012.05903, 2020.

  78. B. B. Yang, Y. D. Zhang, Y. H. Xu, Y. J. Li, H. Zhou, H. J. Bao, G. F. Zhang, Z. P. Cui. Learning object-compositional neural radiance field for editable scene rendering. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Montreal, Canada, pp. 13759–13768, 2021. DOI: https://doi.org/10.1109/ICCV48922.2021.01352.

    Google Scholar 

  79. Y. C. Huo, S. E. Yoon. A survey on deep learning-based Monte Carlo denoising. Computational Visual Media, vol. 7, no. 2, pp. 169–185, 2021. DOI: https://doi.org/10.1007/s41095-021-0209-9.

    Article  Google Scholar 

  80. K. Gao, Y. N. Gao, H. J. He, D. N. Lu, L. L. Xu, J. Li. NeRF: Neural radiance field in 3D vision, a comprehensive evview, [Online], Available: https://arxiv.org/abs/2210.00379, 2022.

  81. P. Wang, L. J. Liu, Y. Liu, C. Theobalt, T. Komura, W. P. Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pp. 27171–27183, 2021.

  82. Z. H. Yu, S. Y. Peng, M. Niemeyer, T. Sattler, A. Geiger. MonoSDF: Exploring monocular geometric cues for neural implicit surface reconstruction, [Online], Available: https://arxiv.org/abs/2206.00665, 2022.

  83. J. T. Kajiya. The rendering equation. In Proceedings of the 13th Annual Conference on Computer Graphics and Interactive Techniques, ACM, Dallas, USA, pp. 143–150, 1986. DOI: https://doi.org/10.1145/15922.15902.

  84. E. Veach. Robust Monte Carlo methods for light transport simulation, Ph. D. dissertation, Stanford University, Stanford, USA, 1998.

    Google Scholar 

  85. M. Pharr, W. Jakob, G. Humphreys. Physically Based Rendering: From Theory to Implementation, 3rd ed., Cambridge, USA: Morgan Kaufmann, 2016.

    Google Scholar 

  86. A. Dosovitskiy, J. T. Springenberg, T. Brox. Learning to generate chairs with convolutional neural networks. In Proceedings of IEEE Conference on Computer Vision and Pattern recognition, Boston, USA, pp. 1538–1546, 2015. DOI: https://doi.org/10.1109/CVPR.2015.7298761.

  87. Y. Blau, T. Michaeli. The perception-distortion tradeoff. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 6228–6237, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00652.

    Google Scholar 

  88. X. E. Zhang, R. Ng, Q. F. Chen. Single image reflection separation with perceptual losses. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Salt Lake City, USA, pp. 4786–4794, 2018. DOI: https://doi.org/10.1109/CVPR.2018.00503.

    Google Scholar 

  89. C. Atapattu, B. Rekabdar. Improving the realism of synthetic images through a combination of adversarial and perceptual losses. In Proceedings of International Joint Conference on Neural Networks, IEEE, Budapest, Hungary, pp. 1–7, 2019. DOI: https://doi.org/10.1109/IJCNN.2019.8852449.

    Google Scholar 

  90. J. Johnson, A. Alaahi, L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, pp. 694–711, 2016. DOI: https://doi.org/10.1007/978-3-319-46475-643.

    Google Scholar 

  91. A. Rehman, Z. Wang. SSIM-based non-local means image denoising. In Proceedings of the 18th IEEE International Conference on Image Processing, Brussels, Belgium, pp. 217–220, 2011. DOI: https://doi.org/10.1109/ICIP.2011.6116065.

  92. J. Hwang, C. S. Yu, Y. Shin. SAR-to-optical image translation using SSIM and perceptual loss based cycle-consistent GAN. In Proceedings of International Conference on Information and Communication Technology Convergence, IEEE, Jeju, Republic of Korea, pp. 191–194, 2020. DOI: https://doi.org/10.1109/ICTC49870.2020.9289381.

    Google Scholar 

  93. T. Karras, S. Laine, T. Aila. A style-based generator architecture obr generative adversarial networks, In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Long Beach USA, pp. 4396–4405, 2019. DOI: https://doi.org/10.1109/CVPR.2019.00453.

    Google Scholar 

  94. L. Bavoil. Horizon-based Ambient Occlusion Using Compute Shaders, NVIDIA, USA, 2011.

    Google Scholar 

  95. J. Wilhelms, A. Van Gelder. A coherent projection approach for direct volume rendering. ACM SIGGRAPH Computer Graphics, vol. 25, no. 4, pp. 275–284, 1991. DOI: https://doi.org/10.1145/127719.122758.

    Article  Google Scholar 

  96. P. Kutz, R. Habel, Y. K. Li, J. Novák. Spectral and decomposition tracking for rendering heterogeneous volumes. ACM Transactions on Graphics, vol. 36, no. 4, Article number 111, 2017. DOI: https://doi.org/10.1145/3072959.3073665.

  97. B. Miller, I. Georgiev, W. Jarosz. A null-scattering path integral formulation of light transport. ACM Transactions on Graphics, vol. 38, no. 4, Article number 44, 2019. DOI: https://doi.org/10.1145/3306346.3323025.

  98. C. Donner, J. Lawrence, R. Ramamoorthi, T. Hachisuka, H. W. Jensen, S. Nayar. An empirical BSSRDF model. New Orleans Louisiana, USA, Article number 30, 2009. DOI: https://doi.org/10.1145/1576246.1531336.

  99. B. Mildenhall, P. P. Srinivasan, R. Ortiz-Cayon, N. K. Kalantari, R. Ramamoorthi, R. Ng, A. Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics, vol. 38, no. 4, Article number 29, 2019. DOI: https://doi.org/10.1145/3306346.3322980.

  100. C. Crassin, F. Neyret, M. Sainz, S. Green, E. Eisemann. Interactive indirect illumination using voxel cone tracing. Computer Graphics Forum, vol. 30, no. 7, pp. 1921–1930, 2011. DOI: https://doi.org/10.1111/j.1467-8659.2011.02063.x.

    Article  Google Scholar 

  101. Y. Tokuyoshi, S. Ogaki. Real-time bidirectional path tracing via rasterization. In Proceedings of ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, Costa Mesa, USA, pp. 183–190, 2012. DOI: https://doi.org/10.1145/2159616.2159647.

  102. O. Ronneberger, P. Fischer, T. Brox. U-Net: Convolutional networks or biomedical image segmentation. In Proceedings of the 18th International Conference on Medical Image Computing and Computer-assisted Intervention, Springer, Munich, Germany, pp. 234–241, 2015. DOI: https://doi.org/10.1007/978-3-319-24574-428.

    Google Scholar 

  103. S. R. Song, F. Yu, A. Zeng, A. X. Chang, M. Savva, T. Funkhouser. Semantic scene completion rom a single depth image. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, pp. 190–198, 2017. DOI: https://doi.org/10.1109/CVPR.2017.28.

  104. T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, T. Aila. Analyzing and improving the image quality of StyleGAN. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Seattle, USA, pp. 8107–8116, 2020. DOI: https://doi.org/10.1109/CVPR42600.2020.00813.

    Google Scholar 

  105. D. E. Worrall, S. J. Garbin, D. Turmukhambetov, G. J. Brostow. Interpretable transformations with encoder-decoder networks. In Proceedings of IEEE International Conference on Computer Vision, Venice, Italy, pp. 5737–5746, 2017. DOI: https://doi.org/10.1109/ICCV.2017.611.

  106. J. L. Schönberger, J. M. Frahm. Structure-from-motion revisited. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, pp. 4104–4113, 2016. DOI: https://doi.org/10.1109/CVPR.2016.445.

  107. J. L. Schönberger, E. L. Zheng, J. M. Frahm, M. Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Proceedings of the 14th European Conference on Computer Vision, Springer, Amsterdam, The Netherlands, pp. 501–518, 2016. DOI: https://doi.org/10.1007/978-3319-46487-931.

    Google Scholar 

  108. L. C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. L. Yuille. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 4, pp. 834–848, 2018. DOI: https://doi.org/10.1109/TPAMI.2017.2699184.

    Article  Google Scholar 

  109. R. Q. Charles, S. Hao, K. C. Mo, L. J. Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, pp. 77–85, 2017. DOI: https://doi.org/10.1109/CVPR.2017.16.

  110. K. M. He, X. Y. Zhang, S. Q. Ren, J. Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, Las Vegas, USA, pp. 770–778, 2016. DOI: https://doi.org/10.1109/CVPR.2016.90.

    Google Scholar 

  111. G. H. Li, M. Müller, A. Thabet, B. Ghanem. DeepGCNs: Can GCNs go as deep as CNNs? In Proceedings of 2019 IEEE/CVF International Conference on Computer Vision, IEEE, Seoul, Republic of Korea, pp. 9266–9275, 2019. DOI: https://doi.org/10.1109/ICCV.2019.00936.

    Google Scholar 

  112. L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, A. Geiger. Occupancy networks: Learning 3D reconstruction in function space. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Long Beach, USA, pp. 4455–4465, 2019. DOI: https://doi.org/10.1109/CVPR.2019.00459.

    Google Scholar 

  113. M. Oechsle, L. Mescheder, M. Niemeyer, T. Strauss, A. Geiger. Texture fields: Learning texture representations in function space. In Proceedings of IEEE/CVF International Conference on Computer Vision, IEEE, Seoul, Republic of Korea, pp. 4530–4539, 2019. DOI: https://doi.org/10.1109/ICCV.2019.00463.

    Google Scholar 

  114. K. Zhang, G. Riegler, N. Snavely, V. Koltun. NeRF++: Analyzing and improving neural radiance fields, [Online], Available: https://arxiv.org/abs/2010.07492, 2020.

  115. Y. Yao, Z. X. Luo, S. W. Li, J. Y. Zhang, Y. F. Ren, L. Zhou, T. Fang, L. Quan. BlendedMVS: A large-scale dataset for generalized multi-view stereo networks. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, Seattle, USA, pp. 1787–1796, 2020. DOI: https://doi.org/10.1109/CVPR42600.2020.00186.

    Google Scholar 

  116. L. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, R. Basri, Y. Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, Article number 210, 2020.

  117. V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, G. Wetzstein. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, vol. 33, no. 7, pp. 7462–7473, 2020.

    Google Scholar 

  118. C. Buehler, M. Bosse, L. McMillan, S. Gortler, M. Cohen. Unstructured lumigraph rendering. In Proceedings of the 28th Annual Conference on Computer graphics and Interactive Techniques, ACM, Los Angeles, USA, pp. 425–432, 2001. DOI: https://doi.org/10.1145/383259.383309.

    Chapter  Google Scholar 

  119. V. Lepetit, F. Moreno-Noguer, P. Fua. EPnP: An accurate O(n) solution to the PnP problem. International Journal of Computer Vision, vol. 81, no. 2, pp. 155–166, 2009. DOI: https://doi.org/10.1007/s11263-008-0152-6.

    Article  Google Scholar 

  120. D. Mehta, S. Sridhar, O. Sotnychenko, H. Rhodin, M. Shafiei, H. P. Seidel, W. P. Xu, D. Casas, C. Theobalt. VNect: Real-time 3D human pose estimation with a single RGB camera. ACM Transactions on Graphics, vol. 36, no. 4, Article number 44, 2017. DOI: https://doi.org/10.1145/3072959.3073596.

  121. C. R. Qi, L. Yi, H. Su, L. J. Guibas. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, USA, pp. 5105–5114, 2017.

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory of CAD&CG, Zhejiang University, Hangzhou, 310058, China

    Qi Wang, Zhihua Zhong, Yuchi Huo, Hujun Bao & Rui Wang

  2. Zhejiang Laboratory, Hangzhou, 311121, China

    Yuchi Huo

Authors
  1. Qi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhihua Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuchi Huo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hujun Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rui Wang.

Ethics declarations

The authors declared that they have no conflicts of interest to this work.

Additional information

Colored figures are available in the online version at https://link.springer.com/journal/11633

Qi Wang received the B. Eng. and M. Eng. degrees in computer science and technology from Beijing Institute of Technology, China in 2017 and 2019, respectively. He is currently a Ph. D. degree candidate in computer graphics at State Key Laboratory of CAD&CG, Zhejiang University, China.

His research direction is human-related rendering.

Zhihua Zhong received the B. Eng. degree in computer network from Jinan University, China in 2017. He is currently a master student in computer graphics at State Key Laboratory of CAD&CG, Zhejiang University, China.

His research direction is superresolution in real-time rendering.

Yuchi Huo received the Ph. D. degree from State Key Laboratory of CAD&CG, Zhejiang University, China. He is a “Hundred Talent Program” researcher in State Key Laboratory of CAD&CG, Zhejiang University, China.

His research interests include rendering, deep learning, image processing, and computational optics, which are aiming for the realization of next-generation neural rendering pipeline and physical-neural computation.

Hujun Bao received the B. Eng. and Ph. D. degrees in applied mathematics from Zhejiang University, China in 1987 and 1993, respectively. He is currently the deputy director of Zhejiang Laboratory and the deputy director of Informatics Department of the Science and Technology Committee of the Ministry of Education, China.

His research interests include rendering, modelling and virtual reality.

Rui Wang received the B. Sc. degree in computer science and the Ph. D. degree in mathematics from Zhejiang University, China in 2001 and 2007, respectively. He is currently a professor at State Key Laboratory of CAD&CG, Zhejiang University, China.

His research interests include real-time rendering, realistic rendering, GPU-based computation and 3D display techniques.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Q., Zhong, Z., Huo, Y. et al. State of the Art on Deep Learning-enhanced Rendering Methods. Mach. Intell. Res. 20, 799–821 (2023). https://doi.org/10.1007/s11633-022-1400-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11633-022-1400-x

Keywords

Search

Navigation