Skip to main content
Springer Nature Link
Log in

RoGUENeRF: A Robust Geometry-Consistent Universal Enhancer for NeRF

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15070))

Included in the following conference series:

  • 225 Accesses

Abstract

Recent advances in neural rendering have enabled highly photorealistic 3D scene reconstruction and novel view synthesis. Despite this progress, current state-of-the-art methods struggle to reconstruct high frequency detail, due to factors such as a low-frequency bias of radiance fields and inaccurate camera calibration. One approach to mitigate this issue is to enhance images post-rendering. 2D enhancers can be pre-trained to recover some detail but are agnostic to scene geometry and do not easily generalize to new distributions of image degradation. Conversely, existing 3D enhancers are able to transfer detail from nearby training images in a generalizable manner, but suffer from inaccurate camera calibration and can propagate errors from the geometry into rendered images. We propose a neural rendering enhancer, RoGUENeRF, which exploits the best of both paradigms. Our method is pre-trained to learn a general enhancer while also leveraging information from nearby training images via robust 3D alignment and geometry-aware fusion. Our approach restores high-frequency textures while maintaining geometric consistency and is also robust to inaccurate camera calibration. We show that RoGUENeRF substantially enhances the rendering quality of a wide range of neural rendering baselines, e.g. improving the PSNR of MipNeRF360 by 0.63dB and Nerfacto by 1.34dB on the real world 360v2 dataset. Project page: https://sib1.github.io/projects/roguenerf/.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Aittala, M., Durand, F.: Burst image deblurring using permutation invariant convolutional neural networks. In: ECCV (2018)

    Google Scholar 

  2. Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-NeRF: a multiscale representation for anti-aliasing neural radiance fields. In: ICCV, pp. 5835–5844 (2021)

    Google Scholar 

  3. Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Mip-NeRF 360: unbounded anti-aliased neural radiance fields. In: CVPR (2022)

    Google Scholar 

  4. Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Zip-NeRF: anti-aliased grid-based neural radiance fields. In: ICCV (2023)

    Google Scholar 

  5. Bian, W., Wang, Z., Li, K., Bian, J.W., Prisacariu, V.A.: Nope-NeRF: optimising neural radiance field with no pose prior. In: CVPR, pp. 4160–4169 (2023)

    Google Scholar 

  6. Catley-Chandar, S., Tanay, T., Vandroux, L., Leonardis, A., Slabaugh, G., Pérez-Pellitero, E.: FlexHDR: modeling alignment and exposure uncertainties for flexible HDR imaging. IEEE TIP 31 (2022)

    Google Scholar 

  7. Chen, A., Xu, Z., Geiger, A., Yu, J., Su, H.: TensoRF: tensorial radiance fields. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13692, pp. 333–350. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19824-3_20

    Chapter  Google Scholar 

  8. Dhamo, H., et al.: HeadGaS: real-time animatable head avatars via 3D gaussian splatting. arXiv preprint arXiv:2312.02902 (2023)

  9. Huang, X., Li, W., Hu, J., Chen, H., Wang, Y.: RefSR-NeRF: towards high fidelity and super resolution view synthesis. In: CVPR, pp. 8244–8253. IEEE Computer Society, Los Alamitos (2023)

    Google Scholar 

  10. Işık, M., et al.: HumanRF: high-fidelity neural radiance fields for humans in motion. ACM TOG 42(4), 1–12 (2023)

    Article  Google Scholar 

  11. Jensen, R., Dahl, A., Vogiatzis, G., Tola, E., Aanæs, H.: Large scale multi-view stereopsis evaluation. In: CVPR, pp. 406–413. IEEE (2014)

    Google Scholar 

  12. Jiang, Y., et al.: AligNeRF: high-fidelity neural radiance fields via alignment-aware training. In: CVPR, pp. 46–55 (2023)

    Google Scholar 

  13. Kalantari, N.K., Ramamoorthi, R.: Deep HDR video from sequences with alternating exposures. Comput. Graph. Forum (2019)

    Google Scholar 

  14. Ke, J., Wang, Q., Wang, Y., Milanfar, P., Yang, F.: MUSIQ: multi-scale image quality transformer. In: ICCV, pp. 5148–5157 (2021)

    Google Scholar 

  15. Kingma, D., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)

    Google Scholar 

  16. Li, T., Slavcheva, M., et al.: Neural 3D video synthesis from multi-view video. In: CVPR, pp. 5511–5521 (2021)

    Google Scholar 

  17. Lin, C.H., Ma, W.C., Torralba, A., Lucey, S.: BARF: bundle-adjusting neural radiance fields. In: ICCV (2021)

    Google Scholar 

  18. Liu, Y.L., et al.: Robust dynamic radiance fields. In: CVPR (2023)

    Google Scholar 

  19. Maintainers, T., Contributors: TorchVision: PyTorch’s Computer Vision library (2016)

    Google Scholar 

  20. Meuleman, A., et al.: Progressively optimized local radiance fields for robust view synthesis. In: CVPR, pp. 16539–16548 (2023)

    Google Scholar 

  21. Mildenhall, B., Hedman, P., Martin-Brualla, R., Srinivasan, P.P., Barron, J.T.: NeRF in the dark: high dynamic range view synthesis from noisy raw images. In: CVPR (2022)

    Google Scholar 

  22. Mildenhall, B., et al.: Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM TOG (2019)

    Google Scholar 

  23. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 405–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_24

    Chapter  Google Scholar 

  24. Moreau, A., Piasco, N., Tsishkou, D.V., Stanciulescu, B., de La Fortelle, A.: LENS: localization enhanced by nerf synthesis. In: Conference on Robot Learning (2021)

    Google Scholar 

  25. Moreau, A., Song, J., Dhamo, H., Shaw, R., Zhou, Y., Pérez-Pellitero, E.: Human gaussian splatting: Real-time rendering of animatable avatars. In: CVPR (2024)

    Google Scholar 

  26. Müller, T., Evans, A., Schied, C., Keller, A.: Instant neural graphics primitives with a multiresolution hash encoding. ACM TOG 41(4) (2022)

    Google Scholar 

  27. Niemeyer, M., Geiger, A.: GIRAFFE: representing scenes as compositional generative neural feature fields. In: CVPR (2021)

    Google Scholar 

  28. Park, K., Henzler, P., Mildenhall, B., Barron, J.T., Martin-Brualla, R.: CamP: camera preconditioning for neural radiance fields. ACM Trans. Graph. (2023)

    Google Scholar 

  29. Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché Buc, F., Fox, E., Garnett, R. (eds.) NeurIPS, pp. 8024–8035. Curran Associates, Inc. (2019)

    Google Scholar 

  30. Peng, S., et al.: Animatable neural radiance fields for modeling dynamic human bodies. In: ICCV (2021)

    Google Scholar 

  31. Peng, S., Yan, Y., Shuai, Q., Bao, H., Zhou, X.: Representing volumetric videos as dynamic MLP maps. In: CVPR (2023)

    Google Scholar 

  32. Pumarola, A., Corona, E., Pons-Moll, G., Moreno-Noguer, F.: D-NeRF: neural radiance fields for dynamic scenes. In: CVPR (2020)

    Google Scholar 

  33. Raoult, V., Reid-Anderson, S., Ferri, A., Williamson, J.E.: How reliable is structure from motion (SfM) over time and between observers? A case study using coral reef bommies. Remote Sens. 9(7) (2017)

    Google Scholar 

  34. Roessle, B., Müller, N., Porzi, L., Bulò, S.R., Kontschieder, P., Nießner, M.: GANeRF: leveraging discriminators to optimize neural radiance fields. ACM TOG (2023)

    Google Scholar 

  35. Rong, X., Huang, J.B., Saraf, A., Kim, C., Kopf, J.: Boosting view synthesis with residual transfer. In: CVPR (2022)

    Google Scholar 

  36. Fridovich-Keil, S., Yu, A., Tancik, M., Chen, Q., Recht, B., Kanazawa, A.: Plenoxels: radiance fields without neural networks. In: CVPR (2022)

    Google Scholar 

  37. Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: CVPR (2016)

    Google Scholar 

  38. Schönberger, J.L., Zheng, E., Frahm, J.-M., Pollefeys, M.: Pixelwise view selection for unstructured multi-view stereo. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 501–518. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_31

    Chapter  Google Scholar 

  39. Shaw, R., et al.: SWAGS: sampling windows adaptively for dynamic 3D gaussian splatting. arXiv preprint arXiv:2312.13308 (2023)

  40. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)

    Google Scholar 

  41. Tanay, T., Leonardis, A., Maggioni, M.: Efficient view synthesis and 3D-based multi-frame denoising with multiplane feature representations. In: CVPR (2023)

    Google Scholar 

  42. Tanay, T., Maggioni, M.: Global latent neural rendering. In: CVPR (2024)

    Google Scholar 

  43. Tancik, M., et al.: Fourier features let networks learn high frequency functions in low dimensional domains. In: NeurIPS (2020)

    Google Scholar 

  44. Tancik, M., et al.: NeRFStudio: a modular framework for neural radiance field development. In: ACM SIGGRAPH 2023 Conference Proceedings. SIGGRAPH 2023 (2023)

    Google Scholar 

  45. Truong, P., Rakotosaona, M.J., Manhardt, F., Tombari, F.: SPARF: neural radiance fields from sparse and noisy poses. In: CVPR, pp. 4190–4200 (2023)

    Google Scholar 

  46. Turki, H., et al.: HybridNeRF: efficient neural rendering via adaptive volumetric surfaces. In: Computer Vision and Pattern Recognition (CVPR) (2024)

    Google Scholar 

  47. Verbin, D., Hedman, P., Mildenhall, B., Zickler, T., Barron, J.T., Srinivasan, P.P.: Ref-NeRF: structured view-dependent appearance for neural radiance fields. In: CVPR (2022)

    Google Scholar 

  48. Wang, C., Chai, M., He, M., Chen, D., Liao, J.: CLIP-NeRF: text-and-image driven manipulation of neural radiance fields. In: CVPR, pp. 3835–3844 (2022)

    Google Scholar 

  49. Wang, C., Wu, X., Guo, Y.C., Zhang, S.H., Tai, Y.W., Hu, S.M.: NeRF-SR: high quality neural radiance fields using supersampling. In: ACM MM, MM 2022, pp. 6445–6454. Association for Computing Machinery, New York (2022)

    Google Scholar 

  50. Wang, Y., Han, Q., Habermann, M., Daniilidis, K., Theobalt, C., Liu, L.: NeuS2: fast learning of neural implicit surfaces for multi-view reconstruction. In: ICCV (2023)

    Google Scholar 

  51. Wang, Z., Cun, X., Bao, J., Zhou, W., Liu, J., Li, H.: UFormer: a general U-shaped transformer for image restoration. In: CVPR, pp. 17683–17693 (2022)

    Google Scholar 

  52. Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE TIP 13(4), 600–612 (2004)

    Google Scholar 

  53. Xie, Y., et al.: Neural fields in visual computing and beyond. Comput. Graph. Forum (2022)

    Google Scholar 

  54. Xu, L., et al.: VR-NeRF: high-fidelity virtualized walkable spaces. In: SIGGRAPH Asia Conference Proceedings (2023)

    Google Scholar 

  55. Yan, Q., et al.: Attention-guided network for ghost-free high dynamic range imaging. In: CVPR, pp. 1751–1760 (2019)

    Google Scholar 

  56. Yu, A., Li, R., Tancik, M., Li, H., Ng, R., Kanazawa, A.: PlenOctrees for real-time rendering of neural radiance fields. In: ICCV (2021)

    Google Scholar 

  57. Zhang, K., Riegler, G., Snavely, N., Koltun, V.: NeRF++: analyzing and improving neural radiance fields. arXiv:2010.07492v2 (2020)

  58. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR, pp. 586–595 (2018)

    Google Scholar 

  59. Zhou, K., Li, W., Jiang, N., Han, X., Lu, J.: From NeRFLiX to NeRFLiX++: a general nerf-agnostic restorer paradigm. IEEE TPAMI 1–17 (2023)

    Google Scholar 

  60. Zhou, K., et al.: NeRFLiX: high-quality neural view synthesis by learning a degradation-driven inter-viewpoint mixer. In: CVPR, pp. 12363–12374 (2023)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Huawei Noah’s Ark Lab, Montreal, Canada

    Sibi Catley-Chandar, Richard Shaw & Eduardo Pérez-Pellitero

  2. Queen Mary University of London, London, UK

    Sibi Catley-Chandar & Gregory Slabaugh

Authors
  1. Sibi Catley-Chandar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Richard Shaw
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gregory Slabaugh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eduardo Pérez-Pellitero
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sibi Catley-Chandar .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 335 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Catley-Chandar, S., Shaw, R., Slabaugh, G., Pérez-Pellitero, E. (2025). RoGUENeRF: A Robust Geometry-Consistent Universal Enhancer for NeRF. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15070. Springer, Cham. https://doi.org/10.1007/978-3-031-73254-6_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-73254-6_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-73253-9

  • Online ISBN: 978-3-031-73254-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics