Skip to main content
Springer Nature Link
Log in

Flash-Splat: 3D Reflection Removal with Flash Cues and Gaussian Splats

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15140))

Included in the following conference series:

  • 257 Accesses

Abstract

We introduce a simple yet effective approach for separating transmitted and reflected light. Our key insight is that the powerful novel view synthesis capabilities provided by modern inverse rendering methods (e.g., 3D Gaussian splatting) allow one to perform flash/no-flash reflection separation using unpaired measurements—this relaxation dramatically simplifies image acquisition over conventional paired flash/no-flash reflection separation methods. Through extensive real-world experiments, we demonstrate our method, Flash-Splat, accurately reconstructs both transmitted and reflected scenes in 3D. Our method outperforms existing 3D reflection separation methods, which do not leverage illumination control, by a large margin.

M. Xie and H. Cai—Equal Contribution.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Alayrac, J.B., Carreira, J., Zisserman, A.: The visual centrifuge: model-free layered video representations. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2457–2466 (2019)

    Google Scholar 

  2. Arvanitopoulos, N., Achanta, R., Susstrunk, S.: Single image reflection suppression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4498–4506 (2017)

    Google Scholar 

  3. Baker, A.H., Pinard, A., Hammerling, D.M.: On a structural similarity index approach for floating-point data. IEEE Trans. Vis. Comput. Graph. 1–13 (2023)

    Google Scholar 

  4. Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Mip-NeRF 360: unbounded anti-aliased neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5470–5479 (2022)

    Google Scholar 

  5. Chen, A., Xu, Z., Geiger, A., Yu, J., Su, H.: TensoRF: tensorial radiance fields. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022. ECCV 2022. LNCS, vol.13692, pp. 333–350. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19824-3_20

  6. Chugunov, I., Shustin, D., Yan, R., Lei, C., Heide, F.: Neural spline fields for burst image fusion and layer separation. CVPR (2024)

    Google Scholar 

  7. Dong, Z., Xu, K., Yang, Y., Bao, H., Xu, W., Lau, R.W.: Location-aware single image reflection removal. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5017–5026 (2021)

    Google Scholar 

  8. Fan, Q., Yang, J., Hua, G., Chen, B., Wipf, D.: A generic deep architecture for single image reflection removal and image smoothing. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3238–3247 (2017)

    Google Scholar 

  9. Farid, H., Adelson, E.H.: Separating reflections and lighting using independent components analysis. In: Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149), vol. 1, pp. 262–267. IEEE (1999)

    Google Scholar 

  10. Fridovich-Keil, S., Meanti, G., Warburg, F.R., Recht, B., Kanazawa, A.: K-planes: explicit radiance fields in space, time, and appearance. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12479–12488 (2023)

    Google Scholar 

  11. Gandelsman, Y., Shocher, A., Irani, M.: Double-DIP: unsupervised image decomposition via coupled deep-image-priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11026–11035 (2019)

    Google Scholar 

  12. Guo, X., Cao, X., Ma, Y.: Robust separation of reflection from multiple images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2187–2194 (2014)

    Google Scholar 

  13. Guo, Y.C., Kang, D., Bao, L., He, Y., Zhang, S.H.: NeRFReN: neural radiance fields with reflections. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18409–18418 (2022)

    Google Scholar 

  14. Hariharan, B., Arbeláez, P., Girshick, R., Malik, J.: Hypercolumns for object segmentation and fine-grained localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 447–456 (2015)

    Google Scholar 

  15. Hong, Y., Zheng, Q., Zhao, L., Jiang, X., Kot, A.C., Shi, B.: Panoramic image reflection removal. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7762–7771 (2021)

    Google Scholar 

  16. Hong, Y., Zheng, Q., Zhao, L., Jiang, X., Kot, A.C., Shi, B.: PAR2 NET: end-to-end panoramic image reflection removal. IEEE Trans. Pattern Anal. Mach. Intell. (2023)

    Google Scholar 

  17. Hu, Q., Guo, X.: Trash or treasure? An interactive dual-stream strategy for single image reflection separation. Adv. Neural. Inf. Process. Syst. 34, 24683–24694 (2021)

    Google Scholar 

  18. Hu, Q., Guo, X.: Single image reflection separation via component synergy. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13138–13147 (2023)

    Google Scholar 

  19. Kee, E., Pikielny, A., Blackburn-Matzen, K., Levoy, M.: Removing reflections from raw photos. ArXiv abs/2404.14414 (2024)

    Google Scholar 

  20. Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3D gaussian splatting for real-time radiance field rendering. ACM Trans. Graph. 42(4) (2023)

    Google Scholar 

  21. Kim, S., Huo, Y., Yoon, S.E.: Single image reflection removal with physically-based rendering. arXiv preprint arXiv:1904.11934 (2019)

  22. Kong, N., Tai, Y.W., Shin, J.S.: A physically-based approach to reflection separation: from physical modeling to constrained optimization. IEEE Trans. Pattern Anal. Mach. Intell. 36(2), 209–221 (2013)

    Article  Google Scholar 

  23. Kong, N., Tai, Y.W., Shin, S.Y.: High-quality reflection separation using polarized images. IEEE Trans. Image Process. 20(12), 3393–3405 (2011)

    Article  MathSciNet  Google Scholar 

  24. Lei, C., Chen, Q.: Robust reflection removal with reflection-free flash-only cues. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14811–14820 (2021)

    Google Scholar 

  25. Lei, C., Huang, X., Zhang, M., Yan, Q., Sun, W., Chen, Q.: Polarized reflection removal with perfect alignment in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1750–1758 (2020)

    Google Scholar 

  26. Levin, A., Weiss, Y.: User assisted separation of reflections from a single image using a sparsity prior. IEEE Trans. Pattern Anal. Mach. Intell. 29(9), 1647–1654 (2007)

    Article  Google Scholar 

  27. Levin, A., Zomet, A., Weiss, Y.: Learning to perceive transparency from the statistics of natural scenes. Adv. Neural Inf. Process. Syst. 15 (2002)

    Google Scholar 

  28. Levin, A., Zomet, A., Weiss, Y.: Separating reflections from a single image using local features. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004, vol. 1, p. I. IEEE (2004)

    Google Scholar 

  29. Li, C., Yang, Y., He, K., Lin, S., Hopcroft, J.E.: Single image reflection removal through cascaded refinement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3565–3574 (2020)

    Google Scholar 

  30. Li, R., Qiu, S., Zang, G., Heidrich, W.: Reflection separation via multi-bounce polarization state tracing. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12358, pp. 781–796. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58601-0_46

    Chapter  Google Scholar 

  31. Li, Y., Brown, M.S.: Exploiting reflection change for automatic reflection removal. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2432–2439 (2013)

    Google Scholar 

  32. Li, Y., Liu, M., Yi, Y., Li, Q., Ren, D., Zuo, W.: Two-stage single image reflection removal with reflection-aware guidance. Appl. Intell. 1–16 (2023)

    Google Scholar 

  33. Liu, Y.L., Lai, W.S., Yang, M.H., Chuang, Y.Y., Huang, J.B.: Learning to see through obstructions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14215–14224 (2020)

    Google Scholar 

  34. Lyu, Y., Cui, Z., Li, S., Pollefeys, M., Shi, B.: Reflection separation using a pair of unpolarized and polarized images. Adv. Neural Inf. Process. Syst. 32 (2019)

    Google Scholar 

  35. Mildenhall, B., Hedman, P., Martin-Brualla, R., Srinivasan, P.P., Barron, J.T.: Nerf in the dark: high dynamic range view synthesis from noisy raw images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16190–16199 (2022)

    Google Scholar 

  36. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 405–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_24

    Chapter  Google Scholar 

  37. Nayar, S.K., Fang, X.S., Boult, T.: Separation of reflection components using color and polarization. Int. J. Comput. Vis. 21(3), 163–186 (1997)

    Article  Google Scholar 

  38. Qiu, J., Jiang, P.T., Zhu, Y., Yin, Z.X., Cheng, M.M., Ren, B.: Looking through the glass: neural surface reconstruction against high specular reflections. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20823–20833 (2023)

    Google Scholar 

  39. Ranftl, R., Bochkovskiy, A., Koltun, V.: Vision transformers for dense prediction. ArXiv preprint (2021)

    Google Scholar 

  40. Ranftl, R., Lasinger, K., Hafner, D., Schindler, K., Koltun, V.: Towards robust monocular depth estimation: mixing datasets for zero-shot cross-dataset transfer. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) (2020)

    Google Scholar 

  41. Schönberger, J.L., Zheng, E., Frahm, J.-M., Pollefeys, M.: Pixelwise view selection for unstructured multi-view stereo. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 501–518. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_31

    Chapter  Google Scholar 

  42. Shih, Y., Krishnan, D., Durand, F., Freeman, W.T.: Reflection removal using ghosting cues. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3193–3201 (2015)

    Google Scholar 

  43. Sinha, S.N., Kopf, J., Goesele, M., Scharstein, D., Szeliski, R.: Image-based rendering for scenes with reflections. ACM Trans. Graph. (TOG) 31(4), 1–10 (2012)

    Article  Google Scholar 

  44. Wan, R., Shi, B., Duan, L.Y., Tan, A.H., Gao, W., Kot, A.C.: Region-aware reflection removal with unified content and gradient priors. IEEE Trans. Image Process. 27(6), 2927–2941 (2018)

    Article  MathSciNet  Google Scholar 

  45. Wan, R., Shi, B., Duan, L.Y., Tan, A.H., Kot, A.C.: CRRN: multi-scale guided concurrent reflection removal network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4777–4785 (2018)

    Google Scholar 

  46. Wan, R., Shi, B., Li, H., Duan, L.Y., Kot, A.C.: Reflection scene separation from a single image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2398–2406 (2020)

    Google Scholar 

  47. Wan, R., Shi, B., Li, H., Duan, L.Y., Kot, A.C.: Face image reflection removal. Int. J. Comput. Vis. 129, 385–399 (2021)

    Article  MathSciNet  Google Scholar 

  48. Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: Neus: learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS (2021)

    Google Scholar 

  49. Wang, Y., Han, Q., Habermann, M., Daniilidis, K., Theobalt, C., Liu, L.: Neus2: fast learning of neural implicit surfaces for multi-view reconstruction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3295–3306 (2023)

    Google Scholar 

  50. Wei, K., Yang, J., Fu, Y., Wipf, D., Huang, H.: Single image reflection removal exploiting misaligned training data and network enhancements. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8178–8187 (2019)

    Google Scholar 

  51. Wen, Q., Tan, Y., Qin, J., Liu, W., Han, G., He, S.: Single image reflection removal beyond linearity. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3771–3779 (2019)

    Google Scholar 

  52. Xia, Z., Gharbi, M., Perazzi, F., Sunkavalli, K., Chakrabarti, A.: Deep denoising of flash and no-flash pairs for photography in low-light environments. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2063–2072 (2020)

    Google Scholar 

  53. Xia, Z., Lawrence, J., Achar, S.: A dark flash normal camera. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2410–2419 (2020)

    Google Scholar 

  54. Xue, T., Rubinstein, M., Liu, C., Freeman, W.T.: A computational approach for obstruction-free photography. ACM Trans. Graph. (TOG) 34(4), 1–11 (2015)

    Article  Google Scholar 

  55. Yang, J., Gong, D., Liu, L., Shi, Q.: Seeing deeply and bidirectionally: a deep learning approach for single image reflection removal. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 654–669 (2018)

    Google Scholar 

  56. Yang, Y., Ma, W., Zheng, Y., Cai, J.F., Xu, W.: Fast single image reflection suppression via convex optimization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8141–8149 (2019)

    Google Scholar 

  57. Zhang, X., Ng, R., Chen, Q.: Single image reflection separation with perceptual losses. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4786–4794 (2018)

    Google Scholar 

  58. Zheng, Q., Shi, B., Chen, J., Jiang, X., Duan, L.Y., Kot, A.C.: Single image reflection removal with absorption effect. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13395–13404 (2021)

    Google Scholar 

  59. Zhu, Z., Fan, Z., Jiang, Y., Wang, Z.: FSGS: real-time few-shot view synthesis using gaussian splatting. arXiv preprint arXiv:2312.00451 (2023)

  60. Zou, Z., Lei, S., Shi, T., Shi, Z., Ye, J.: Deep adversarial decomposition: a unified framework for separating superimposed images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12806–12816 (2020)

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by AFOSR Young Investigator Program Award no. FA9550-22-1-0208, ONR Award no. N00014-23-1-2752, NSF CAREER Award no. 2339616, the Joint Directed Energy Transition Office, and a gift from Dolby Labs. We thank Kevin Zhang and Yi-Ting Chen for helpful discussions.

Author information

Authors and Affiliations

  1. University of Maryland, College Park, USA

    Mingyang Xie, Haoming Cai, Sachin Shah, Yiran Xu, Jia-Bin Huang & Christopher A. Metzler

  2. Massachusetts Institute of Technology, Cambridge, USA

    Brandon Y. Feng

Authors
  1. Mingyang Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haoming Cai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sachin Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yiran Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Brandon Y. Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jia-Bin Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Christopher A. Metzler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingyang Xie .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Hessen, Germany

    Stefan Roth

  4. Princeton University, Palo Alto, CA, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1294 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xie, M. et al. (2025). Flash-Splat: 3D Reflection Removal with Flash Cues and Gaussian Splats. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15140. Springer, Cham. https://doi.org/10.1007/978-3-031-73007-8_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-73007-8_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-73006-1

  • Online ISBN: 978-3-031-73007-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics