Skip to main content
SpringerLink
Log in

Photon-Efficient 3D Imaging with A Non-local Neural Network

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12351))

Included in the following conference series:

Abstract

Photon-efficient imaging has enabled a number of applications relying on single-photon sensors that can capture a 3D image with as few as one photon per pixel. In practice, however, measurements of low photon counts are often mixed with heavy background noise, which poses a great challenge for existing computational reconstruction algorithms. In this paper, we first analyze the long-range correlations in both spatial and temporal dimensions of the measurements. Then we propose a non-local neural network for depth reconstruction by exploiting the long-range correlations. The proposed network achieves decent reconstruction fidelity even under photon counts (and signal-to-background ratio, SBR) as low as 1 photon/pixel (and 0.01 SBR), which significantly surpasses the state-of-the-art. Moreover, our non-local network trained on simulated data can be well generalized to different real-world imaging systems, which could extend the application scope of photon-efficient imaging in challenging scenarios with a strict limit on optical flux. Code is available at https://github.com/JiayongO-O/PENonLocal.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Note that we enlarge the illumination periods N to ensure that the returning photons in each period are weak enough so that our image formation model in Eq. 2 can still be valid without suffering from the pile-up effect in simulating these noise levels.

References

  1. Abreu, E., Lightstone, M., Mitra, S.K., Arakawa, K.: A new efficient approach for the removal of impulse noise from highly corrupted images. IEEE Trans. Image Process. 5(6), 1012–1025 (1996)

    Article  Google Scholar 

  2. Altmann, Y., Ren, X., Mccarthy, A., Buller, G., Mclaughlin, S.: Lidar waveform based analysis of depth images constructed using sparse single-photon data. IEEE Trans. Comput. Imaging 25(5), 1935–1946 (2016)

    MathSciNet  MATH  Google Scholar 

  3. Altmann, Y., McLaughlin, S., Padgett, M.J., Goyal, V.K., Hero, A.O., Faccio, D.: Quantum-inspired computational imaging. Science 361(6403), 2298 (2018)

    Article  Google Scholar 

  4. Bar-David, I.: Communication under the poisson regime. IEEE Trans. Inf. Theory 15(1), 31–37 (1969)

    Article  MathSciNet  MATH  Google Scholar 

  5. Barbastathis, G., Ozcan, A., Situ, G.: On the use of deep learning for computational imaging. Optica 6(8), 921–943 (2019)

    Article  Google Scholar 

  6. Buller, G.S., Wallace, A.M., Mccarthy, A., Lamb, R.A.: Ranging and three-dimensional imaging using time-correlated single-photon counting. IEEE J. Sel. Top. Quantum Electron. 13(4), 1006–1015 (2007)

    Article  Google Scholar 

  7. Chan, S., et al.: Long-range depth imaging using a single-photon detector array and non-local data fusion. Sci. Rep. 9(1), 8075 (2019)

    Article  Google Scholar 

  8. Chen, C., Xiong, Z., Tian, X., Wu, F.: Deep boosting for image denoising. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018, Part XI. LNCS, vol. 11215, pp. 3–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01252-6_1

    Chapter  Google Scholar 

  9. Chen, C., Xiong, Z., Tian, X., Zha, Z.J., Wu, F.: Real-world image denoising with deep boosting. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019)

    Google Scholar 

  10. Cheng, Z., Xiong, Z., Liu, D.: Light field super-resolution by jointly exploiting internal and external similarities. IEEE Trans. Circuits Syst. Video Technol. 30(8), 2604–2616 (2019)

    Article  Google Scholar 

  11. Dabov, K., Foi, A., Katkovnik, V., Egiazarian, K.: Image denoising by sparse 3-D transform-domain collaborative filtering. IEEE Trans. Image Process. 16(8), 2080–2095 (2007)

    Article  MathSciNet  Google Scholar 

  12. Dai, T., Cai, J., Zhang, Y., Xia, S.T., Zhang, L.: Second-order attention network for single image super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 11065–11074 (2019)

    Google Scholar 

  13. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 295–307 (2015)

    Article  Google Scholar 

  14. Girshick, R.: Fast R-CNN. In: IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)

    Google Scholar 

  15. Gupta, A., Ingle, A., Gupta, M.: Asynchronous single-photon 3D imaging. In: IEEE International Conference on Computer Vision, pp. 7909–7918 (2019)

    Google Scholar 

  16. Gupta, A., Ingle, A., Velten, A., Gupta, M.: Photon-flooded single-photon 3D cameras. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6770–6779 (2019)

    Google Scholar 

  17. Hadfield, R.H.: Single-photon detectors for optical quantum information applications. Nat. Photonics 3(12), 696 (2009)

    Article  Google Scholar 

  18. Holst, G.C.: CCD Arrays, Cameras, and Displays. SPIE Optical Engineering, Bellingham (1998)

    Google Scholar 

  19. Ingle, A., Velten, A., Gupta, M.: High flux passive imaging with single-photon sensors. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 6760–6769 (2019)

    Google Scholar 

  20. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  21. Kirmani, A., et al.: First-photon imaging. Science 343(6166), 58–61 (2014)

    Article  Google Scholar 

  22. Köllner, M., Wolfrum, J.: How many photons are necessary for fluorescence-lifetime measurements? Chem. Phys. Lett. 200(1–2), 199–204 (1992)

    Article  Google Scholar 

  23. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: International Conference on Neural Information Processing Systems, pp. 1097–1105 (2012)

    Google Scholar 

  24. Li, Z.P., et al.: Single-photon computational 3D imaging at 45 km. Photon. Res. 8(9), 1532–1540 (2020)

    Article  Google Scholar 

  25. Li, Z.P., et al.: All-time single-photon 3D imaging over 21 km. In: Conference on Lasers and Electro-Optics, p. SM1N.1 (2019)

    Google Scholar 

  26. Lindell, D.B., O’Toole, M., Wetzstein, G.: Single-photon 3D imaging with deep sensor fusion. ACM Trans. Graph. 37(4), 113 (2018)

    Article  Google Scholar 

  27. Liu, P., Chang, S., Huang, X., Tang, J., Cheung, J.C.K.: Contextualized non-local neural networks for sequence learning. In: Association for the Advancement of Artificial Intelligence, pp. 6762–6769 (2019)

    Google Scholar 

  28. Liu, X., et al.: Non-line-of-sight imaging using phasor-field virtual wave optics. Nature 572(7771), 620–623 (2019)

    Article  Google Scholar 

  29. O’Toole, M., Heide, F., Lindell, D.B., Zang, K., Diamond, S., Wetzstein, G.: Reconstructing transient images from single-photon sensors. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1539–1547 (2017)

    Google Scholar 

  30. O’Toole, M., Lindell, D.B., Wetzstein, G.: Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555(7696), 338 (2018)

    Article  Google Scholar 

  31. Pawlikowska, A.M., Halimi, A., Lamb, R.A., Buller, G.S.: Single-photon three-dimensional imaging at up to 10 kilometers range. Opt. Express 25(10), 11919–11931 (2017)

    Article  Google Scholar 

  32. Pediredla, A.K., Sankaranarayanan, A.C., Buttafava, M., Tosi, A., Veeraraghavan, A.: Signal processing based pile-up compensation for gated single-photon avalanche diodes. arXiv preprint arXiv:1806.07437 (2018)

  33. Peng, J., Xiong, Z., Liu, D., Chen, X.: Unsupervised depth estimation from light field using a convolutional neural network. In: International Conference on 3D Vision, pp. 295–303 (2018)

    Google Scholar 

  34. Peng, J., Xiong, Z., Wang, Y., Zhang, Y., Liu, D.: Zero-shot depth estimation from light field using a convolutional neural network. IEEE Trans. Comput. Imaging 6, 682–696 (2020)

    Article  Google Scholar 

  35. Rapp, J., Goyal, V.K.: A few photons among many: unmixing signal and noise for photon-efficient active imaging. IEEE Trans. Comput. Imaging 3(3), 445–459 (2017)

    Article  MathSciNet  Google Scholar 

  36. Ren, X., et al.: High-resolution depth profiling using a range-gated CMOS SPAD quanta image sensor. Opt. Express 26(5), 5541–5557 (2018)

    Article  MathSciNet  Google Scholar 

  37. Renker, D.: Geiger-mode avalanche photodiodes, history, properties and problems. Nucl. Instrum. Methods Phys. Res. 567(1), 48–56 (2006)

    Article  Google Scholar 

  38. Richardson, J.A., Grant, L.A., Henderson, R.K.: Low dark count single-photon avalanche diode structure compatible with standard nanometer scale CMOS technology. IEEE Photon. Technol. Lett. 21(14), 1020–1022 (2009)

    Article  Google Scholar 

  39. Saunders, C., Murray-Bruce, J., Goyal, V.K.: Computational periscopy with an ordinary digital camera. Nature 565(7740), 472 (2019)

    Article  Google Scholar 

  40. Scharstein, D., Pal, C.: Learning conditional random fields for stereo. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)

    Google Scholar 

  41. Schwartz, D.E., Charbon, E., Shepard, K.L.: A single-photon avalanche diode array for fluorescence lifetime imaging microscopy. IEEE J. Solid-State Circuits 43(11), 2546–2557 (2008)

    Article  Google Scholar 

  42. Shi, Z., Chen, C., Xiong, Z., Liu, D., Wu, F.: HSCNN+: Advanced CNN-based hyperspectral recovery from RGB images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (2018)

    Google Scholar 

  43. Shin, D., Kirmani, A., Goyal, V.K., Shapiro, J.H.: Photon-efficient computational 3-D and reflectivity imaging with single-photon detectors. IEEE Trans. Comput. Imaging 1(2), 112–125 (2015)

    Article  MathSciNet  Google Scholar 

  44. Shin, D., et al.: Photon-efficient imaging with a single-photon camera. Nat. Commun. 7, 12046 (2016)

    Article  Google Scholar 

  45. Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 746–760. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33715-4_54

    Chapter  Google Scholar 

  46. Villa, F., et al.: CMOS imager with 1024 SPADs and TDCs for single-photon timing and 3D time-of-flight. IEEE J. Sel. Top. Quantum Electron. 20(6), 364–373 (2014)

    Article  Google Scholar 

  47. Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)

    Google Scholar 

  48. Xiong, Z., Shi, Z., Li, H., Wang, L., Liu, D., Wu, F.: HSCNN: CNN-based hyperspectral image recovery from spectrally undersampled projections. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (2017)

    Google Scholar 

  49. Yao, M., Xiong, Z., Wang, L., Liu, D., Chen, X.: Spectral-depth imaging with deep learning based reconstruction. Opt. Express 27(26), 38312–38325 (2019)

    Article  Google Scholar 

  50. Yue, K., Sun, M., Yuan, Y., Zhou, F., Ding, E., Xu, F.: Compact generalized non-local network. In: International Conference on Neural Information Processing Systems, pp. 6510–6519 (2018)

    Google Scholar 

Download references

Acknowledgements

We acknowledge funding from National Key R&D Program of China under Grants 2017YFA0700800 and 2018YFB0504300, National Natural Science Foundation of China under Grants 61671419 and 61771443, the Shanghai Municipal Science and Technology Major Project (2019SHZDZX01), the Shanghai Science and Technology Development Funds (18JC1414700), and the Fundamental Research Funds for the Central Universities (WK2340000083).

Author information

Authors and Affiliations

  1. University of Science and Technology of China, Hefei, China

    Jiayong Peng, Zhiwei Xiong, Xin Huang, Zheng-Ping Li, Dong Liu & Feihu Xu

Authors
  1. Jiayong Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhiwei Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xin Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zheng-Ping Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Feihu Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhiwei Xiong .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 481 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Peng, J., Xiong, Z., Huang, X., Li, ZP., Liu, D., Xu, F. (2020). Photon-Efficient 3D Imaging with A Non-local Neural Network. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12351. Springer, Cham. https://doi.org/10.1007/978-3-030-58539-6_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58539-6_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58538-9

  • Online ISBN: 978-3-030-58539-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation