Skip to main content
Springer Nature Link
Log in

Photon Inhibition for Energy-Efficient Single-Photon Imaging

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15134))

Included in the following conference series:

  • 359 Accesses

Abstract

Single-photon cameras (SPCs) are emerging as sensors of choice for various challenging imaging applications. One class of SPCs based on the single-photon avalanche diode (SPAD) detects individual photons using an avalanche process; the raw photon data can then be processed to extract scene information under extremely low light, high dynamic range, and rapid motion. Yet, single-photon sensitivity in SPADs comes at a cost—each photon detection consumes more energy than that of a CMOS camera. This avalanche power significantly limits sensor resolution and could restrict widespread adoption of SPAD-based SPCs. We propose a computational-imaging approach called photon inhibition to address this challenge. Photon inhibition strategically allocates detections in space and time based on downstream inference task goals and resource constraints. We develop lightweight, on-sensor computational inhibition policies that use past photon data to disable SPAD pixels in real-time, to select the most informative future photons. As case studies, we design policies tailored for image reconstruction and edge detection, and demonstrate, both via simulations and real SPC captured data, considerable reduction in photon detections (over 90% of photons) while maintaining task performance metrics. Our work raises the question of “which photons should be detected?”, and paves the way for future energy-efficient single-photon imaging. Source code for our experiments is available at https://wisionlab.com/project/inhibition.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    It has been shown, perhaps counter-intuitively, that SPADs do not saturate even under extremely bright conditions [26, 27]. Therefore, SPADs are not restricted to low-flux environments, but are being considered for vision applications across a wide dynamic range of lighting conditions (e.g., from a dark tunnel to bright sunlight).

  2. 2.

    We borrow the term “inhibition” from the phenomenon of “lateral inhibition” found in biological vision systems [3].

  3. 3.

    Sequence \(T' := \{1,1,1,3,3,3,8,8,25\}\) yielded similar results. No extensive search over the policy space was performed.

References

  1. Arbeláez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2011). https://doi.org/10.1109/TPAMI.2010.161

    Article  Google Scholar 

  2. Ardelean, A.: Computational Imaging SPAD Cameras. Ph.D. thesis, EPFL (2023). https://doi.org/10.5075/epfl-thesis-9501

  3. Barlow, H.B.: Summation and inhibition in the frog’s retina. J. Physiol. 119(1), 69 (1953)

    Article  Google Scholar 

  4. Bian, L., et al.: High-resolution single-photon imaging with physics-informed deep learning. Nat. Commun. 14(1), 5902 (2023). https://doi.org/10.1038/s41467-023-41597-9

    Article  Google Scholar 

  5. Boso, G., Buttafava, M., Villa, F., Tosi, A.: Low-cost and compact single-photon counter based on a CMOS SPAD smart pixel. IEEE Photonics Technol. Lett. 27(23), 2504–2507 (2015)

    Article  Google Scholar 

  6. Canon Inc.: Canon Launches MS-500 - The World’s First Ultra-High-Sensitivity Interchangeable-Lens SPAD Sensor Camera. https://www.usa.canon.com/newsroom/2023/20230801-ms500, Canon Press Release 8/1/2023. Accessed 25 Feb 2024

  7. Carey, S.J., Lopich, A., Barr, D.R., Wang, B., Dudek, P.: A 100,000 fps vision sensor with embedded 535 GOPS/W 256\(\times \)256 SIMD processor array. In: 2013 Symposium on VLSI Circuits, pp. C182–C183 (2013)

    Google Scholar 

  8. Chan, S.H.: What does a one-bit quanta image sensor offer? IEEE Trans. Comput. Imaging 8, 770–783 (2022). https://doi.org/10.1109/TCI.2022.3202012

    Article  MathSciNet  Google Scholar 

  9. Charbon, E., Bruschini, C., Lee, M.J.: 3D-Stacked CMOS SPAD image sensors: technology and applications. In: 2018 25th IEEE International Conference on Electronics, Circuits and Systems (ICECS), pp. 1–4 (2018). https://doi.org/10.1109/ICECS.2018.8617983

  10. Chi, Y., Gnanasambandam, A., Koltun, V., Chan, S.H.: Dynamic low-light imaging with quanta image sensors. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12366, pp. 122–138. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58589-1_8

    Chapter  Google Scholar 

  11. Della Rocca, F.M., et al.: A 128\(\times \)128 SPAD motion-triggered time-of-flight image sensor with in-pixel histogram and column-parallel vision processor. IEEE J. Solid-State Circ. 55(7), 1762–1775 (2020). https://doi.org/10.1109/JSSC.2020.2993722

    Article  Google Scholar 

  12. Diamond, J.S.: Inhibitory interneurons in the retina: types, circuitry, and function. Ann. Rev. Vis. Sci. 3(1), 1–24 (2017). https://doi.org/10.1146/annurev-vision-102016-061345

  13. Dollar, P., Zitnick, C.L.: Structured forests for fast edge detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1841–1848 (2013)

    Google Scholar 

  14. Dutton, N.A.W., Al Abbas, T., Gyongy, I., Mattioli Della Rocca, F., Henderson, R.K.: High dynamic range imaging at the quantum limit with single photon avalanche diode-based image sensors. Sensors 18(4), 1166 (2018). https://doi.org/10.3390/s18041166

  15. Fossum, E.R.: Modeling the performance of single-bit and multi-bit quanta image sensors. IEEE J. Electron Devices Soc. 1(9), 166–174 (2013). https://doi.org/10.1109/JEDS.2013.2284054

    Article  Google Scholar 

  16. Franke, K., Berens, P., Schubert, T., Bethge, M., Euler, T., Baden, T.: Inhibition decorrelates visual feature representations in the inner retina. Nature 542(7642), 439–444 (2017). https://doi.org/10.1038/nature21394

    Article  Google Scholar 

  17. Gallego, G., et al.: Event-based vision: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 44(1), 154–180 (2020)

    Article  Google Scholar 

  18. Gnanasambandam, A., Chan, S.H.: HDR imaging with quanta image sensors: theoretical limits and optimal reconstruction. IEEE Trans. Comput. Imaging 6, 1571–1585 (2020). https://doi.org/10.1109/TCI.2020.3041093

    Article  Google Scholar 

  19. Gnanasambandam, A., Chan, S.H.: Exposure-referred signal-to-noise ratio for digital image sensors. IEEE Trans. Comput. Imaging 8, 561–575 (2022). https://doi.org/10.1109/TCI.2022.3187657

    Article  MathSciNet  Google Scholar 

  20. Gupta, M., Iso, D., Nayar, S.K.: Fibonacci exposure bracketing for high dynamic range imaging. In: 2013 IEEE International Conference on Computer Vision, pp. 1473–1480. IEEE, Sydney, Australia (2013). https://doi.org/10.1109/ICCV.2013.186

  21. Gyongy, I., Dutton, N.A., Henderson, R.K.: Direct time-of-flight single-photon imaging. IEEE Trans. Electron Devices 69(6), 2794–2805 (2021)

    Article  Google Scholar 

  22. Gyongy, I., et al.: High-speed vision with a 3D-stacked SPAD image sensor. In: Advanced Photon Counting Techniques XV. vol. 11721, p. 1172105. SPIE (2021)

    Google Scholar 

  23. Haldane, J.B.S.: On a method of estimating frequencies. Biometrika 33(3), 222–225 (1945). https://doi.org/10.1093/biomet/33.3.222

    Article  MathSciNet  Google Scholar 

  24. Henderson, R.K., et al.: 256 \(\times \) 256 40nm/90nm CMOS 3D-stacked 120 dB dynamic-range reconfigurable time-resolved SPAD imager. In: 2019 IEEE International Solid-State Circuits Conference-(ISSCC), pp. 106–108. IEEE (2019). https://doi.org/10.1109/ISSCC.2019.8662355

  25. Iams, H., Salzberg, B.: The secondary emission phototube. Proc. Inst. Radio Eng. 23(1), 55–64 (1935)

    Google Scholar 

  26. Ingle, A., et al.: Passive inter-photon imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8585–8595 (2021)

    Google Scholar 

  27. Ingle, A., Velten, A., Gupta, M.: High flux passive imaging with single-photon sensors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6760–6769 (2019)

    Google Scholar 

  28. J Yoshida: Breaking Down iPad Pro 11’s LiDAR Scanner. https://www.eetimes.com/breaking-down-ipad-pro-11s-lidar-scanner/, EE Times 6/5/2020. Accessed 5 Jun 2021

  29. Jocher, G., Chaurasia, A., Qiu, J.: Ultralytics YOLOv8 (2023). https://github.com/ultralytics/ultralytics. Accessed 3 Jun 2024

  30. Koppal, S.J., et al.: Toward wide-angle microvision sensors. IEEE Trans. Pattern Anal. Mach. Intell. 35(12), 2982–2996 (2013). https://doi.org/10.1109/TPAMI.2013.22

    Article  Google Scholar 

  31. Liu, Y., Gutierrez-Barragan, F., Ingle, A., Gupta, M., Velten, A.: Single-photon camera guided extreme dynamic range imaging. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1575–1585 (2022)

    Google Scholar 

  32. Ma, J., Masoodian, S., Starkey, D.A., Fossum, E.R.: Photon-number-resolving megapixel image sensor at room temperature without avalanche gain. Optica 4(12), 1474–1481 (2017). https://doi.org/10.1364/OPTICA.4.001474

    Article  Google Scholar 

  33. Ma, J., Zhang, D., Elgendy, O.A., Masoodian, S.: A 0.19e- rms read noise 16.7Mpixel stacked quanta image sensor with 1.1 \(M\)m-pitch backside illuminated pixels. IEEE Electron Device Lett. 42(6), 891–894 (2021). https://doi.org/10.1109/LED.2021.3072842

  34. Ma, S., Gupta, S., Ulku, A.C., Bruschini, C., Charbon, E., Gupta, M.: Quanta burst photography. ACM Trans. Graph. 39(4), 79:1–79:16 (2020). https://doi.org/10.1145/3386569.3392470

  35. Ma, S., Mos, P., Charbon, E., Gupta, M.: Burst vision using single-photon cameras. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5375–5385 (2023)

    Google Scholar 

  36. Medin, S.C., Murray-Bruce, J., Castañón, D., Goyal, V.K.: Beyond binomial and negative binomial: adaptation in Bernoulli parameter estimation. IEEE Trans. Comput. Imaging 5(4), 570–584 (2019). https://doi.org/10.1109/TCI.2019.2913108

    Article  MathSciNet  Google Scholar 

  37. Morimoto, K., et al.: 3.2 megapixel 3D-stacked charge focusing SPAD for low-light imaging and depth sensing. In: 2021 IEEE International Electron Devices Meeting (IEDM), pp. 20.2.1–20.2.4 (2021). https://doi.org/10.1109/IEDM19574.2021.9720605

  38. Morimoto, K., et al.: Megapixel time-gated SPAD image sensor for 2D and 3D imaging applications. Optica 7(4), 346–354 (2020). https://doi.org/10.1364/OPTICA.386574

    Article  Google Scholar 

  39. Morimoto, K., Charbon, E.: A scaling law for SPAD pixel miniaturization. Sensors 21(10), 3447 (2021)

    Article  Google Scholar 

  40. Ogi, J., et al.: A 3.06 \(\mu \)m SPAD pixel with embedded metal contact and power grid on deep trench pixel isolation for high-resolution photon-counting. In: 2023 International Image Sensor Workshop (IISW) (2023)

    Google Scholar 

  41. Ogi, J., et al.: A 250fps 124dB dynamic-range SPAD image sensor stacked with pixel-parallel photon counter employing sub-frame extrapolating architecture for motion artifact suppression. In: 2021 IEEE International Solid- State Circuits Conference (ISSCC). vol. 64, pp. 113–115 (2021). https://doi.org/10.1109/ISSCC42613.2021.9365977

  42. Ota, Y., et al.: A 0.37W 143dB-dynamic-range 1Mpixel backside-illuminated charge-focusing SPAD image sensor with pixel-wise exposure control and adaptive clocked recharging. In: 2022 IEEE International Solid- State Circuits Conference (ISSCC). vol. 65, pp. 94–96 (2022). https://doi.org/10.1109/ISSCC42614.2022.9731644

  43. Po, R., Pediredla, A., Gkioulekas, I.: Adaptive gating for single-photon 3D imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16354–16363 (2022)

    Google Scholar 

  44. Severini, F., Cusini, I., Berretta, D., Pasquinelli, K., Incoronato, A., Villa, F.: SPAD pixel with sub-ns dead-time for high-count rate applications. IEEE J. Sel. Top. Quantum Electron. 28(2: Optical Detectors), 1–8 (2021)

    Google Scholar 

  45. Shenzhen CM Technology company Ltd: 16 MP MIPI Camera Module with SONY IMX206 sensor. http://www.camera-module.com/product/mipicameramodule/16mp-mipi-camera-module-sony-imx206-sensor.html (2023)

  46. Sundar, V., Ardelean, A., Swedish, T., Brusschini, C., Charbon, E., Gupta, M.: SoDaCam: software-defined cameras via single-photon imaging. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2023)

    Google Scholar 

  47. Takatsuka, T., et al.: A 3.36 \(\mu \)m-pitch SPAD photon-counting image sensor using clustered multi-cycle clocked recharging technique with intermediate most-significant-bit readout. In: 2023 IEEE Symposium on VLSI Technology and Circuits (VLSI Technology and Circuits), pp. 1–2 (2023). https://doi.org/10.23919/VLSITechnologyandCir57934.2023.10185241

  48. Tilmon, B., et al.: Energy-efficient adaptive 3D sensing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5054–5063 (2023)

    Google Scholar 

  49. Ulku, A.C., et al.: A 512 \(\times \) 512 SPAD image sensor with integrated gating for widefield FLIM. IEEE J. Sel. Top. Quantum Electron. 25(1), 1–12 (2019). https://doi.org/10.1109/jstqe.2018.2867439

  50. Wang, X.: Laplacian operator-based edge detectors. IEEE Trans. Pattern Anal. Mach. Intell. 29(5), 886–890 (2007)

    Article  Google Scholar 

  51. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  52. Wayne, M.A., Bienfang, J.C., Migdall, A.L.: Low-noise photon counting above 100\(\times \)106 counts per second with a high-efficiency reach-through single-photon avalanche diode system. Appl. Phys. Lett. 118(13), 134002 (2021)

    Google Scholar 

  53. Xie, S., Tu, Z.: Holistically-nested edge detection. In: Proceedings of IEEE International Conference on Computer Vision, pp. 1395–1403 (2015)

    Google Scholar 

  54. Xu, Y., Lu, J., Wu, Z.: A compact high-speed active quenching and recharging circuit for SPAD detectors. IEEE Photonics J. 12(5), 1–8 (2020)

    Google Scholar 

  55. Yang, F., Lu, Y.M., Sbaiz, L., Vetterli, M.: Bits from photons: oversampled image acquisition using binary Poisson statistics. IEEE Trans. Image Process. 21(4), 1421–1436 (2011)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgments

The authors acknowledge the Minnesota Supercomputing Institute (MSI) at the University of Minnesota for providing resources. A.I. was supported in part by NSF ECCS-2138471. S.G and M.G. were supported in part by NSF CAREER award 1943149, NSF award CNS-2107060, and Wisconsin Alumni Research Foundation (WARF).

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, University of St. Thomas, St. Paul, MN, 55105, USA

    Lucas J. Koerner

  2. Department of Computer Sciences, University of Wisconsin-Madison, Madison, WI, 53706, USA

    Shantanu Gupta & Mohit Gupta

  3. Department of Computer Science, Portland State University, Portland, OR, 97201, USA

    Atul Ingle

Authors
  1. Lucas J. Koerner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shantanu Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Atul Ingle
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mohit Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lucas J. Koerner .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (zip 39378 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Koerner, L.J., Gupta, S., Ingle, A., Gupta, M. (2025). Photon Inhibition for Energy-Efficient Single-Photon Imaging. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15134. Springer, Cham. https://doi.org/10.1007/978-3-031-73116-7_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-73116-7_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-73115-0

  • Online ISBN: 978-3-031-73116-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics