skip to main content
10.1145/3388769.3407481acmconferencesArticle/Chapter ViewAbstractPublication PagessiggraphConference Proceedingsconference-collections
course

Computational time-resolved imaging, single-photon sensing, and non-line-of-sight imaging

Published: 17 August 2020 Publication History

Abstract

Emerging detector technologies are capable of ultrafast capture of single photons, enabling imaging at the speed of light. Not only can these detectors be used for imaging at essentially trillion frame-per-second rates, but combining them with computational algorithms has given rise to unprecedented new imaging capabilities. Computational time-resolved imaging has enabled new techniques for 3D imaging, light transport analysis, imaging around corners or behind occluders, and imaging through scattering media such as fog, murky water, or human tissue. With applications in autonomous navigation, robotic vision, human-computer interaction, and more, this is an area of rapidly growing interest. In this course, we provide an introduction to computational time-resolved imaging and single photon sensing with a focus on hardware, applications, and algorithms. We describe various types of emerging single-photon detectors, including single-photon avalanche diodes and avalanche photodiodes, which are among the most popular time-resolved detectors. Physically accurate models for these detectors are described, including modeling parameters and noise statistics used in most computational algorithms. From the application side, we discuss the use of ultrafast active illumination for 3D imaging and transient imaging, and we describe the state of the art in non-line-of-sight imaging, which requires modelling and inverting the propagation and scattering of light from a visible surface to a hidden object and back. We describe time-resolved computational algorithms used in each of these applications and offer insights on potential future directions.

Supplementary Material

VTT File (3388769.3407481.vtt)
MP4 File (3388769.3407481.mp4)
Presentation video

References

[1]
N. Abramson. Light-in-flight recording by holography. Opt. Lett., 3(4):121--123, Oct 1978. 5, 6
[2]
B. Ahn, A. Dave, A. Veeraraghavan, I. Gkioulekas, and A. C. Sankaranarayanan. Convolutional approximations to the general non-line-of-sight imaging operator. In Proc. ICCV, 2019. 7
[3]
V. Arellano, D. Gutierrez, and A. Jarabo. Fast back-projection for non-line of sight reconstruction. Opt. Express, 25(10):11574--11583, 2017. 8
[4]
M. Buttafava, J. Zeman, A. Tosi, K. Eliceiri, and A. Velten. Non-line-of-sight imaging using a time-gated single photon avalanche diode. Opt. Express, 23(16):20997--21011, 2015. 7
[5]
S. Chan, R. E. Warburton, G. Gariepy, J. Leach, and D. Faccio. Non-line-of-sight tracking of people at long range. Opt. Express, 25(9):10109--10117, 2017. 7
[6]
G. Gariepy, F. Tonolini, R. Henderson, J. Leach, and D. Faccio. Detection and tracking of moving objects hidden from view. Nat. Photonics, 10, Jan 2016. 7
[7]
I. Gkioulekas, A. Levin, F. Durand, and T. Zickler. Micron-scale light transport decomposition using inter-ferometry. ACM Trans. Graph., 34(4):37:1--37:14, July 2015. 5, 6, 7
[8]
O. Gupta, T. Willwacher, A. Velten, A. Veeraraghavan, and R. Raskar. Reconstruction of hidden 3d shapes using diffuse reflections. Opt. Express, 20(17):19096--19108, 2012. 8
[9]
F. Heide, S. Diamond, D. B. Lindell, and G. Wetzstein. Sub-picosecond photon-efficient 3d imaging using single-photon sensors. Sci. Rep., 8(1), 2018. 5, 7
[10]
F. Heide, M. B. Hullin, J. Gregson, and W. Heidrich. Low-budget transient imaging using photonic mixer devices. ACM Trans. Graph., 32(4):45:1--45:10, July 2013. 5, 6
[11]
F. Heide, M. O'Toole, K. Zang, D. B. Lindell, S. Diamond, and G. Wetzstein. Non-line-of-sight imaging with partial occluders and surface normals. ACM Trans. Graph., 38(3), 2019. 8
[12]
F. Heide, L. Xiao, W. Heidrich, and M. B. Hullin. Diffuse mirrors: 3d reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In Proc. CVPR, 2014. 7
[13]
J. Iseringhausen and M. B. Hullin. Non-line-of-sight reconstruction using efficient transient rendering. ACM Trans. Graph., 39(1):1--14, 2020. 8
[14]
A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar. Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. ACM Trans. Graph., 32(6):167, 2013. 5
[15]
A. Kirmani, T. Hutchison, J. Davis, and R. Raskar. Looking around the corner using transient imaging. In Proc. ICCV, 2009. 7
[16]
J. Klein, C. Peters, J. Martín, M. Laurenzis, and M. B. Hullin. Tracking objects outside the line of sight using 2d intensity images. Sci. Rep., 6:32491, 2016. 7
[17]
H. Kubo, S. Jayasuriya, T. Iwaguchi, T. Funatomi, Y. Mukaigawa, and S. G. Narasimhan. Programmable non-epipolar indirect light transport: Capture and analysis. IEEE Trans. Vis. Comput. Graph., 2019. 5
[18]
M. La Manna, F. Kine, E. Breitbach, J. Jackson, T. Sultan, and A. Velten. Error backprojection algorithms for non-line-of-sight imaging. IEEE Trans. Pattern Anal. Mach. Intell., 2018. 8
[19]
D. B. Lindell, M. O'Toole, and G. Wetzstein. Single-photon 3d imaging with deep sensor fusion. ACM Trans. Graph., 37(4), 2018. 5
[20]
D. B. Lindell, M. O'Toole, and G. Wetzstein. Towards transient imaging at interactive rates with single-photon detectors. In Proc. ICCP, 2018. 5, 7
[21]
D. B. Lindell, G. Wetzstein, and M. O'Toole. Wave-based non-line-of-sight imaging using fast f-k migration. ACM Trans. Graph., 38(4):116, 2019. 5, 7, 8
[22]
X. Liu, S. Bauer, and A. Velten. Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems. Nat. Commun., 11(1):1--13, 2020. 8
[23]
X. Liu, I. Guillén, M. La Manna, J. H. Nam, S. A. Reza, T. H. Le, A. Jarabo, D. Gutierrez, and A. Velten. Non-line-of-sight imaging using phasor-field virtual wave optics. Nature, 572(7771):620--623, 2019. 5
[24]
M. O'Toole, F. Heide, D. B. Lindell, K. Zang, S. Diamond, and G. Wetzstein. Reconstructing transient images from single-photon sensors. In Proc. CVPR, 2017. 5, 6, 7
[25]
M. O'Toole, D. B. Lindell, and G. Wetzstein. Real-time non-line-of-sight imaging. In ACM SIGGRAPH ETECH, 2018. 7
[26]
M. O'Toole, D. B. Lindell, and G. Wetzstein. Confocal non-line-of-sight imaging based on the light-cone transform. Nature, 555(7696):338, 2018. 5, 7, 8
[27]
A. K. Pediredla, M. Buttafava, A. Tosi, O. Cossairt, and A. Veeraraghavan. Reconstructing rooms using photon echoes: A plane based model and reconstruction algorithm for looking around the corner. In Proc. ICCP, 2017. 7
[28]
G. Satat, B. Heshmat, D. Raviv, and R. Raskar. All photons imaging through volumetric scattering. Sci. Rep., 6:33946, 2016. 5
[29]
G. Satat, M. Tancik, O. Gupta, B. Heshmat, and R. Raskar. Object classification through scattering media with deep learning on time resolved measurement. Opt. express, 25(15):17466--17479, 2017. 5
[30]
G. Satat, M. Tancik, and R. Raskar. Towards photography through realistic fog. In Proc. ICCP, 2018. 5
[31]
C. Saunders, J. Murray-Bruce, and V. K. Goyal. Computational periscopy with an ordinary digital camera. Nature, 565(7740):472, 2019. 7
[32]
B. Schwarz. Mapping the world in 3d. Nat. Photonics, 4(7):429--430, 2010. 5
[33]
D. Shin, F. Xu, D. Venkatraman, R. Lussana, F. Villa, F. Zappa, V. K. Goyal, F. N. C. Wong, and J. H. Shapiro. Photon-efficient imaging with a single-photon camera. Nat. Commun., 7(12046):1--7, 2016. 7
[34]
O. Stavroudis. The optics of rays, wavefronts, and caustics, volume 38. Elsevier, 2012. 8
[35]
S. Su, F. Heide, R. Swanson, J. Klein, C. Callenberg, M. Hullin, and W. Heidrich. Material classification using raw time-of-flight measurements. In Proc. CVPR, June 2016. 5
[36]
C.-Y. Tsai, K. N. Kutulakos, S. G. Narasimhan, and A. C. Sankaranarayanan. The geometry of first-returning photons for non-line-of-sight imaging. In Proc. CVPR, 2017. 5, 7
[37]
C.-Y. Tsai, A. Sankaranarayanan, and I. Gkioulekas. Beyond volumetric albedo---A surface optimization framework for non-line-of-sight imaging. In Proc. CVPR, 2019. 8
[38]
A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nat. Commun., 3, 2012. 5, 6, 7, 8
[39]
A. Velten, D. Wu, A. Jarabo, B. Masia, C. Barsi, C. Joshi, E. Lawson, M. Bawendi, D. Gutierrez, and R. Raskar. Femto-photography: capturing and visualizing the propagation of light. ACM Trans. Graph., 32(4), 2013. 5
[40]
D. Wu, A. Velten, M. O'Toole, B. Masia, A. Agrawal, Q. Dai, and R. Raskar. Decomposing global light transport using time of flight imaging. Int. J. Comput. Vision, 107(2):123--138, Apr. 2014. 5
[41]
S. Xin, S. Nousias, K. N. Kutulakos, A. C. Sankaranarayanan, S. G. Narasimhan, and I. Gkioulekas. A theory of Fermat paths for non-line-of-sight shape reconstruction. In Proc. CVPR, 2019. 5, 8
[42]
F. Xu, G. Shulkind, C. Thrampoulidis, J. H. Shapiro, A. Torralba, F. N. Wong, and G. W. Wornell. Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging. Opt. Express, 26(8):9945--9962, 2018. 7

Cited By

View all
  • (2023)Computational Imaging and Artificial Intelligence: The Next Revolution of Mobile VisionProceedings of the IEEE10.1109/JPROC.2023.3338272111:12(1607-1639)Online publication date: Dec-2023

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGGRAPH '20: ACM SIGGRAPH 2020 Courses
August 2020
3010 pages
ISBN:9781450379724
DOI:10.1145/3388769
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 August 2020

Check for updates

Qualifiers

  • Course

Conference

SIGGRAPH '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)38
  • Downloads (Last 6 weeks)3
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Computational Imaging and Artificial Intelligence: The Next Revolution of Mobile VisionProceedings of the IEEE10.1109/JPROC.2023.3338272111:12(1607-1639)Online publication date: Dec-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media