Skip to main content
Springer Nature Link
Log in

Light-in-Flight for a World-in-Motion

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15092))

Included in the following conference series:

  • 199 Accesses

Abstract

Although time-of-flight (ToF) cameras are becoming the sensor-of-choice for numerous 3D imaging applications in robotics, augmented reality (AR) and human-computer interfaces (HCI), they do not explicitly consider scene or camera motion. Consequently, current ToF cameras do not provide 3D motion information, and the estimated depth and intensity often suffer from significant motion artifacts in dynamic scenes. In this paper, we propose a novel ToF imaging method for dynamic scenes, with the goal of simultaneously estimating 3D geometry, intensity, and 3D motion using a single indirect ToF (I-ToF) camera. Our key observation is that we can estimate 3D motion, as well as motion artifact-free depth and intensity by designing optical-flow-like algorithms that operate on coded correlation images captured by an I-ToF camera. Through the integration of a multi-frequency I-ToF approach with burst imaging, we demonstrate high-quality all-in-one (3D geometry, intensity, 3D motion) imaging even in challenging low signal-to-noise ratio scenarios. We show the effectiveness of our approach through thorough simulations and real experiments conducted across a wide range of motion and imaging scenarios, including indoor and outdoor dynamic scenes.

R. J. Suess—Independent Reseracher.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    A short integration time is also required for instantaneous motion estimation.

  2. 2.

    We assume a unipolar demodulation function \(\left( 0 \le D(t) \le 1 \right) \) for ease of noise analysis. The same analysis can be extended to a bipolar demodulation function \(\left( -1 \le D(t) \le 1 \right) \). See the supplementary report.

  3. 3.

    We assume \(Z\ne 0\).

  4. 4.

    Most I-ToF cameras provide a temporal stream of the correlation image sets.

  5. 5.

    Due to motion, the absolute value of the estimated intensity image does not preserve its brightness even along the true motion. See the supplementary report.

  6. 6.

    The intensity image I records the signal photons, while the image used in traditional optical flow records the background photons (e.g., sunlight reflected from the scene).

  7. 7.

    We simulated velocity estimation from depth difference and Doppler ToF under Poisson noise. \(\sigma _{v_{\varDelta z}}\) and \(\sigma _{v_{\varDelta f}}\) were computed from 1000 repetitions.

References

  1. http://www.ignorancia.org/index.php?page=lightsys. Accessed 14 July 2024

  2. http://www.povray.org/. Accessed 14 July 2024

  3. 3d depth sensing development kits, pmd. https://3d.pmdtec.com/en/3d-cameras/flexx2/. Accessed 14 July 2024

  4. Azure kinect dk, microsoft. https://www.microsoft.com/en-us/d/azure-kinect-dk/8pp5vxmd9nhq?activetab=pivot:overviewtab. Accessed 14 July 2024

  5. Time-of-flight sensors, texas instruments. https://www.ti.com/sensors/specialty-sensors/time-of-flight/products.html. Accessed 14 July 2024

  6. Agresti, G., Schaefer, H., Sartor, P., Zanuttigh, P.: Unsupervised domain adaptation for tof data denoising with adversarial learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5584–5593 (2019)

    Google Scholar 

  7. Agresti, G., Zanuttigh, P.: Deep learning for multi-path error removal in tof sensors. In: Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp. 0–0 (2018)

    Google Scholar 

  8. Attal, B., et al.: Törf: time-of-flight radiance fields for dynamic scene view synthesis. Adv. Neural. Inf. Process. Syst. 34, 26289–26301 (2021)

    Google Scholar 

  9. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) Computer Vision - ECCV 2004: 8th European Conference on Computer Vision, Prague, Czech Republic, May 11-14, 2004. Proceedings, Part IV, pp. 25–36. Springer Berlin Heidelberg, Berlin, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24673-2_3

    Chapter  Google Scholar 

  10. Dong, G., Zhang, Y., Xiong, Z.: Spatial hierarchy aware residual pyramid network for time-of-flight depth denoising. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIV, pp. 35–50. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-58586-0_3

    Chapter  Google Scholar 

  11. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: Carla: an open urban driving simulator. In: Conference on Robot Learning, pp. 1–16. PMLR (2017)

    Google Scholar 

  12. Godard, C., Matzen, K., Uyttendaele, M.: Deep burst denoising. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 538–554 (2018)

    Google Scholar 

  13. Hasinoff, S.W., et al.: Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Trans. Graph.(ToG) 35(6), 1–12 (2016)

    Article  Google Scholar 

  14. Heide, F., Diamond, S., Nießner, M., Ragan-Kelley, J., Heidrich, W., Wetzstein, G.: Proximal: efficient image optimization using proximal algorithms. ACM Trans. Graph. (TOG) 35(4), 1–15 (2016)

    Article  Google Scholar 

  15. Heide, F., Heidrich, W., Hullin, M., Wetzstein, G.: Doppler time-of-flight imaging. ACM Trans. Graph. (ToG) 34(4), 1–11 (2015)

    Article  Google Scholar 

  16. Heide, F., et al.: Flexisp: a flexible camera image processing framework. ACM Trans. Graph. (ToG) 33(6), 1–13 (2014)

    Article  Google Scholar 

  17. Hoegg, T., Lefloch, D., Kolb, A.: Real-time motion artifact compensation for PMD-ToF images. In: Grzegorzek, M., Theobalt, C., Koch, R., Kolb, A. (eds.) Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications, pp. 273–288. Springer Berlin Heidelberg, Berlin, Heidelberg (2013). https://doi.org/10.1007/978-3-642-44964-2_13

    Chapter  Google Scholar 

  18. Horn, B.K., Schunck, B.G.: Determining optical flow. Artif. Intell. 17(1–3), 185–203 (1981)

    Article  Google Scholar 

  19. Hu, Y., Miyashita, L., Ishikawa, M.: Differential frequency heterodyne time-of-flight imaging for instantaneous depth and velocity estimation. ACM Trans. Graph. (TOG) 42(1), 1–13 (2022)

    Article  Google Scholar 

  20. Hussmann, S., Hermanski, A., Edeler, T.: Real-time motion artifact suppression in tof camera systems. IEEE Trans. Instrum. Meas. 60(5), 1682–1690 (2011)

    Article  Google Scholar 

  21. Jaimez, M., Souiai, M., Gonzalez-Jimenez, J., Cremers, D.: A primal-dual framework for real-time dense rgb-d scene flow. In: 2015 IEEE International Conference on Robotics And Automation (ICRA), pp. 98–104. IEEE (2015)

    Google Scholar 

  22. Jongenelen, A.P., Bailey, D.G., Payne, A.D., Dorrington, A.A., Carnegie, D.A.: Analysis of errors in tof range imaging with dual-frequency modulation. IEEE Trans. Instrum. Meas. 60(5), 1861–1868 (2011)

    Article  Google Scholar 

  23. Lange, R.: 3D ToF distance measurement with custom solid-state image sensors in cmos-ccd-technology. Ph.D. Thesis (2000)

    Google Scholar 

  24. Lange, R., Seitz, P., Biber, A., Lauxtermann, S.C.: Demodulation pixels in ccd and cmos technologies for time-of-flight ranging, vol. 3965 (2000)

    Google Scholar 

  25. Lefloch, D., Hoegg, T., Kolb, A.: Real-time motion artifacts compensation of tof sensors data on gpu. In: Three-Dimensional Imaging, Visualization, and Display 2013. vol. 8738, pp. 166–172. SPIE (2013)

    Google Scholar 

  26. Letouzey, A., Petit, B., Boyer, E.: Scene flow from depth and color images. In: BMVC 2011-British Machine Vision Conference, pp. 46–1. BMVA Press (2011)

    Google Scholar 

  27. Lindner, M., Kolb, A.: Compensation of motion artifacts for time-of-flight cameras. In: Kolb, A., Koch, R. (eds.) Dyn3D 2009. LNCS, vol. 5742, pp. 16–27. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-03778-8_2

    Chapter  Google Scholar 

  28. Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: IJCAI’81: 7th International Joint Conference on Artificial Intelligence, vol. 2, pp. 674–679 (1981)

    Google Scholar 

  29. Ma, S., Gupta, S., Ulku, A.C., Bruschini, C., Charbon, E., Gupta, M.: Quanta burst photography. ACM Trans. Graph. (TOG) 39(4), 1–79 (2020)

    Article  Google Scholar 

  30. Marco, J., et al.: Deeptof: off-the-shelf real-time correction of multipath interference in time-of-flight imaging. ACM Trans. Graph. (ToG) 36(6), 1–12 (2017)

    Article  Google Scholar 

  31. Mildenhall, B., Barron, J.T., Chen, J., Sharlet, D., Ng, R., Carroll, R.: Burst denoising with kernel prediction networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2502–2510 (2018)

    Google Scholar 

  32. Payne, A.D., Jongenelen, A.P., Dorrington, A.A., Cree, M.J., Carnegie, D.A.: Multiple frequency range imaging to remove measurement ambiguity. In: Optical 3-d measurement techniques (2009)

    Google Scholar 

  33. Payne, J.M.: An optical distance measuring instrument. Rev. Sci. Instrum. 44(3) (1973)

    Google Scholar 

  34. Su, S., Heide, F., Wetzstein, G., Heidrich, W.: Deep end-to-end time-of-flight imaging. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6383–6392 (2018)

    Google Scholar 

  35. Sun, D., Sudderth, E.B., Pfister, H.: Layered rgbd scene flow estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 548–556 (2015)

    Google Scholar 

  36. Teed, Z., Deng, J.: RAFT: recurrent all-pairs field transforms for optical flow. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II, pp. 402–419. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_24

    Chapter  Google Scholar 

  37. Vedula, S., Rander, P., Collins, R., Kanade, T.: Three-dimensional scene flow. IEEE Trans. Pattern Anal. Mach. Intell. 27(3), 475–480 (2005)

    Article  Google Scholar 

Download references

Acknowledgement

This research was supported in part by NSF CAREER award 1943149, Cruise LLC, WARF, and ONR award N00014-24-1-2155.

Author information

Authors and Affiliations

  1. University of Wisconsin-Madison, Madison, WI, 53706, USA

    Jongho Lee & Mohit Gupta

  2. Seattle, USA

    Ryan J. Suess

Authors
  1. Jongho Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ryan J. Suess
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohit Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jongho Lee .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Germany

    Stefan Roth

  4. Princeton University, Princeton, NJ, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1549 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lee, J., Suess, R.J., Gupta, M. (2025). Light-in-Flight for a World-in-Motion. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15092. Springer, Cham. https://doi.org/10.1007/978-3-031-72754-2_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72754-2_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72753-5

  • Online ISBN: 978-3-031-72754-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics