Skip to main content
SpringerLink
Log in

Lensless Imaging with Focusing Sparse URA Masks in Long-Wave Infrared and Its Application for Human Detection

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12364))

Included in the following conference series:

Abstract

We introduce a lensless imaging framework for contemporary computer vision applications in long-wavelength infrared (LWIR). The framework consists of two parts: a novel lensless imaging method that utilizes the idea of local directional focusing for optimal binary sparse coding, and lensless imaging simulator based on Fresnel-Kirchhoff diffraction approximation. Our lensless imaging approach, besides being computationally efficient, is calibration-free and allows for wide FOV imaging. We employ our lensless imaging simulation software for optimizing reconstruction parameters and for synthetic image generation for CNN training. We demonstrate the advantages of our framework on a dual-camera system (RGB-LWIR lensless), where we perform CNN-based human detection using the fused RGB-LWIR data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Antipa, N., et al.: DiffuserCam: lensless single-exposure 3D imaging. Optica 5(1), 1 (2018)

    Article  Google Scholar 

  2. Asif, M.S., Ayremlou, A., Sankaranarayanan, A., Veeraraghavan, A., Baraniuk, R.G.: FlatCam: thin, lensless cameras using coded aperture and computation. IEEE Trans. Comput. Imaging 3(3), 384–397 (2016)

    Article  MathSciNet  Google Scholar 

  3. Barrera Campo, F., Lumbreras Ruiz, F., Sappa, A.D.: Multimodal stereo vision system: 3D data extraction and algorithm evaluation. IEEE J. Sel. Top. Signal Process. 6(5), 437–446 (2012)

    Article  Google Scholar 

  4. Barrett, H.H.: Fresnel zone plate imaging in nuclear medicine. J. Nucl. Med. 13(6), 382–385 (1972)

    Google Scholar 

  5. Benenson, R., Omran, M., Hosang, J., Schiele, B.: Ten years of pedestrian detection, what have we learned? In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8926, pp. 613–627. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16181-5_47

    Chapter  Google Scholar 

  6. Born, M., Wolf, E.: Principles of Optics: Electromagnetic Theory of Propagation, Interference and Diffraction of Light, 7th edn. Cambridge University Press, Cambridge (1999)

    Book  MATH  Google Scholar 

  7. Busboom, A., Elders-Boll, H., Schotten, H.: Uniformly redundant arrays. Exp. Astron. 8, 97–123 (1998). https://doi.org/10.1023/A:1007966830741

    Article  Google Scholar 

  8. Chakrabarti, A.: Learning sensor multiplexing design through back-propagation. In: Advances in Neural Information Processing Systems, pp. 3089–3097 (2016)

    Google Scholar 

  9. Chang, J., Wetzstein, G.: Deep optics for monocular depth estimation and 3D object detection. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), pp. 10192–10201 (2019)

    Google Scholar 

  10. Chu, Y.S., et al.: Hard-X-ray microscopy with Fresnel zone plates reaches 40 nm Rayleigh resolution. Appl. Phys. Lett. 92(10), 103119 (2008)

    Article  Google Scholar 

  11. Cieślak, M.J., Gamage, K.A., Glover, R.: Coded-aperture imaging systems: past, present and future development - a review. Radiat. Meas. 92, 59–71 (2016)

    Article  Google Scholar 

  12. DeWeert, M.J., Farm, B.P.: Lensless coded-aperture imaging with separable Doubly-Toeplitz masks. Opt. Eng. 54(2), 023102 (2015)

    Article  Google Scholar 

  13. Dollár, P., Wojek, C., Schiele, B., Perona, P.: Pedestrian detection: an evaluation of the state of the art. IEEE Trans. Pattern Anal. Mach. Intell. 34(4), 743–761 (2012)

    Article  Google Scholar 

  14. Fenimore, E.E., Cannon, T.M.: Coded aperture imaging with uniformly redundant arrays. Appl. Opt. 17(3), 337 (1978)

    Article  Google Scholar 

  15. Fenimore, E.E., Cannon, T.M.: Uniformly redundant arrays: digital reconstruction methods. Appl. Opt. 20(10), 1858 (1981)

    Article  Google Scholar 

  16. Fergus, R., Torralba, A., Freeman, W.T.: Random Lens Imaging. MIT-CSAIL-TR-2006-058, September 2006

    Google Scholar 

  17. Gade, R., Moeslund, T.B.: Thermal cameras and applications: a survey. Mach. Vision Appl. 25(1), 245–262 (2014). https://doi.org/10.1007/s00138-013-0570-5

    Article  Google Scholar 

  18. Gill, P.R., et al.: Thermal Escher sensors: pixel-efficient lensless imagers based on tiled optics. In: Optics InfoBase Conference Papers, vol. Part F46-COSI 2017, p. CTu3B.3. OSA - The Optical Society, June 2017

    Google Scholar 

  19. Goodman, J.W.: Introduction to Fourier Optics, 3rd edn. Roberts, Greenwood Village (2005)

    Google Scholar 

  20. Grulois, T., Druart, G., Guérineau, N., Crastes, A., Sauer, H., Chavel, P.: Extra-thin infrared camera for low-cost surveillance applications. Opt. Lett. 39(11), 3169 (2014)

    Article  Google Scholar 

  21. Hwang, S., Park, J., Kim, N., Choi, Y., Kweon, I.S.: Multispectral pedestrian detection: benchmark dataset and baseline. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), 07–12 June 2015, pp. 1037–1045. IEEE Computer Society, October 2015

    Google Scholar 

  22. in’t Zand, J.: A coded-mask imager as monitor of Galactic X-ray sources. Ph.D. thesis, Space Research Organization Netherlands, Sorbonnelaan 2, 3584 CA Utrecht, The Netherlands (1992)

    Google Scholar 

  23. Karasawa, T., Watanabe, K., Ha, Q., Tejero-De-Pablos, A., Ushiku, Y., Harada, T.: Multispectral object detection for autonomous vehicles. In: Proceedings of Thematic Workshops of ACM Multimedia, pp. 35–43. Association for Computing Machinery Inc, New York, October 2017

    Google Scholar 

  24. Khan, S.S., Adarsh, V.R., Boominathan, V., Tan, J., Veeraraghavan, A., Mitra, K.: Towards photorealistic reconstruction of highly multiplexed lensless images. In: Proceddings of International Conference on Computer Vision (ICCV), pp. 7859–7868. IEEE, October 2019

    Google Scholar 

  25. Kim, G., Isaacson, K., Palmer, R., Menon, R.: Lensless photography with only an image sensor. Appl. Opt. 56(23), 6450–6456 (2017)

    Article  Google Scholar 

  26. Kirz, J.: Phase zone plates for X rays and the extreme UV. J. Opt. Soc. Am. 64(3), 301–309 (1974)

    Article  Google Scholar 

  27. Konig, D., Adam, M., Jarvers, C., Layher, G., Neumann, H., Teutsch, M.: Fully convolutional region proposal networks for multispectral person detection. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), July 2017, pp. 243–250. IEEE Computer Society, August 2017

    Google Scholar 

  28. Leykin, A., Ran, Y., Hammoud, R.: Thermal-visible video fusion for moving target tracking and pedestrian classification. In: Proceedings of Computer Vision and Pattern Recognition (CVPR) (2007)

    Google Scholar 

  29. Liu, J., Zhang, S., Wang, S., Metaxas, D.N.: Multispectral deep neural networks for pedestrian detection. In: British Machine Vision Conference (BMVC), September 2016, pp. 73.1–73.13 (2016)

    Google Scholar 

  30. Metzler, C.A., Ikoma, H., Peng, Y., Wetzstein, G.: Deep optics for single-shot high-dynamic-range imaging. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), August 2020

    Google Scholar 

  31. Miezianko, R., Pokrajac, D.: People detection in low resolution infrared videos. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2008)

    Google Scholar 

  32. Mudau, A.E., Willers, C.J., Griffith, D., Le Roux, F.P.: Non-uniformity correction and bad pixel replacement on LWIR and MWIR images. In: Saudi International Electronics, Communications and Photonics Conference (SIECPC) (2011)

    Google Scholar 

  33. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017)

    Article  Google Scholar 

  34. Sitzmann, V., et al.: End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Trans. Graph. 37(4), 1–13 (2018)

    Article  Google Scholar 

  35. Stylianou, A., Pless, R.: SparkleGeometry: glitter imaging for 3D point tracking. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 919–926. IEEE Computer Society, December 2016

    Google Scholar 

  36. Sun, Q., Zhang, J., Dun, X., Ghanem, B., Peng, Y., Heidrich, W.: End-to-end learned, optically coded super-resolution SPAD camera. ACM Trans. Graph. 39(2), 1–14 (2020)

    Article  Google Scholar 

  37. Tanida, J., et al.: Thin observation module by bound optics (TOMBO): concept and experimental verification. Appl. Opt. 40(11), 1806 (2001)

    Article  Google Scholar 

  38. Tikhonov, A.N., Arsenin, V.Y.: Solutions of Ill-Posed Problems. W.H. Winston, Washington (1977)

    MATH  Google Scholar 

  39. Wagner, J., Fischer, V., Herman, M., Behnke, S.: Multispectral pedestrian detection using deep fusion convolutional neural networks. In: Proceedings of European Symposium on Artificial Neural Networks (ESANN) (2016)

    Google Scholar 

  40. Wang, Z., Simoncelli, E.P.: Translation insensitive image similarity in complex wavelet domain. In: Proceedings of International Conference on Acoustics, Speech and Signal Processing (ICASSP), vol. II (2005)

    Google Scholar 

  41. Wu, Y., Boominathan, V., Chen, H., Sankaranarayanan, A., Veeraraghavan, A.: PhaseCam3D - learning phase masks for passive single view depth estimation (2019)

    Google Scholar 

  42. Xu, D., Ouyang, W., Ricci, E., Wang, X., Sebe, N.: Learning cross-modal deep representations for robust pedestrian detection. In: Proceedings of Computer Vision and Pattern Recognition (CVPR), January 2017, pp. 4236–4244. Institute of Electrical and Electronics Engineers Inc., November 2017

    Google Scholar 

  43. Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Sony Corporation, Tokyo, Japan

    Ilya Reshetouski, Hideki Oyaizu, Kenichiro Nakamura, Ryuta Satoh, Suguru Ushiki, Ryuichi Tadano, Atsushi Ito & Jun Murayama

Authors
  1. Ilya Reshetouski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hideki Oyaizu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kenichiro Nakamura
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ryuta Satoh
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Suguru Ushiki
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ryuichi Tadano
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Atsushi Ito
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jun Murayama
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ilya Reshetouski .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 2 (mp4 27273 KB)

Supplementary material 1 (pdf 306 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Reshetouski, I. et al. (2020). Lensless Imaging with Focusing Sparse URA Masks in Long-Wave Infrared and Its Application for Human Detection. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12364. Springer, Cham. https://doi.org/10.1007/978-3-030-58529-7_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58529-7_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58528-0

  • Online ISBN: 978-3-030-58529-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation