Skip to main content
SpringerLink
Log in

Deblur and deep depth from single defocus image

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

In this paper, we tackle depth estimation and blur removal from a single out-of-focus image. Previously, depth is estimated, and blurred is removed using multiple images; for example, from multiview or stereo scenes, but doing so with a single image is challenging. Earlier works of monocular images for depth estimated and deblurring either exploited geometric characteristics or priors using hand-crafted features. Lately, there is enough evidence that deep convolutional neural networks (CNN) significantly improved numerous vision applications; hence, in this article, we present a depth estimation method that leverages rich representations learned from cascaded convolutional and fully connected neural networks operating on a patch-pooled set of feature maps. Furthermore, from this depth, we computationally reconstruct an all-focus image, i.e., removing the blur and achieve synthetic re-focusing, all from a single image. Our method is fast, and it substantially improves depth accuracy over the state-of-the-art alternatives. Our proposed depth estimation approach can be utilized for everyday scenes without any geometric priors or extra information. Furthermore, our experiments on two benchmark datasets consist images of indoor and outdoor scenes, i.e., Make3D and NYU-v2 demonstrate superior performance in comparison with other available depth estimation state-of-the-art methods by reducing the root-mean-squared error by 57% and 46%, and state-of-the-art blur removal methods by 0.36 dB and 0.72 dB in PSNR, respectively. This improvement in-depth estimation and deblurring is further demonstrated by the superior performance using real defocus images against images captured with a prototype lens.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. Taken from literature for fair comparison.

References

  1. Zhu, X., Ramanan, D.: Face detection, pose estimation, and landmark localization in the wild. in CVPR (2012)

  2. Eigen, D., Fergus, R.: Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. in ICCV (2015)

  3. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. in CVPR (2001)

  4. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. in CVPR (2016)

  5. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. in CVPR (2015)

  6. Fergus, R., Singh, B., Hertzmann, A., Roweis, S. T., Freeman, W. T.: Removing camera shake from a single photograph (2006)

  7. Levin, A.: Blind motion deblurring using image statistics. in NIPS (2006)

  8. Shan, Q., Jia, J., Agarwala, A.: High-quality motion deblurring from a single image. ACM Trans. Graph (2008)

  9. Cho, S., Lee, S.: Fast motion deblurring. ACM Trans. Gr. (TOG) (2009)

  10. Levin, A., Weiss, Y., Durand, F., Freeman, W. T.: Understanding blind deconvolution algorithms. TPAMI (2011)

  11. Nayar, S. K., Ben-Ezra, M.: Motion-based motion deblurring. TPAMI (2004)

  12. Li, F., Yu, J., Chai, J.: A hybrid camera for motion deblurring and depth map super-resolution. in CVPR (2008)

  13. Tai, Y.-W., Du, H., Brown, M. S., Lin, S.: Image/video deblurring using a hybrid camera. in CVPR (2008)

  14. Yuan, L., Sun, J., Quan, L., Shum, H.-Y.: Image deblurring with blurred/noisy image pairs. Ser. SIGGRAPH (2007)

  15. Nathan Silberman, P. K., Hoiem, D., Fergus, R.: Indoor segmentation and support inference from RGBD images. in ECCV (2012)

  16. Anwar, S., Hayder, Z., Porikli, F.: Depth estimation and blur removal from a single out-of-focus image. BMVC 1, 2 (2017)

    Google Scholar 

  17. Levin, A., Fergus, R., Durand, F., Freeman, W.T.: Image and depth from a conventional camera with a coded aperture. ACM Trans. Gr. (2007)

  18. Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., Tumblin, J.: Dappled photography: Mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Gr. (2007)

  19. Moreno-Noguer, F., Belhumeur, P. N., Nayar, S. K.: Active refocusing of images and videos. ACM Trans. Gr. (2007)

  20. Zhou, C., Cossairt, O., Nayar, S.: Depth from diffusion. in CVPR (2010)

  21. Zhou, C., Nayar, S.: What are good apertures for defocus deblurring? in ICCP (2009)

  22. Zhou, C., Lin, S., Nayar, S. K.: Coded aperture pairs for depth from defocus and defocus deblurring. IJCV (2011)

  23. Levin, A.: Analyzing depth from coded aperture sets. in ECCV (2010)

  24. Pertuz, S., Puig, D., Garcia, M. A.: Analysis of focus measure operators for shape-from-focus. PR (2013)

  25. Mahmood, M., Choi, T. S.: Nonlinear approach for enhancement of image focus volume in shape from focus. TIP (2012)

  26. Shim, S. O., Choi, T. S.: A fast and robust depth estimation method for 3D cameras. in ICCE (2012)

  27. Subbarao, M., Choi, T.: Accurate recovery of three-dimensional shape from image focus. TPAMI (1995)

  28. Bae, S., Durand, F.: Defocus magnification. CG Forum (2007)

  29. Calderero, F., Caselles, V.: Recovering relative depth from low-level features without explicit t-junction detection and interpretation. IJCV (2013)

  30. Cao, Y., Fang, S., Wang, F.: Single image multi-focusing based on local blur estimation. in ICIG (2011)

  31. Zhuo, S., Sim, T.: Defocus map estimation from a single image. PR (2011)

  32. Namboodiri, V. P., Chaudhuri, S.: Recovery of relative depth from a single observation using an uncalibrated (real-aperture) camera. in CVPR (2008)

  33. Liu, M., Salzmann, M., He, X.: Discrete-continuous depth estimation from a single image. in CVPR (2014)

  34. Liu, F., Shen, C., Lin, G.: Deep convolutional neural fields for depth estimation from a single image. in CVPR (2015)

  35. Watanabe, M., Nayar, S. K.: Rational filters for passive depth from defocus. IJCV (1998)

  36. Paramanand, C., Rajagopalan, A. N.: Non-uniform motion deblurring for bilayer scenes. in CVPR (2013)

  37. Xu, L., Jia, J.: Depth-aware motion deblurring. in ICCP (2012)

  38. Li, C., Su, S., Matsushita, Y., Zhou, K., Lin, S.: Bayesian depth-from-defocus with shading constraints. in CVPR (2013)

  39. Farid, M.S., Mahmood, A., Al-Maadeed, S.A.: Multi-focus image fusion using content adaptive blurring. Inf Fus 45, 96–112 (2019)

    Article  Google Scholar 

  40. Krizhevsky, A., Sutskever, I., Hinton, G. E.: Imagenet classification with deep convolutional neural networks. in NIPS (2012)

  41. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  42. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. in CVPR (2014)

  43. Girshick, R.: Fast r-CNN. in ICCV (2015)

  44. Razavian, A., Azizpour, H., Sullivan, J., Carlsson, S.: CNN features off-the-shelf: an astounding baseline for recognition. in CVPR Workshops (2014)

  45. Su, H., Huang, Q., Mitra, N. J., Li, Y., Guibas, L.: Estimating image depth using shape collections. TG (2014)

  46. Kar, A., Tulsiani, S., Carreira, J., Malik, J.: Category-specific object reconstruction from a single image. in CVPR (2015)

  47. Eigen, D., Puhrsch, C., Fergus, R.: Depth map prediction from a single image using a multi-scale deep network. in NIPS (2014)

  48. Li, J., Guo, X., Lu, G., Zhang, B., Xu, Y., Wu, F., Zhang, D.: Drpl: Deep regression pair learning for multi-focus image fusion. IEEE Trans Image Process 29, 4816–4831 (2020)

    Article  Google Scholar 

  49. Krishnan, D., Tay, T., Fergus, R.: Blind deconvolution using a normalized sparsity measure. in CVPR (2011)

  50. Xu, L., Jia, J.: Two-phase kernel estimation for robust motion deblurring. in ECCV (2010)

  51. Whyte, O., Sivic, J., Zisserman, A., Ponce, J.: Non-uniform deblurring for shaken images. IJCV (2012)

  52. Joshi, N., Szeliski, R., Kriegman, D.: PSF estimation using sharp edge prediction. in CVPR (2008)

  53. Cho, T. S., Paris, S., Horn, B. K., Freeman, W. T.: Blur kernel estimation using the radon transform. in CVPR (2011)

  54. Krishnan, D., Fergus, R.: Fast image deconvolution using hyper-Laplacian priors. in NIPS (2009)

  55. Whyte, O., Sivic, J., and Zisserman A.: Deblurring shaken and partially saturated images. IJCV (2014)

  56. Pan, J., Hu, Z., Su, Z., Yang, M. H.: Deblurring text images via L0 regularized intensity and gradient prior. in CVPR (2014)

  57. Zoran, D., Weiss, Y.: From learning models of natural image patches to whole image restoration. in ICCV (2011)

  58. Sun, L., Cho, S., Wang, J., Hays, J.: Edge-based blur Kernel estimation using patch priors. in ICCP (2013)

  59. Michaeli, T., Irani, M.: Blind deblurring using internal patch recurrence. in ECCV (2014)

  60. Schuler, C. J., Hirsch, M., Harmeling, S., Schölkopf, B.: Learning to deblur. TPAMI (2016)

  61. Chakrabarti, A.: A neural approach to blind motion deblurring. in ECCV (2016)

  62. Anwar, S., Phuoc Huynh, C., Porikli, F.: Class-specific image deblurring. in ICCV (2015)

  63. Anwar, S., Huynh, C. P., Porikli, F.: Image deblurring with a class-specific prior. TPAMI (2017)

  64. Joshi, N., Matusik, W., Adelson, E.H., Kriegman, D.J.: Personal photo enhancement using example images. ACM Trans. Gr. (2010)

  65. Hacohen, Y., Shechtman, E., Lischinski, D.: Deblurring by example using dense correspondence. in ICCV. (2013)

  66. Sun, L., Cho, S., Wang, J., Hays, J.: Good image priors for non-blind deconvolution—generic versus specific. in ECCV (2014)

  67. Pan, J., Hu, Z., Su, Z., Yang, M.: Deblurring face images with exemplars. in ECCV (2014)

  68. Saxena, A., Sun, M., Ng, A. Y.: Make3d: Learning 3d scene structure from a single still image. TPAMI (2009)

  69. He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. in ECCV (2014)

  70. Levin, A., Zomet, A., Weiss, Y.: Learning how to inpaint from global image statistics. in ICCV (2003)

  71. Karsch, K., Liu, C., Kang, S. B.: Depth transfer: depth extraction from video using non-parametric sampling. TPAMI. (2014)

  72. Chakrabarti, A., Zickler, T.: Depth and deblurring from a spectrally-varying depth-of-field. in ECCV (2012)

  73. Levin, A., Weiss, Y., Durand, F., Freeman, W. T.: Efficient marginal likelihood optimization in blind deconvolution. in CVPR (2011)

  74. Cho, S., Lee, S.: Fast motion deblurring. Ser SIGGRAPH Asia (2009)

Download references

Author information

Authors and Affiliations

  1. College of Engineering and Computer Science (CECS), The Australian National University (ANU), Canberra, Australia

    Saeed Anwar, Zeeshan Hayder & Fatih Porikli

  2. Data61, The Commonwealth Scientific and Industrial Research Organisation (CSIRO), Canberra, Australia

    Saeed Anwar

Authors
  1. Saeed Anwar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zeeshan Hayder
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fatih Porikli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Saeed Anwar.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Anwar, S., Hayder, Z. & Porikli, F. Deblur and deep depth from single defocus image. Machine Vision and Applications 32, 34 (2021). https://doi.org/10.1007/s00138-020-01162-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-020-01162-6

Keywords

Search

Navigation