Skip to main content

Advertisement

SpringerLink
Log in

Enforcing spatially coherent structures in shape from focus

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Most of the depth optimization schemes in shape from focus (SFF) enforce spatial coherence through convex energy functionals which oversmooth depth edges. Further, usually, no additional information about the scene is incorporated while estimating depth. In this work, we tackle the first issue by employing a nonconvex penalty that preserves depth edges effectively. For the second issue, we design a novel guidance map for SFF based on the cross-correlation between image sequence and focus volume (FV). This cross-correlation-based guidance enforces the coherent structures between the image sequence and FV. The proposed regularization framework fuses information from the guidance map, iteratively updated depth map, and the structural similarity between them. The nonconvex objective function has been solved through the majorize-minimization algorithm. An analysis has been presented that indicates the convergence behavior of the solver. Experiments have been conducted using a variety of synthetic and real image sequences. Qualitative and quantitative comparison with state-of-the-art methods indicates that the proposed method provides better depth maps.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data Availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Notes

  1. Assuming, as compared to the dark pixel, high value is assigned to the bright pixel.

References

  1. Ahmad MB, Choi TS (2005) A heuristic approach for finding best focused shape. IEEE Trans Circuits Syst Video Technol 15(4):566–574

    Article  Google Scholar 

  2. Ali U, Mahmood MT (2018) Depth from focus enhancement through joint filtering. In: International conference on consumer electronics-asia (ICCE-Asia). IEEE, pp 206–212

  3. Ali U, Mahmood MT (2019) 3D shape recovery by aggregating 3D wavelet transform-based image focus volumes through 3D weighted least squares. J Math Imaging Vis:1–19

  4. Ali U, Mahmood MT (2021) Enforcing spatially coherent structures for accurate 3d shape recovery. In: 2021 International conference on information and communication technology convergence (ICTC). IEEE, pp 1504–1506

  5. Ali U, Mahmood MT (2021) Robust focus volume regularization in shape from focus. IEEE Trans Image Process 30:7215–7227

    Article  Google Scholar 

  6. Alicona B (2022) Optical measurement solutions in use. https://www.alicona.com/applications/

  7. Asif M, Choi TS (2001) Shape from focus using multilayer feedforward neural networks. IEEE Trans Image Process 10(11):1670–1675

    Article  MATH  Google Scholar 

  8. Boshtayeva M, Hafner D, Weickert J (2015) A focus fusion framework with anisotropic depth map smoothing. Pattern Recognit 48(11):3310–3323

    Article  Google Scholar 

  9. Ceruso S, Bonaque-González S, Oliva-García R, Rodríguez-Ramos JM (2021) Relative multiscale deep depth from focus. Signal Process Image Commun:116417

  10. Choi TS, Yun J (2000) Three-dimensional shape recovery from the focused-image surface. Optical Eng 39(5):1321–1327

    Article  Google Scholar 

  11. Dansereau D (2022) Light field toolbox v0.4. https://www.mathworks.com/matlabcentral/fileexchange/49683-light-field-toolbox-v0-4/

  12. Farbman Z, Fattal R, Lischinski D, Szeliski R (2008) Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Trans Graph (TOG) 27(3):1–10

    Article  Google Scholar 

  13. Fujimura Y, Iiyama M, Funatomi T, Mukaigawa Y (2022) Deep Depth from focal stack with defocus model for camera-setting invariance. arXiv:220213055

  14. Gaganov V, Ignatenko A (2009) Robust shape from focus via Markov random fields. In: Proceedings of graphicon conference, pp 74–80

  15. Guo X, Li Y, Ma J, Ling H (2020) Mutually guided image filtering. IEEE Trans Pattern Anal Mach Intell 42(3):694–707. https://doi.org/10.1109/TPAMI.2018.2883553

    Article  Google Scholar 

  16. Ham B, Cho M, Ponce J (2018) Robust guided image filtering using nonconvex potentials. IEEE Trans Pattern Anal Mach Intell 40(1):192–207. https://doi.org/10.1109/TPAMI.2017.2669034

    Article  Google Scholar 

  17. Hazirbas C, Soyer SG, Staab MC, Leal-Taixé L, Cremers D (2018) Deep depth from focus. In: Asian conference on computer vision. Springer, pp 525–541

  18. He K, Sun J, Tang X (2012) Guided image filtering. IEEE Trans Pattern Anal Mach Intell 35(6):1397–1409

    Article  Google Scholar 

  19. Honauer K, Johannsen O, Kondermann D, Goldluecke B (2016) A dataset and evaluation methodology for depth estimation on 4D light fields. In: Asian conference on computer vision. Springer, pp 19–34

  20. Huang SJ, Ying FR (2012) Stereo vision system for moving object detecting and locating based on CMOS image sensor and DSP chip. Pattern Anal Appl 15(2):189–202

    Article  MathSciNet  Google Scholar 

  21. Javidnia H, Corcoran P (2018) Application of preconditioned alternating direction method of multipliers in depth from focal stack. J Electr Imaging 27 (2):023019

    Article  Google Scholar 

  22. Mahmood MT, Choi TS (2012) Nonlinear approach for enhancement of image focus volume in shape from focus. IEEE Trans Image Process 21(5):2866–2873

    Article  MathSciNet  MATH  Google Scholar 

  23. Mairal J (2015) Incremental majorization-minimization optimization with application to large-scale machine learning. SIAM J Optim 25(2):829–855

    Article  MathSciNet  MATH  Google Scholar 

  24. Maximov M, Galim K, Leal-Taixé L (2020) Focus on defocus: bridging the synthetic to real domain gap for depth estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1071–1080

  25. Minhas R, Mohammed AA, Wu QJ, Sid-Ahmed MA (2009) 3D shape from focus and depth map computation using steerable filters. In: International conference image analysis and recognition. Springer, pp 573–583

  26. Moeller M, Benning M, Schönlieb C, Cremers D (2015) Variational depth from focus reconstruction. IEEE Trans Image Process 24(12):5369–5378

    Article  MathSciNet  MATH  Google Scholar 

  27. Muhammad M, Choi TS (2012) Sampling for shape from focus in optical microscopy. IEEE Trans Pattern Anal Mach Intell 34(3):564–573

    Article  Google Scholar 

  28. Nayar S K, Nakagawa Y (1994) Shape from focus. IEEE Trans Pattern Anal Mach Intell 16(8):824–831

    Article  Google Scholar 

  29. Peng S, Yin B, Hao X, Yang Q, Kumar A, Wang L (2021) Depth and edge auxiliary learning for still image crowd density estimation. Pattern Anal Appl 24(4):1777–1792

    Article  Google Scholar 

  30. Pertuz S, Puig D, Garcia MA (2013) Analysis of focus measure operators for shape-from-focus. Pattern Recognit 46(5):1415–1432

    Article  MATH  Google Scholar 

  31. Petschnigg G, Szeliski R, Agrawala M, Cohen M, Hoppe H, Toyama K (2004) Digital photography with flash and no-flash image pairs. ACM Trans Graph 23(3):664–672

    Article  Google Scholar 

  32. Saini D, Kumar S, Gulati TR (2017) NURBS-based geometric inverse reconstruction of free-form shapes. J King Saud Univ-Comput Inf Sci 29 (1):116–133

    Google Scholar 

  33. Shim SO, Choi TS (2010) A novel iterative shape from focus algorithm based on combinatorial optimization. Pattern Recognit 43(10):3338–3347

    Article  MATH  Google Scholar 

  34. Sotoca JM, Latorre-Carmona P, Espinos-Morato H, Pla F, Javidi B (2019) Depth estimation improvement in 3D integral imaging using an edge removal approach. Pattern Anal Appl 22(1):33–45

    Article  MathSciNet  Google Scholar 

  35. Subbarao M, Choi T (1995) Accurate recovery of three-dimensional shape from image focus. IEEE Trans Pattern Anal Mach Intell 17(3):266–274

    Article  Google Scholar 

  36. Suwajanakorn S, Hernandez C, Seitz SM (2015) Depth from focus with your mobile phone. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3497–3506

  37. Tseng CY, Wang SJ (2014) Shape-from-focus depth reconstruction with a spatial consistency model. IEEE Trans Circuits Syst Video Technol 24 (12):2063–2076

    Article  Google Scholar 

  38. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  39. Wang NH, Wang R, Liu YL, Huang YH, Chang YL, Chen CP et al (2021) Bridging unsupervised and supervised depth from focus via all-in-focus supervision. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 12621–12631

  40. Wu CJ (1983) On the convergence properties of the EM algorithm. Annals Stat:95–103

  41. Yang F, Huang X, Zhou Z (2022) Deep depth from focus with differential focus volume. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12642–12651

  42. Zhang Z, Kwok JT, Yeung DY (2004) Surrogate maximization/minimization algorithms for adaboost and the logistic regression model. In: Proceedings of the twenty-first international conference on Machine learning, p 117

  43. Zhang Q, Shen X, Xu L, Jia J (2014) Rolling guidance filter. In: ECCV. Springer, pp 815–830

Download references

Funding

This work was supported in parts by Creative Challenge Research Program (2021R1I1A1A01052521) funded by the Ministry of Education, by the Basic Research Program (2022R1F1A1071452) and Basic Science Research Program (2022R1A4A3033571) funded by the Korea government (MSIT: Ministry of Science and ICT) through the National Research Foundation (NRF) of Korea.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Sungkyunkwan University, 2066 Seobu-ro, Jangan-gu, 16419, Suwon, Republic of Korea

    Usman Ali

  2. Future Convergence Engineering, Korea University of Technology and Education, 1600 Chungjeolno, Byeogchunmyun, Cheonan, 31253, Chungcheongnam-do, Republic of Korea

    Muhammad Tariq Mahmood

Authors
  1. Usman Ali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Muhammad Tariq Mahmood
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Tariq Mahmood.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ali, U., Mahmood, M.T. Enforcing spatially coherent structures in shape from focus. Multimed Tools Appl 82, 36431–36447 (2023). https://doi.org/10.1007/s11042-023-14984-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14984-z

Keywords

Search

Navigation