Skip to main content
SpringerLink
Log in

A higher performance shape from focus strategy based on unsupervised deep learning for 3D shape reconstruction

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Shape From Focus (SFF) is one of the most popular strategies for reconstructing object’s 3D shape, which doesn’t require any additional technology. SFF strategies generate the object’s 3D shape using a sequence of 2D images with the same field of perspective and various areas in-focus. Although SFF has many literature studies and is one of the most preferred strategies in computer vision to produce 3D shapes of objects, this research field still contains critical shortcomings such as high computational costs and complexity, less effectiveness in noisy or poorly textured areas, insufficient focus information to be extracted from the input images, generally adapting a pre- or post-processing technique and using only gray levels of original images to acquire the pixel’s focus values. In order to minimize these shortcomings, an unsupervised deep learning-based SFF strategy that gives higher performance than current strategies is suggested in this study. When compared with the studies generated for 3D shape reconstruction in the literature, the proposed SFF strategy provides various fundamental contributions such as the first study proposing SFF strategy based on unsupervised deep learning, providing a high-quality focus measurement operator, acquiring the pixel’s focus values from deep features and not requiring any pre- or post-processing technique. In order to assess the efficiency of the proposed SFF strategy, well-known focus measurement operators are analyzed using 4 different synthetic and microscope image sequences. In order to determine which SFF strategy can extract more essential details from the 2D images with various areas in-focus on these sequences, quality assessment criteria with and without requiring a ground-truth are preferred, which are Root Mean Square Error (RMSE), Peak Signal Noise Ratio (PSNR), Universal Quality Index (UQI), Correlation Coefficient (CC), Standard Deviation (SD) and Kurtosis Metric (KM). Both qualitative and quantitative evaluations reveal that the suggested SFF strategy with highest PSNR (28.1725, 25.0854), UQI (0.9454, 0.7543), CC (0.9457, 0.7132), and lowest RMSE (0.0390, 0.0563), SD (3.1666, 1.1768), KM (2.1254, 3.6362) values produces higher performance, and the designed unsupervised deep learning model is more efficient to transmit crucial details from 2D images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

The data sets analysed in this study are available in EPFL Biomedical Imaging Group repository (http://bigwww.epfl.ch/demo/edf/index.html

References

  1. Billiot B, Cointault F, Journaux L, Simon JC, Gouton P (2013) 3d image acquisition system based on shape from focus technique. In: Sensors, pp 5040–5053

  2. Shim SO, Malik AS, Choi TS (2009) Accurate shape from focus based on focus adjustment in optical microscopy. Microsc Res Tech 72(5):362–370

    Google Scholar 

  3. Pertuz S, Puig D, Garcia MA (2013) Analysis of focus measure operators for shape-from-focus. Pattern Recogn 46(5):1415–1432

    Google Scholar 

  4. Geusebroek JM, Cornelissen F, Smeulders AW, Geerts H (2000) Robust autofocusing in microscopy. Cytometry: J Int Soc Anal Cytol 39(1):1–9

  5. Malik AS, Choi TS (2008) A novel algorithm for estimation of depth map using image focus for 3d shape recovery in the presence of noise. Pattern Recogn 41(7):2200–2225

    Google Scholar 

  6. Ahmad MB, Choi TS (2007) Application of three dimensional shape from image focus in lcd/tft displays manufacturing. IEEE Trans Consum Electron 53(1):1–4

    Google Scholar 

  7. Pech Pacheco JL, Cristóbal G, Chamorro Martinez J, Fernández Valdivia J (2000) Diatom autofocusing in brightfield microscopy: a comparative study. In: Pattern recognition, 2000. Proceedings. 15th international conference on, vol 3, pp 314–317. IEEE

  8. Thelen A, Frey S, Hirsch S, Hering P (2009) Improvements in shape-fromfocus for holographic reconstructions with regard to focus operators, neighborhood-size, and height value interpolation. IEEE Trans Image Process 18(1):151–157

    MathSciNet  Google Scholar 

  9. Nayar SK, Nakagawa Y (1990) Shape from focus: an effective approach for rough surfaces. In: Robotics and automation, 1990. Proceedings., 1990 IEEE international conference on, pp 218-225. IEEE

  10. Nayar SK (1992) Shape from focus system. In: Proceedings 1992 IEEE computer society conference on computer vision and pattern recognition, pp 302–308. IEEE

  11. An Y, Kang G, Kim IJ, Chung HS, Park J (2008) Shape from focus through laplacian using 3d window. In: Future generation communication and networking, 2008. FGCN’08. Second international conference on, vol 2, pp 46–50. IEEE

  12. Yan T, Hu Z, Qian Y, Qiao Z, Zhang L (2020) 3d shape reconstruction from multifocus image fusion using a multidirectional modified laplacian operator. Pattern Recogn 98:107065

    Google Scholar 

  13. Xie H, Rong W, Sun L (2006) Wavelet-based focus measure and 3-d surface reconstruction method for microscopy images. In: Intelligent robots and systems, 2006 IEEE/RSJ international conference on, pp 229–234. IEEE

  14. Xie H, Rong W, Sun L (2007) Construction and evaluation of a waveletbased focus measure for microscopy imaging. Microscopy research and technique 70(11):987–995

    Google Scholar 

  15. Ali U, Mahmood MT (2019) 3d shape recovery by aggregating 3d wavelet transform-based image focus volumes through 3d weighted least squares. J Math Imaging Vis 1–19

  16. Yap PT, Raveendran P (2004) Image focus measure based on chebyshev moments. IEE Proc-Vis Image Sig Process 151(2):128–136

    Google Scholar 

  17. Wee CY, Paramesran R (2007) Measure of image sharpness using eigenvalues. Inf Sci 177(12):2533–2552

    Google Scholar 

  18. Lee SY, Yoo JT, Kumar Y, Kim SW (2009) Reduced energy-ratio measure for robust autofocusing in digital camera. IEEE Sig Process Lett 16(2):133–136

    Google Scholar 

  19. Shen CH, Chen HH (2006) Robust focus measure for low-contrast images. In: Consumer electronics, 2006. ICCE’06. 2006 digest of technical papers. International conference on, pp 69–70. IEEE

  20. Lee SY, Kumar Y, Cho JM, Lee SW, Kim SW (2008) Enhanced autofocus algorithm using robust focus measure and fuzzy reasoning. IEEE Trans Circ Syst Video Technol 18(9):1237–1246

    Google Scholar 

  21. Helmli FS, Scherer S (2001) Adaptive shape from focus with an error estimation in light microscopy. In: Image and signal processing and analysis, 2001. ISPA 2001. Proceedings of the 2nd international symposium on, pp 188–193. IEEE

  22. Lorenzo J, Castrillon M, Méndez J, Deniz O (2008) Exploring the use of local binary patterns as focus measure. In: Computational intelligence for modelling control & automation, 2008 international conference on, pp 855–860. IEEE

  23. Nanda H, Cutler R (2001) Practical calibrations for a real-time digital omnidirectional camera. CVPR Technical Sketch 20:2

    Google Scholar 

  24. Minhas R, Mohammed AA, Wu QJ (2011) Shape from focus using fast discrete curvelet transform. Pattern Recogn 44(4):839–853

    Google Scholar 

  25. Shirvaikar MV (2004) An optimal measure for camera focus and exposure. In: System Theory, 2004. Proceedings of the thirty-sixth southeastern symposium on, pp 472–475. IEEE

  26. Minhas R, Mohammed AA, Wu QJ, Sid Ahmed MA (2009) 3d shape from focus and depth map computation using steerable filters. In: International conference image analysis and recognition. Springer, pp 573–583.

  27. Fan T, Yu H (2018) A novel shape from focus method based on 3d steerable filters for improved performance on treating textureless region. Opt Commun 410:254–261

    Google Scholar 

  28. Mahmood F, Mahmood J, Zeb A, Iqbal J (2018) 3d shape recovery from image focus using gabor features. In: Tenth international conference on machine vision (ICMV 2017). vol 10696, pp 106961. International Society for Optics and Photonics

  29. Doğan H, Doğan RÖ (2023) A new high quality focus measurement operator based on nonsubsampled shearlet transform for 3d shape reconstruction. Mühendislik Bilimleri ve Araştırmaları Dergisi 5(1):9–19

  30. Pertuz S, Puig D, Garcia MA (2013) Reliability measure for shape-from-focus. Image Vis Comput 31(10):725–734

    Google Scholar 

  31. Lee I, Mahmood MT, Choi TS (2013) Adaptive window selection for 3d shape recovery from image focus. Optics Laser Technol 45:21–31

    Google Scholar 

  32. Muhammad MS, Mutahira H, Choi KW, Kim WY, Ayaz Y (2014) Calculating accurate window size for shape-from-focus. In: Information science and applications (ICISA):2014 international conference on, pp 1–4. IEEE

  33. Tseng CY, Wang SJ et al (2014) Shape-from-focus depth reconstruction with a spatial consistency model. IEEE Trans Circuits Syst Video Techn 24(12):2063–2076

    Google Scholar 

  34. Liu W, Key XW (2015) Semi-global depth from focus. In: 2015 3rd IAPR asian conference on pattern recognition (ACPR), pp 624–629

  35. Tsai D, Chen HH (2016) Focus profile modeling. IEEE Trans Image Process 25(2):818–828

    MathSciNet  Google Scholar 

  36. Surh J, Jeon H-G, Park Y, Im S, Ha H, So Kweon I (2017) Noise robust depth from focus using a ring difference filter. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6328–6337

  37. Kumar PG, Ranjan Sahay R (2017) Accurate structure recovery via weighted nuclear norm: a low rank approach to shape-from-focus. In: Proceedings of the IEEE international conference on computer vision, pp 563–574

  38. Jang HS, Muhammad MS, Choi TS (2018) Optimal depth estimation using modified kalman filter in the presence of non-gaussian jitter noise. Microscopy Research and Technique, pp 1–8

  39. Chen M, Zhong Y, Li Z, Wu J (2018) A novel 3d shape reconstruction method based on maximum correntropy kalman filtering. Sensor Review 0

  40. Jang H-S, Muhammad MS, Choi T-S (2019) Bayes filter based jitter noise removal in shape recovery from image focus. J Imaging Sci Technol 63(2):20501–1

    Google Scholar 

  41. Jang H-S, Muhammad MS, Yun G, Kim DH (2019) Sampling based on kalman filter for shape from focus in the presence of noise. Appl Sci 9(16):3276

    Google Scholar 

  42. Lee S-A, Jang H-S, Lee B-G (2019) Jitter elimination in shape recovery by using adaptive neural network filter. Sensors 19(11):2566

    Google Scholar 

  43. Ali U, Pruks V, Mahmood MT (2019) Image focus volume regularization for shape from focus through 3d weighted least squares. Inf Sci 489:155–166

    MathSciNet  Google Scholar 

  44. Jang H-S, Muhammad MS, Choi T-S (2019) Optimizing image focus for shape from focus through locally weighted non-parametric regression. IEEE Access 7:74393–74400

    Google Scholar 

  45. Tang J, Qiu Z, Li T (2019) A novel measurement method and application for grinding wheel surface topography based on shape from focus. Measurement 133:495–507

    Google Scholar 

  46. Ma Z, Kim D, Shin Y-G (2020) Shape-from-focus reconstruction using nonlocal matting laplacian prior followed by mrf-based refinement. Pattern Recognit 103:103272

  47. Onogi S, Kawase T, Sugino T, Nakajima Y (2021) Investigation of shape-from-focus precision by texture frequency analysis. Electron 10(16):1870

  48. Jang H-S, Yun G, Mutahira H, Muhammad MS (2021) A new focus measure operator for enhancing image focus in 3d shape recovery. Microsc Res Tech 84(10):2483–2493

    Google Scholar 

  49. Mutahira H, Shin V, Muhammad MS, Shin DR (2021) Sampling-noise modeling & removal in shape from focus systems through kalman filter. IEEE Access 9:102520–102541

    Google Scholar 

  50. Shim S-O (2022) Multidirectional focus measure for accurate three-dimensional shape recovery of microscopic objects. Microsc Res Tech 85(3):940–947

    Google Scholar 

  51. Mutahira H, Shin V, Park U, Muhammad MS (2022) Jitter noise modeling and its removal using recursive least squares in shape from focus systems. Scientific Reports 12(1):1–20

    Google Scholar 

  52. Hou L, Zou J, Zhang W, Chen Y, Shao W, Li Y, Chen S (2022) An improved shape from focus method for measurement of three-dimensional features of fuel nozzles. Sensors 23(1):265

    Google Scholar 

  53. Ali U, Lee IH, Mahmood MT (2023) Incorporating structural prior for depth regularization in shape from focus. Comput Vis Image Underst 227:103619

    Google Scholar 

  54. Fu B, He R, Yuan Y, Jia W, Yang S, Liu F (2023) Shape from focus using gradient of focus measure curve. Optics Lasers Eng 160:107320

    Google Scholar 

  55. Muhammad M, Choi TS (2012) Sampling for shape from focus in optical microscopy. IEEE Trans Pattern Anal Mach Intell 34(3):564–573

    Google Scholar 

  56. Tan W, Tiwari P, Pandey HM, Moreira C, Jaiswal AK (2020) Multimodal medical image fusion algorithm in the era of big data. Neural Comput Appl 1–21

  57. Hermessi H, Mourali O, Zagrouba E (2021) Multimodal medical image fusion review: theoretical background and recent advances. Signal Process 183:108036

  58. Liu Y, Wang L, Cheng J, Li C, Chen X (2020) Multi-focus image fusion: a survey of the state of the art. Inform Fusion 64:71–91

    Google Scholar 

  59. Wang K, Zheng M, Wei H, Qi G, Li Y (2020) Multi-modality medical image fusion using convolutional neural network and contrast pyramid. Sensors 20(8):2169

    Google Scholar 

  60. Jose J, Gautam N, Tiwari M, Tiwari T, Suresh A, Sundararaj V, Rejeesh M (2021) An image quality enhancement scheme employing adolescent identity search algorithm in the nsst domain for multimodal medical image fusion. Biomed Signal Process Control 66:102480

  61. Liu S, Wang M, Yin L, Sun X, Zhang Y-D, Zhao J (2022) Two-scale multimodal medical image fusion based on structure preservation. Front Comput Neurosci 15:133

  62. Ye F, Li X, Zhang X (2019) Fusioncnn: a remote sensing image fusion algorithm based on deep convolutional neural networks. Multimed Tools Appl 78:14683–14703

    Google Scholar 

  63. Huang M, Liu S, Li Z, Feng S, Wu D, Wu Y, Shu F (2022) Remote sensing image fusion algorithm based on two-stream fusion network and residual channel attention mechanism. Wireless Commun Mobile Comput 2022:1–14

    Google Scholar 

  64. Wang W, Han C, Zhou T, Liu D (2022) Visual recognition with deep nearest centroids. arXiv preprint arXiv:2209.07383

  65. Salahuddin Z, Woodruff HC, Chatterjee A, Lambin P (2022) Transparency of deep neural networks for medical image analysis: a review of interpretability methods. Comput Biol Med 140:105111

  66. Fan F-L, Xiong J, Li M, Wang G (2021) On interpretability of artificial neural networks: a survey. IEEE Trans Radiation Plasma Med Sci 5(6):741–760

    Google Scholar 

  67. Kishor A, Chakraborty C, Jeberson W (2021) Reinforcement learning for medical information processing over heterogeneous networks. Multimed Tools Appl 80(16):23983–24004

    Google Scholar 

  68. Negi A, Kumar K, Chaudhari NS, Singh N, Chauhan P (2021) Predictive analytics for recognizing human activities using residual network and fine-tuning. In: Big data analytics: 9th international conference, BDA 2021, virtual event, December 15–18, 2021, Proceedings 9. Springer, pp 296–310

  69. Kishor A, Chakarbarty C (2021) Task offloading in fog computing for using smart ant colony optimization. Wireless Personal Commun 1–22

  70. Negi A, Kumar K (2021) Classification and detection of citrus diseases using deep learning. In: Data science and its applications. Chapman and Hall/CRC, pp 63–85

  71. Negi A, Kumar K (2021) Face mask detection in real-time video stream using deep learning. Comput Intell Healthcare Inform 255–268

  72. Chakraborty C, Kishor A, Rodrigues JJ (2022) Novel enhanced-grey wolf optimization hybrid machine learning technique for biomedical data computation. Comput Electrical Eng 99:107778

  73. Kumar A, Purohit K, Kumar K (2021) Stock price prediction using recurrent neural network and long short-term memory. In: Conference proceedings of ICDLAIR2019, pp 153–160. Springer

  74. Negi A, Kumar K, Chauhan P (2021) Deep neural network-based multi-class image classification for plant diseases. Agricultural informatics: automation using the IoT and machine learning, pp 117–129

  75. Alok N, Krishan K, Chauhan P (2021) Deep learning-based image classifier for malaria cell detection. Mach Learn Healthcare Appl 187–197

  76. Sharma S, Kumar K, Singh N (2017) D-fes: deep facial expression recognition system. In: 2017 Conference on information and communication technology (CICT). IEEE, pp 1–6

  77. Vijayvergia A, Kumar K (2018) Star: rating of reviews by exploiting variation in emotions using transfer learning framework. In: 2018 Conference on information and communication technology (CICT). IEEE, pp 1–6

  78. Kumar K, Shrimankar DD (2017) F-des: fast and deep event summarization. IEEE Trans Multimed 20(2):323–334

    Google Scholar 

  79. Kumar K, Shrimankar DD (2018) Deep event learning boost-up approach: delta. Multimed Tools Appl 77:26635–26655

    Google Scholar 

  80. Aguet F, Van De Ville D, Unser M (2008) Model-based 2.5-d deconvolution for extended depth of field in brightfield microscopy. IEEE Trans Image Process 17(7):1144–1153

  81. Li S, Yang Z, Li H (2017) Statistical evaluation of no-reference image quality assessment metrics for remote sensing images. ISPRS Inter J Geo-Inform 6(5):133

    Google Scholar 

  82. Lin T-Y, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: common objects in context. In: European conference on computer vision. Springer, pp 740–755

  83. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S et al (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

  84. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 10012–10022

Download references

Author information

Authors and Affiliations

  1. Department of Software Engineering, Karadeniz Technical University, Trabzon, 61080, Türkiye

    Hulya Dogan

Authors
  1. Hulya Dogan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hulya Dogan.

Ethics declarations

Conflicts of interest

The author declares that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dogan, H. A higher performance shape from focus strategy based on unsupervised deep learning for 3D shape reconstruction. Multimed Tools Appl 83, 35825–35848 (2024). https://doi.org/10.1007/s11042-023-16721-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-16721-y

Keywords

Search

Navigation