Skip to main content
Log in

An automatic framework for endoscopic image restoration and enhancement

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Despite its success in the field of minimally invasive surgery, endoscopy image analysis remains challenging due to limited image settings and control conditions. The low resolution and existence of large number of reflections in endoscopy images are the major problems in the automatic detection of objects. To address these issues, we presented a novel framework based on the convolutional neural networks. The proposed approach consists of three major parts. First, a deep learning (DL)-based image evaluation method is used to classify the input images into two groups, namely, specular highlights and weakly illuminated groups. Second, the specular highlight is detected using the DL-based method, and the reflected areas are recovered through a patch-based restoration operation. Lastly, gamma correction with optimized reflectance and illumination estimation is adopted to enhance the weakly illuminated images. The proposed method is compared against the existing ones, and the experimental results demonstrate that the former outperforms the latter in terms of subjective and objective assessments. This finding indicates that the proposed approach can serve as a potential tool for improving the quality of the endoscopy images used to examine internal body organs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Sdiri B, Cheikh FA, Dragusha K, Beghdadi A (2015) Comparative study of endoscopic image enhancement techniques. In: 2015 Colour and Visual Computing Symposium (CVCS), pp 1–5

  2. Domingues I, Sampaio IL, Duarte H, Santos JAM, Abreu PH (2019) Computer vision in esophageal cancer: a literature review. IEEE Access 7:103080–103094

  3. Fu G, Zhang Q, Song C, Lin Q, Xiao C (2019) Specular Highlight Removal for Real-world Images. Comput Graph Forum 38(7):253–263

  4. Son M, Lee Y, Chang HS (2020) Toward specular removal from natural images based on statistical reflection models. IEEE Trans Image Process 29:4204–4218

    Article  MathSciNet  Google Scholar 

  5. Alsaleh SM, Aviles AI, Sobrevilla P, Casals A, Hahn JK (2016) Adaptive Segmentation and Mask-Specific Sobolev Inpainting of Specular Highlights for Endoscopic Images, 38th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., pp 1196–1199

  6. Guo J, Shen DF, Lin GS, Huang JC, Liu KC, Lie WN (2016) A specular reflection suppression method for endoscopic images, Proc. - 2016 IEEE 2nd Int. Conf. Multimed. Big Data, BigMM 2016, pp 125–128

  7. Hsia C, Chiang J, Li H, Lin C, Chou K (2016) A 3D endoscopic imaging system with Content-Adaptive filtering and hierarchical similarity analysis. IEEE Sens J 16(11):4521–4530

    Article  Google Scholar 

  8. Saint-Pierre CA, Boisvert J, Grimard G, Cheriet F (2011) Detection and correction of specular reflections for automatic surgical tool segmentation in thoracoscopic images. Mach Vis Appl 22(1):171–180

    Article  Google Scholar 

  9. Meslouhi O, Kardouchi M, Allali H, Gadi T, Benkaddour Y (2011) Automatic detection and inpainting of specular reflections for colposcopic images. Open Comput Sci 1(3):341–354

    Article  Google Scholar 

  10. Zimmerman-Moreno G, Greenspan H (2006) Automatic Detection of Specular Reflections in Uterine Cervix Images. SPIE Med. imaging, pp 61446E—-61446E—-9

  11. Imtiaz MS, Wahid K (2014) Image enhancement and space-variant color reproduction method for endoscopic images using adaptive sigmoid function. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp 3905–3908

  12. Imtiaz MS, Wahid K (2014) A color reproduction method with image enhancement for endoscopic images. In: 2nd Middle East Conference on Biomedical Engineering, pp 135–138

  13. Li B, Meng MQ-H (2012) Wireless capsule endoscopy images enhancement via adaptive contrast diffusion. J Vis Commun Image Represent 23(1):222–228

    Article  MathSciNet  Google Scholar 

  14. Huang S, Lee F, Miao R, Si Q, Lu C, Chen Q (2020) A deep convolutional neural network architecture for interstitial lung disease pattern classification. Med Biol Eng Comput:1–13

  15. Hesamian MH, Jia W, He X, Kennedy P (2019) Deep learning techniques for medical image segmentation: Achievements and challenges. J Digit Imaging 32(4):582–596

    Article  Google Scholar 

  16. Zhao Z-Q, Zheng P, Xu S, Wu X (2019) Object detection with deep learning: a review. IEEE Trans Neural Netw Learn Syst 30(11):3212–3232

    Article  Google Scholar 

  17. Nassif AB, Shahin I, Attili I, Azzeh M, Shaalan K (2019) Speech recognition using deep neural networks: a systematic review. IEEE Access 7:19143–19165

    Article  Google Scholar 

  18. Guo G, Zhang N (2019) A survey on deep learning based face recognition. Comput. Vis. Image Underst. 189:102805

  19. Lin H, Yang P, Zhang F (2019) Review of scene text detection and recognition. Arch Comput Methods Eng:1–22

  20. Lavecchia A (2019) Deep learning in drug discovery: opportunities, challenges and future prospects. Drug Discov Today 24(10):2017–2032

    Article  Google Scholar 

  21. Adeel A, Gogate M, Hussain A, Whitmer WM (2019) Lip-reading driven deep learning approach for speech enhancement. IEEE Trans Emerg Top Comput Intell

  22. Shrestha A, Mahmood A (2019) Review of deep learning algorithms and architectures. IEEE Access 7:53040–53065

    Article  Google Scholar 

  23. Bernal J, et al. (2017) Comparative validation of polyp detection methods in video colonoscopy: Results from the MICCAI 2015 endoscopic vision challenge. IEEE Trans Med Imaging 36(6):1231–1249

    Article  Google Scholar 

  24. Pogorelov PT et al (2017) KVASIR. In: Proceedings of the 8th ACM on Multimedia Systems Conference, pp 164–169

  25. Bernal F, Sanchez J, Vilarino J (2012) Towards automatic polyp detection with a polyp appearance model. Pattern Recognit 45(9):3166–3182

    Article  Google Scholar 

  26. Silva B, Histace J, Romain A, Dray O, Granado X (2014) Towards embedded detection of polyps in WCE images for early diagnosis of colorectal cancer To cite this version. Int J Comput Assist Radiol Surg 9(2):283–293

    Article  Google Scholar 

  27. North American Society for Pediatric Gastroenterology, Hepatology and Nutrition. [Online]. Available: https://www.naspghan.org/content/97/en/professional-education/resources/endoscopy-photo-gallery

  28. Goodfellow Y, Pouget-Abadie I, Mirza J, Xu M, Warde-Farley B, Ozair D, Courville S, Bengio A (2014) Benerative adversarial networks. Adv Neural Inf Process Syst.:2672–2680

  29. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation, in International Conference on Medical image computing and computer-assisted intervention, pp 234–241

  30. Darabi S, Shechtman E, Barnes C, Goldman DB, Sen P (2012) Image melding: combining inconsistent images using patch-based synthesis. ACM Trans Graph 31(4):1–10

    Article  Google Scholar 

  31. Wexler Y, Shechtman E, Irani M (2004) Space-Time Video Completion, in Computer Vision and Pattern Recognition (CVPR), pp 120–127

  32. Fu X, Zeng D, Huang Y, Zhang X-P, Ding X (2016) A weighted variational model for simultaneous reflectance and illumination estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2782–2790

  33. Goldstein T, Osher S (2009) The split Bregman method for L1-regularized problems. SIAM J Imaging Sci 2(2):323–343

    Article  MathSciNet  Google Scholar 

  34. Mittal A, Soundararajan R, Bovik AC (2012) Making a ‘completely blind’ image quality analyzer. IEEE Signal Process Lett 20(3):209–212

    Article  Google Scholar 

  35. Kimmel R, Elad M, Shaked D, Keshet R, Sobel I (2003) A variational framework for retinex. Int J Comput Vis 52(1):7–23

    Article  Google Scholar 

  36. Selka F, Nicolau SA, Agnus V, Bessaid A, Marescaux J, Soler L (2013) Evaluation of Endoscopic Image Enhancement for Feature Tracking: A New Validation Framework. In: Augmented Reality Environments for Medical Imaging and Computer-Assisted Interventions, pp 75–85

  37. Abdullah-Al-Wadud M, Kabir MH, Dewan MAA, Chae O (2007) A dynamic histogram equalization for image contrast enhancement. IEEE Trans Consum Electron 53(2):593– 600

    Article  Google Scholar 

  38. Perona P, Malik J (1990) Scale-space and edge detection using anisotropic diffusion. IEEE Trans Pattern Anal Mach Intell 12(7):629–639

    Article  Google Scholar 

  39. Li L, Zouthe YX, Li Y (2013) Wireless capsule endoscopy images enhancement based on adaptive anisotropic diffusion. In: 2013 IEEE China Summit and International Conference on Signal and Information Processing, pp 273–277

  40. Erickson BJ, Korfiatis P, Kline TL, Akkus Z, Philbrick K, Weston AD (2018) Deep learning in radiology: does one size fit all?. J Am Coll Radiol 15(3):521–526

    Article  Google Scholar 

  41. Sahiner B, et al. (2019) Deep learning in medical imaging and radiation therapy. Med Phys 46 (1):e1–e36

    Article  MathSciNet  Google Scholar 

  42. Wong KCL, Syeda-Mahmood T, Moradi M (2018) Building medical image classifiers with very limited data using segmentation networks. Med Image Anal 49:105–116

    Article  Google Scholar 

  43. Li C, Guo J, Porikli F, Pang Y (2018) Lightennet: A convolutional neural network for weakly illuminated image enhancement. Pattern Recognit Lett 104:15–22

    Article  Google Scholar 

  44. Wang W, Wei C, Yang W, Liu J (2018) GLADNet: Low-light enhancement network with global awareness, in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp 751–755

  45. Lv F, Lu F, Wu J, Lim C (2018) MBLLEN: Low-Light Image/Video Enhancement Using CNNs. In: BMVC, pp 220

  46. Litjens G, et al (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88

    Article  Google Scholar 

  47. Greenspan H, Van Ginneken B, Summers RM (2016) Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique. IEEE Trans Med Imaging 35(5):1153–1159

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Hong Song or Jian Yang.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Asif, M., Chen, L., Song, H. et al. An automatic framework for endoscopic image restoration and enhancement. Appl Intell 51, 1959–1971 (2021). https://doi.org/10.1007/s10489-020-01923-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01923-w

Keywords

Navigation