Abstract
Despite its success in the field of minimally invasive surgery, endoscopy image analysis remains challenging due to limited image settings and control conditions. The low resolution and existence of large number of reflections in endoscopy images are the major problems in the automatic detection of objects. To address these issues, we presented a novel framework based on the convolutional neural networks. The proposed approach consists of three major parts. First, a deep learning (DL)-based image evaluation method is used to classify the input images into two groups, namely, specular highlights and weakly illuminated groups. Second, the specular highlight is detected using the DL-based method, and the reflected areas are recovered through a patch-based restoration operation. Lastly, gamma correction with optimized reflectance and illumination estimation is adopted to enhance the weakly illuminated images. The proposed method is compared against the existing ones, and the experimental results demonstrate that the former outperforms the latter in terms of subjective and objective assessments. This finding indicates that the proposed approach can serve as a potential tool for improving the quality of the endoscopy images used to examine internal body organs.
Similar content being viewed by others
References
Sdiri B, Cheikh FA, Dragusha K, Beghdadi A (2015) Comparative study of endoscopic image enhancement techniques. In: 2015 Colour and Visual Computing Symposium (CVCS), pp 1–5
Domingues I, Sampaio IL, Duarte H, Santos JAM, Abreu PH (2019) Computer vision in esophageal cancer: a literature review. IEEE Access 7:103080–103094
Fu G, Zhang Q, Song C, Lin Q, Xiao C (2019) Specular Highlight Removal for Real-world Images. Comput Graph Forum 38(7):253–263
Son M, Lee Y, Chang HS (2020) Toward specular removal from natural images based on statistical reflection models. IEEE Trans Image Process 29:4204–4218
Alsaleh SM, Aviles AI, Sobrevilla P, Casals A, Hahn JK (2016) Adaptive Segmentation and Mask-Specific Sobolev Inpainting of Specular Highlights for Endoscopic Images, 38th Annu. Int. Conf. IEEE Eng. Med. Biol. Soc., pp 1196–1199
Guo J, Shen DF, Lin GS, Huang JC, Liu KC, Lie WN (2016) A specular reflection suppression method for endoscopic images, Proc. - 2016 IEEE 2nd Int. Conf. Multimed. Big Data, BigMM 2016, pp 125–128
Hsia C, Chiang J, Li H, Lin C, Chou K (2016) A 3D endoscopic imaging system with Content-Adaptive filtering and hierarchical similarity analysis. IEEE Sens J 16(11):4521–4530
Saint-Pierre CA, Boisvert J, Grimard G, Cheriet F (2011) Detection and correction of specular reflections for automatic surgical tool segmentation in thoracoscopic images. Mach Vis Appl 22(1):171–180
Meslouhi O, Kardouchi M, Allali H, Gadi T, Benkaddour Y (2011) Automatic detection and inpainting of specular reflections for colposcopic images. Open Comput Sci 1(3):341–354
Zimmerman-Moreno G, Greenspan H (2006) Automatic Detection of Specular Reflections in Uterine Cervix Images. SPIE Med. imaging, pp 61446E—-61446E—-9
Imtiaz MS, Wahid K (2014) Image enhancement and space-variant color reproduction method for endoscopic images using adaptive sigmoid function. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp 3905–3908
Imtiaz MS, Wahid K (2014) A color reproduction method with image enhancement for endoscopic images. In: 2nd Middle East Conference on Biomedical Engineering, pp 135–138
Li B, Meng MQ-H (2012) Wireless capsule endoscopy images enhancement via adaptive contrast diffusion. J Vis Commun Image Represent 23(1):222–228
Huang S, Lee F, Miao R, Si Q, Lu C, Chen Q (2020) A deep convolutional neural network architecture for interstitial lung disease pattern classification. Med Biol Eng Comput:1–13
Hesamian MH, Jia W, He X, Kennedy P (2019) Deep learning techniques for medical image segmentation: Achievements and challenges. J Digit Imaging 32(4):582–596
Zhao Z-Q, Zheng P, Xu S, Wu X (2019) Object detection with deep learning: a review. IEEE Trans Neural Netw Learn Syst 30(11):3212–3232
Nassif AB, Shahin I, Attili I, Azzeh M, Shaalan K (2019) Speech recognition using deep neural networks: a systematic review. IEEE Access 7:19143–19165
Guo G, Zhang N (2019) A survey on deep learning based face recognition. Comput. Vis. Image Underst. 189:102805
Lin H, Yang P, Zhang F (2019) Review of scene text detection and recognition. Arch Comput Methods Eng:1–22
Lavecchia A (2019) Deep learning in drug discovery: opportunities, challenges and future prospects. Drug Discov Today 24(10):2017–2032
Adeel A, Gogate M, Hussain A, Whitmer WM (2019) Lip-reading driven deep learning approach for speech enhancement. IEEE Trans Emerg Top Comput Intell
Shrestha A, Mahmood A (2019) Review of deep learning algorithms and architectures. IEEE Access 7:53040–53065
Bernal J, et al. (2017) Comparative validation of polyp detection methods in video colonoscopy: Results from the MICCAI 2015 endoscopic vision challenge. IEEE Trans Med Imaging 36(6):1231–1249
Pogorelov PT et al (2017) KVASIR. In: Proceedings of the 8th ACM on Multimedia Systems Conference, pp 164–169
Bernal F, Sanchez J, Vilarino J (2012) Towards automatic polyp detection with a polyp appearance model. Pattern Recognit 45(9):3166–3182
Silva B, Histace J, Romain A, Dray O, Granado X (2014) Towards embedded detection of polyps in WCE images for early diagnosis of colorectal cancer To cite this version. Int J Comput Assist Radiol Surg 9(2):283–293
North American Society for Pediatric Gastroenterology, Hepatology and Nutrition. [Online]. Available: https://www.naspghan.org/content/97/en/professional-education/resources/endoscopy-photo-gallery
Goodfellow Y, Pouget-Abadie I, Mirza J, Xu M, Warde-Farley B, Ozair D, Courville S, Bengio A (2014) Benerative adversarial networks. Adv Neural Inf Process Syst.:2672–2680
Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation, in International Conference on Medical image computing and computer-assisted intervention, pp 234–241
Darabi S, Shechtman E, Barnes C, Goldman DB, Sen P (2012) Image melding: combining inconsistent images using patch-based synthesis. ACM Trans Graph 31(4):1–10
Wexler Y, Shechtman E, Irani M (2004) Space-Time Video Completion, in Computer Vision and Pattern Recognition (CVPR), pp 120–127
Fu X, Zeng D, Huang Y, Zhang X-P, Ding X (2016) A weighted variational model for simultaneous reflectance and illumination estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2782–2790
Goldstein T, Osher S (2009) The split Bregman method for L1-regularized problems. SIAM J Imaging Sci 2(2):323–343
Mittal A, Soundararajan R, Bovik AC (2012) Making a ‘completely blind’ image quality analyzer. IEEE Signal Process Lett 20(3):209–212
Kimmel R, Elad M, Shaked D, Keshet R, Sobel I (2003) A variational framework for retinex. Int J Comput Vis 52(1):7–23
Selka F, Nicolau SA, Agnus V, Bessaid A, Marescaux J, Soler L (2013) Evaluation of Endoscopic Image Enhancement for Feature Tracking: A New Validation Framework. In: Augmented Reality Environments for Medical Imaging and Computer-Assisted Interventions, pp 75–85
Abdullah-Al-Wadud M, Kabir MH, Dewan MAA, Chae O (2007) A dynamic histogram equalization for image contrast enhancement. IEEE Trans Consum Electron 53(2):593– 600
Perona P, Malik J (1990) Scale-space and edge detection using anisotropic diffusion. IEEE Trans Pattern Anal Mach Intell 12(7):629–639
Li L, Zouthe YX, Li Y (2013) Wireless capsule endoscopy images enhancement based on adaptive anisotropic diffusion. In: 2013 IEEE China Summit and International Conference on Signal and Information Processing, pp 273–277
Erickson BJ, Korfiatis P, Kline TL, Akkus Z, Philbrick K, Weston AD (2018) Deep learning in radiology: does one size fit all?. J Am Coll Radiol 15(3):521–526
Sahiner B, et al. (2019) Deep learning in medical imaging and radiation therapy. Med Phys 46 (1):e1–e36
Wong KCL, Syeda-Mahmood T, Moradi M (2018) Building medical image classifiers with very limited data using segmentation networks. Med Image Anal 49:105–116
Li C, Guo J, Porikli F, Pang Y (2018) Lightennet: A convolutional neural network for weakly illuminated image enhancement. Pattern Recognit Lett 104:15–22
Wang W, Wei C, Yang W, Liu J (2018) GLADNet: Low-light enhancement network with global awareness, in 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp 751–755
Lv F, Lu F, Wu J, Lim C (2018) MBLLEN: Low-Light Image/Video Enhancement Using CNNs. In: BMVC, pp 220
Litjens G, et al (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88
Greenspan H, Van Ginneken B, Summers RM (2016) Guest editorial deep learning in medical imaging: Overview and future promise of an exciting new technique. IEEE Trans Med Imaging 35(5):1153–1159
Author information
Authors and Affiliations
Corresponding authors
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Asif, M., Chen, L., Song, H. et al. An automatic framework for endoscopic image restoration and enhancement. Appl Intell 51, 1959–1971 (2021). https://doi.org/10.1007/s10489-020-01923-w
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-020-01923-w