Intrinsic layer based automatic specular reflection detection in endoscopic images
Introduction
Endoscopy enables doctors to observe the internal body structure. However, monitoring disease progression is challenging due to the uncertainty posed by endoscopy, the limited point of view, and the presence of artifacts in imaging. The strong highlights that appear on endoscopic images at the time of capture are called specular reflection (SR), which limits visual understanding. Thus, physicians require SR-free endoscopic images because inherited illumination distracts the observer and affects further automatic image analysis [[1], [2], [3]]. This occurrence is illustrated in Fig. 1.
A non-linear filtering and color-based segmentation method was utilized in [4] for the automatic detection of SR. The two-level approach is proposed for inpainting. First, the reflected pixels are replaced with the centroid color of the pixels. Then, Gaussian filter is used for smoothing the image. Zimmerman and Greenspan [5] implemented two or four Gaussian distributions for modeling the specular and non-specular regions by separating hue as a specular region and intensity as a non-specular region from the HIS color space of an image. However, It is not useful in many examples because it requires prior knowledge about Gaussian distributions. The filling of targeted pixels is propagated by neighboring color pixels. Oh et al. [6] divided the image brightness into two regions, that is, absolute bright and relative bright, by converting the RGB image to HSV image space. Then, the reflected region was separated by applying two thresholds on S and V values. SR removal is performed by replacing the reflected pixel values with the average boundary pixel values. The methods used in [[4], [5], [6]] exclude reflection based on an averaging algorithm and compute the average by neighborhood pixels with the assumption that the reflected areas have a homogenous background. However, these methods fail when the reflection covers a large area. The method in [7] separates the reflection by comparing chrominance and luminance of the enhanced endoscopic images, in which lower chrominance pixels refer to reflected regions, while image inpainting has been done by the use of multi-resolution inpainting technique. T. Stehle [8] developed a method for the segmentation of SR detection by using a threshold in YUV color space and for the correction of detected pixels by using interpolation. The authors in [9] found differences between specular and non-specular regions in endoscopic images by using non-linear filtering and corrected the segmented reflection on the basis of the fast digital inpainting method proposed in [10]. The digital inpainting method (also known as diffusion) means image interpolation, and it suffers from insufficient effectiveness. Stoyanov et al. [11] performed laparoscopic surgery and used a static threshold on color features to eliminate SR. Ciuti et al. [12] used the threshold method [11] for image-guided navigation in locomotion capsule endoscopy to improve the three-dimensional shape reconstruction. The abovementioned methods overcome the SR problem by choosing a fixed threshold on the intensity values of a single static image. The under- or overestimation of SR is the main issue in these methods. However, these methods do not effectively eliminate SR in a different sequence of endoscopic images. Alsaleh et al. [13] proposed a gradient-based edge detection method and utilized the color adaptive threshold to extract a specular region from endoscopic images. Nevertheless, this method does not work on large SR regions and cannot cleanly detect SR from endoscopic images.
The intrinsic image decomposition technique separates two unknown layers from a single input image. When capturing the image behind the glass, image reflection becomes a part of the resulting image. These images consist of two layers, namely, the reflectance of the scene and illuminance [14]. , where R shows the reflected layer, L shows the luminance layer, and I is the captured image, which is the sum of the two layers (R and L). Each pixel level has a reflection or albedo property. Similarly, every pixel has illuminance. The illuminance varies throughout the image, but reflectance remains constant [[15], [16], [17], [18]]. However, the reflected layer is smoother than the illuminance layer. The intrinsic feature is being recovered from the images and derived from intrinsic image by using the sequence of the image [14,19]. Bousseau et al. [20] extracted an intrinsic image through user markup. Moreover, another type of work is done on intrinsic image layer separation (IILS) by using a single image as an input for layer separation [15,16,18,[21], [22], [23]]. The authors of [22] proposed a modified version of [24] for effective intrinsic layer separation by applying a blur filter on the reflecting layer . where represents the illuminance layer and represents reflected, and h is for blur filter.
A well-known patch-based approach is presented in [25], which copies and pastes image patches from the texture example image to the real image. Such techniques are called “exemplary” techniques or “patch-based” methods of painting. Subsequent studies changed the quest and sampling method to improve the preservation of the structure. Pritch et al. [26] proposed a shift map for image editing by using different rearrangements of geometry problems. Wexler et al. [27] formulated the problems of completion as global optimization, which achieves more reliable fills. Additional applications such as image summarization, stitching collages, and image morphing [28], were illustrated using an additional objective term capturing local resemblance of the source to the target [29]. Barnes et al. [29] used PatchMatch, a fast randomized patch search algorithm, to expedite these techniques. This approach has been expanded to searching for rotations and scales for computer vision applications [30] and searching for bias and gains per color channel to find correlations between various shared content photographs [31]. Arias et al. [32] implemented a gradient concept in patch similarity and added an norm for gradients to manage extremely detailed textures in regions. Darabi et al. [33] proposed the addition of a space in the image melding method to the patch search with geometric and photometric transformations, the integration of image gradients, and substitution of normal image gradients with a screened Poisson equation solver for color averaging. The progressive transition from source to source sharpness of texture without compromising is handled by the mixed norm for gradients and colors.
Nevertheless, the previous methods for SR detection have unsolved issues and limitations. The contributions of our work are as follows.
- •
We introduced a new approach for the segmentation of highly penetrated reflected areas in endoscopic images without losing the semantical information by using IILS, which helps to produce better results for the posterior procedure.
- •
The proposed approach does not use direct empirical reflectance values for the segmentation of the reflected regions and removes the threshold-based methodologies barrier.
- •
A patch-based approach is utilized to properly reconstruct the reflected regions. The proposed solution can be extended to various endoscopic image analysis procedures.
Section snippets
SR detection method
SR removal from endoscopy images is very essential because the highlighted area disrupts the continuity in the image and occludes the image appearance and additional features. All endoscopy images are usually required to remove the SR for further automatic image analysis. The proposed SR detection method consists of three steps: (i) preprocessing, (ii) reflected layer separation, and (iii) SR detection. Fig. 2 shows the flow diagram of the proposed method for the automatic detection of SR. Most
Database and evaluation
The proposed method is evaluated on publicly available dataset CVC-EndoSceneStill [37] with SR Ground Truth (GT). The CVC-EndoSceneStill is the combination of already proposed CVC-ColonDB [38] and Cvc-Clinicdb [39] in the MICCAI 2015 challenge [40] for polyp benchmarks. The [37] contain on 912 (300 + 612) endoscopic images acquired from 36 (13 + 23) patients and obtained from 44 (13 + 31) videos with ((500 574), (384 288)) image resolution. All the tests are performed on MATLAB R2017a, and
Discussion
Although the proposed approach can better remove highlight pixels, it also has limitations. It does not work well because of the existence of a very large SR field. Because there are many regions in the original image that are comparatively highlight and adaptive to the regions around them. The highlights almost fill the full image, and it cannot be cleanly removed by our method. The failure is that the image has too many highlights and the specular portion of reflection that can no longer be
Conclusion
In this study, a novel IILS-based method was developed for automatic SR detection from endoscopic images. We evaluated the performance of the proposed method by conducting experiments on 912 endoscopic images. The results demonstrated that our method obtains better qualitative and quantitative assessments compared with other state-of-the-art methods. Moreover, our automatic SR detection method can be used as the preprocessing step for further analysis of endoscopic images. In the future, we
Declaration of competing interest
The authors declare that they have no competing interests.
Acknowledgments
This work was supported by National Key R& D Program of China 2017YFC0110700 and National Natural Science Foundation of China (61771056 and 61672099).
Muhammad Asif Ph.D. student at the School of Computer Science and Technology, Beijing Institute of Technology, China. His Major is Software Engineering. His research interest in Medical Image Processing and currently working on Endoscopy images. He is graduated from the National University of Science and Technology (NUST), Pakistan, in 2016.
References (51)
- et al.
Detection of specular reflection and segmentation of cervix region in uterine cervix images for cervical cancer screening
Irbm
(2017) - et al.
Informative frame classification for endoscopy video
Med. Image Anal.
(2007) - et al.
WM-DOVA maps for accurate polyp highlighting in colonoscopy: validation vs. saliency maps from physicians
Comput. Med. Imag. Graph.
(2015) - et al.
An endoscopic 3D scanner based on structured light
Med. Image Anal.
(2012) - et al.
A survey of specularity removal methods
Comput. Graph. Forum
(2011) Making the invisible visible: highlight substitution by color light fields
Conf. Colour Graph., Imag., Vision
(2002)- et al.
Automatic segmentation and inpainting of specular highlights for endoscopic imaging
Eurasip J. Image Video Process
(2010) - et al.
Automatic detection of specular reflections in uterine cervix images
- et al.
Automatic detection and inpainting of specular reflections for colposcopic images
Open Comput. Sci.
(2011) Removal of specular reflections in endoscopic images
Acta Polytech. J. Adv. Eng.
(2006)
Detection and correction of specular reflections for automatic surgical tool segmentation in thoracoscopic images
Mach. Vis. Appl.
Fast digital image inpainting
Soft-tissue motion tracking and structure estimation for robotic assisted MIS procedures
Lect. Notes Comput. Sci.
Intra-operative monocular 3D reconstruction for image-guided navigation in active locomotion capsule endoscopy
Proc. IEEE RAS EMBS Int. Conf. Biomed. Robot. Biomechatron.
Adaptive Segmentation and Mask-specific Sobolev Inpainting of Specular Highlights for Endoscopic Images
Recovering intrinsic scene characteristics from images
Comput. Vis. Syst.
Recovering intrinsic images from a single image
IEEE Trans. Pattern Anal. Mach. Intell.
Ground truth dataset and baseline evaluations for intrinsic image algorithms
Proc. IEEE Int. Conf. Comput. Vis.
Recovering intrinsic images with a global sparsity prior on reflectance
NIPS (News Physiol. Sci.)
Intrinsic images decomposition using a local and global sparse representation of reflectance
IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn.
Deriving intrinsic images from image sequences
User-assisted intrinsic images
ACM Trans. Graph.
Intrinsic image decomposition with non-local texture cues
“Single image layer separation using relative smoothness
Comput. Vis. Pattern Recognit. (CVPR), 2014 IEEE Conf.
Relative smoothness for image layer separation by second order laplacian filter
Cited by (13)
Specular highlight removal of light field image combining dichromatic reflection with exemplar patch filling
2024, Optics and Lasers in EngineeringEnhancing Wireless Capsule Endoscopy images from intense illumination specular reflections using the homomorphic filter
2023, Biomedical Signal Processing and ControlSpecular highlight removal of light field based on dichromatic reflection and total variation optimizations
2022, Optics and Lasers in EngineeringCitation Excerpt :Unlike multiple exposures, the recovery algorithm from single exposure is more challenging despite its low complexity of image acquisition. Early studies based on intrinsic image decomposition methods had been developed to deal with specular highlight removal [7,8]. Kim et al. presented the maximum posteriori optimization framework to separate specular reflection from single image, which had achieved considerable results, but it was slightly inferior in complex texture areas [9].
Feed-forward LPQNet based Automatic Alzheimer's Disease Detection Model
2021, Computers in Biology and MedicineComputer-aided diagnosis of polyp classification using scale invariant features and extreme gradient boosting
2023, Journal of Medical Physics
Muhammad Asif Ph.D. student at the School of Computer Science and Technology, Beijing Institute of Technology, China. His Major is Software Engineering. His research interest in Medical Image Processing and currently working on Endoscopy images. He is graduated from the National University of Science and Technology (NUST), Pakistan, in 2016.
Hong Song is a professor of the School of Computer Science and Technology, Beijing Institute of Technology. Her focus is on the application of machine learning and deep learning methods for medical image analysis and improved computer-aided diagnosis. She received her Ph.D. at the School of Computer Science and Technology, Beijing Institute of Technology.
Lei Chen is a Ph.D. student at the School of Computer Science and Technology, Beijing Institute of Technology, China. His major is software engineering. His research interests lie in medical image processing using machine learning and deep learning methods. He graduated from the Tianjin Institute of Technology, China, in 2016.
Jian Yang is a professor of the School of Optics and Photonics, Beijing Institute of Technology. His focus is the application of virtual reality, medical image processing and computer vision. He received his Ph.D. at the School of Optics and Photonics, Beijing Institute of Technology, and was a Research Fellow at the Hospital for Sick Children, Canada, from 2007 to 2009.
Alejandro F Frangi is a Professor of Biomedical Image Computing at the University of Leeds and affiliated to the School of Computing and School of Medicine, UK. He is also Director of the Center for Computational Imaging and Simulation Technologies in Biomedicine. Prof Frangi is a Fellow of IEEE.