Intrinsic layer based automatic specular reflection detection in endoscopic images

https://doi.org/10.1016/j.compbiomed.2020.104106Get rights and content

Highlights

  • Specular reflection (SR) in endoscopy images affects the performance of Minimally Invasive Surgery.

  • Researchers have shown that SR removal can improve diagnostic and operational precision over a shorter period.

  • A novel Intrinsic Image layer Separation (IILS) method is presented to detect SR from the endoscopy images.

  • The CVC-EndoScenceStill DB has been utilized for experiment evaluation.

  • Experimental results are demonstrated to support the proposed methodology as compared to the state of the art methods.

Abstract

Endoscopic images are used to observe the internal structure of the human body. Specular reflection (SR) images are mostly a consequence of the strong light and bright regions appearing on endoscopic images, which affects the performance of minimally invasive surgery. In this study, we propose a novel method for automatic SR detection based on intrinsic image layer separation (IILS). The proposed method consists of three steps. Initially, it involves the normalization of the image followed by the extraction of high gradient area, and the separation of SR is done on the basis of the color model. The image melding technique is utilized to reconstruct the reflected pixels. The experiments were conducted on 912 endoscopic images from CVC-EndoSceneStill. The results of accuracy, sensitivity, specificity, precision, Jaccard index, Dice coefficient, standard deviation, and pixel count difference show that the detection performance of the proposed method outperforms that of other state-of-the-art methods. The evaluation of the proposed IILS-based SR detection demonstrates that our method obtains better qualitative and quantitative assessments compared with other methods, which can be used as a promising preprocessing step for further analysis of endoscopic images.

Introduction

Endoscopy enables doctors to observe the internal body structure. However, monitoring disease progression is challenging due to the uncertainty posed by endoscopy, the limited point of view, and the presence of artifacts in imaging. The strong highlights that appear on endoscopic images at the time of capture are called specular reflection (SR), which limits visual understanding. Thus, physicians require SR-free endoscopic images because inherited illumination distracts the observer and affects further automatic image analysis [[1], [2], [3]]. This occurrence is illustrated in Fig. 1.

A non-linear filtering and color-based segmentation method was utilized in [4] for the automatic detection of SR. The two-level approach is proposed for inpainting. First, the reflected pixels are replaced with the centroid color of the pixels. Then, Gaussian filter is used for smoothing the image. Zimmerman and Greenspan [5] implemented two or four Gaussian distributions for modeling the specular and non-specular regions by separating hue as a specular region and intensity as a non-specular region from the HIS color space of an image. However, It is not useful in many examples because it requires prior knowledge about Gaussian distributions. The filling of targeted pixels is propagated by neighboring color pixels. Oh et al. [6] divided the image brightness into two regions, that is, absolute bright and relative bright, by converting the RGB image to HSV image space. Then, the reflected region was separated by applying two thresholds on S and V values. SR removal is performed by replacing the reflected pixel values with the average boundary pixel values. The methods used in [[4], [5], [6]] exclude reflection based on an averaging algorithm and compute the average by neighborhood pixels with the assumption that the reflected areas have a homogenous background. However, these methods fail when the reflection covers a large area. The method in [7] separates the reflection by comparing chrominance and luminance of the enhanced endoscopic images, in which lower chrominance pixels refer to reflected regions, while image inpainting has been done by the use of multi-resolution inpainting technique. T. Stehle [8] developed a method for the segmentation of SR detection by using a threshold in YUV color space and for the correction of detected pixels by using interpolation. The authors in [9] found differences between specular and non-specular regions in endoscopic images by using non-linear filtering and corrected the segmented reflection on the basis of the fast digital inpainting method proposed in [10]. The digital inpainting method (also known as diffusion) means image interpolation, and it suffers from insufficient effectiveness. Stoyanov et al. [11] performed laparoscopic surgery and used a static threshold on color features to eliminate SR. Ciuti et al. [12] used the threshold method [11] for image-guided navigation in locomotion capsule endoscopy to improve the three-dimensional shape reconstruction. The abovementioned methods overcome the SR problem by choosing a fixed threshold on the intensity values of a single static image. The under- or overestimation of SR is the main issue in these methods. However, these methods do not effectively eliminate SR in a different sequence of endoscopic images. Alsaleh et al. [13] proposed a gradient-based edge detection method and utilized the color adaptive threshold to extract a specular region from endoscopic images. Nevertheless, this method does not work on large SR regions and cannot cleanly detect SR from endoscopic images.

The intrinsic image decomposition technique separates two unknown layers from a single input image. When capturing the image behind the glass, image reflection becomes a part of the resulting image. These images consist of two layers, namely, the reflectance of the scene and illuminance [14]. I=RL, where R shows the reflected layer, L shows the luminance layer, and I is the captured image, which is the sum of the two layers (R and L). Each pixel level has a reflection or albedo property. Similarly, every pixel has illuminance. The illuminance varies throughout the image, but reflectance remains constant [[15], [16], [17], [18]]. However, the reflected layer is smoother than the illuminance layer. The intrinsic feature is being recovered from the images and derived from intrinsic image by using the sequence of the image [14,19]. Bousseau et al. [20] extracted an intrinsic image through user markup. Moreover, another type of work is done on intrinsic image layer separation (IILS) by using a single image as an input for layer separation [15,16,18,[21], [22], [23]]. The authors of [22] proposed a modified version of [24] for effective intrinsic layer separation by applying a blur filter on the reflecting layer I=LB+LRh. where LB represents the illuminance layer and LR represents reflected, and h is for blur filter.

A well-known patch-based approach is presented in [25], which copies and pastes image patches from the texture example image to the real image. Such techniques are called “exemplary” techniques or “patch-based” methods of painting. Subsequent studies changed the quest and sampling method to improve the preservation of the structure. Pritch et al. [26] proposed a shift map for image editing by using different rearrangements of geometry problems. Wexler et al. [27] formulated the problems of completion as global optimization, which achieves more reliable fills. Additional applications such as image summarization, stitching collages, and image morphing [28], were illustrated using an additional objective term capturing local resemblance of the source to the target [29]. Barnes et al. [29] used PatchMatch, a fast randomized patch search algorithm, to expedite these techniques. This approach has been expanded to searching for rotations and scales for computer vision applications [30] and searching for bias and gains per color channel to find correlations between various shared content photographs [31]. Arias et al. [32] implemented a gradient concept in patch similarity and added an L1 norm for gradients to manage extremely detailed textures in regions. Darabi et al. [33] proposed the addition of a space in the image melding method to the patch search with geometric and photometric transformations, the integration of image gradients, and substitution of normal image gradients with a screened Poisson equation solver for color averaging. The progressive transition from source to source sharpness of texture without compromising is handled by the mixed L2/L0 norm for gradients and colors.

Nevertheless, the previous methods for SR detection have unsolved issues and limitations. The contributions of our work are as follows.

  • We introduced a new approach for the segmentation of highly penetrated reflected areas in endoscopic images without losing the semantical information by using IILS, which helps to produce better results for the posterior procedure.

  • The proposed approach does not use direct empirical reflectance values for the segmentation of the reflected regions and removes the threshold-based methodologies barrier.

  • A patch-based approach is utilized to properly reconstruct the reflected regions. The proposed solution can be extended to various endoscopic image analysis procedures.

Section snippets

SR detection method

SR removal from endoscopy images is very essential because the highlighted area disrupts the continuity in the image and occludes the image appearance and additional features. All endoscopy images are usually required to remove the SR for further automatic image analysis. The proposed SR detection method consists of three steps: (i) preprocessing, (ii) reflected layer separation, and (iii) SR detection. Fig. 2 shows the flow diagram of the proposed method for the automatic detection of SR. Most

Database and evaluation

The proposed method is evaluated on publicly available dataset CVC-EndoSceneStill [37] with SR Ground Truth (GT). The CVC-EndoSceneStill is the combination of already proposed CVC-ColonDB [38] and Cvc-Clinicdb [39] in the MICCAI 2015 challenge [40] for polyp benchmarks. The [37] contain on 912 (300 + 612) endoscopic images acquired from 36 (13 + 23) patients and obtained from 44 (13 + 31) videos with ((500 574), (384 288)) image resolution. All the tests are performed on MATLAB R2017a, and

Discussion

Although the proposed approach can better remove highlight pixels, it also has limitations. It does not work well because of the existence of a very large SR field. Because there are many regions in the original image that are comparatively highlight and adaptive to the regions around them. The highlights almost fill the full image, and it cannot be cleanly removed by our method. The failure is that the image has too many highlights and the specular portion of reflection that can no longer be

Conclusion

In this study, a novel IILS-based method was developed for automatic SR detection from endoscopic images. We evaluated the performance of the proposed method by conducting experiments on 912 endoscopic images. The results demonstrated that our method obtains better qualitative and quantitative assessments compared with other state-of-the-art methods. Moreover, our automatic SR detection method can be used as the preprocessing step for further analysis of endoscopic images. In the future, we

Declaration of competing interest

The authors declare that they have no competing interests.

Acknowledgments

This work was supported by National Key R& D Program of China 2017YFC0110700 and National Natural Science Foundation of China (61771056 and 61672099).

Muhammad Asif Ph.D. student at the School of Computer Science and Technology, Beijing Institute of Technology, China. His Major is Software Engineering. His research interest in Medical Image Processing and currently working on Endoscopy images. He is graduated from the National University of Science and Technology (NUST), Pakistan, in 2016.

References (51)

  • C.A. Saint-Pierre et al.

    Detection and correction of specular reflections for automatic surgical tool segmentation in thoracoscopic images

    Mach. Vis. Appl.

    (2011)
  • M.M.O.B.B. Richard et al.

    Fast digital image inpainting

  • D. Stoyanov et al.

    Soft-tissue motion tracking and structure estimation for robotic assisted MIS procedures

    Lect. Notes Comput. Sci.

    (2005)
  • G. Ciuti et al.

    Intra-operative monocular 3D reconstruction for image-guided navigation in active locomotion capsule endoscopy

    Proc. IEEE RAS EMBS Int. Conf. Biomed. Robot. Biomechatron.

    (2012)
  • S.M. Alsaleh et al.

    Adaptive Segmentation and Mask-specific Sobolev Inpainting of Specular Highlights for Endoscopic Images

    (2016)
  • H.G. Barrow et al.

    Recovering intrinsic scene characteristics from images

    Comput. Vis. Syst.

    (1978)
  • M.F. Tappen et al.

    Recovering intrinsic images from a single image

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2005)
  • R. Grosse et al.

    Ground truth dataset and baseline evaluations for intrinsic image algorithms

    Proc. IEEE Int. Conf. Comput. Vis.

    (2009)
  • P.V. Gehler

    Recovering intrinsic images with a global sparsity prior on reflectance

    NIPS (News Physiol. Sci.)

    (2011)
  • L. Shen et al.

    Intrinsic images decomposition using a local and global sparse representation of reflectance

    IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recogn.

    (2011)
  • Y. Weiss

    Deriving intrinsic images from image sequences

  • A. Bousseau et al.

    User-assisted intrinsic images

    ACM Trans. Graph.

    (2009)
  • L. Shen et al.

    Intrinsic image decomposition with non-local texture cues

  • Y. Li et al.

    “Single image layer separation using relative smoothness

    Comput. Vis. Pattern Recognit. (CVPR), 2014 IEEE Conf.

    (2014)
  • M. Asif et al.

    Relative smoothness for image layer separation by second order laplacian filter

  • Cited by (13)

    • Specular highlight removal of light field based on dichromatic reflection and total variation optimizations

      2022, Optics and Lasers in Engineering
      Citation Excerpt :

      Unlike multiple exposures, the recovery algorithm from single exposure is more challenging despite its low complexity of image acquisition. Early studies based on intrinsic image decomposition methods had been developed to deal with specular highlight removal [7,8]. Kim et al. presented the maximum posteriori optimization framework to separate specular reflection from single image, which had achieved considerable results, but it was slightly inferior in complex texture areas [9].

    View all citing articles on Scopus

    Muhammad Asif Ph.D. student at the School of Computer Science and Technology, Beijing Institute of Technology, China. His Major is Software Engineering. His research interest in Medical Image Processing and currently working on Endoscopy images. He is graduated from the National University of Science and Technology (NUST), Pakistan, in 2016.

    Hong Song is a professor of the School of Computer Science and Technology, Beijing Institute of Technology. Her focus is on the application of machine learning and deep learning methods for medical image analysis and improved computer-aided diagnosis. She received her Ph.D. at the School of Computer Science and Technology, Beijing Institute of Technology.

    Lei Chen is a Ph.D. student at the School of Computer Science and Technology, Beijing Institute of Technology, China. His major is software engineering. His research interests lie in medical image processing using machine learning and deep learning methods. He graduated from the Tianjin Institute of Technology, China, in 2016.

    Jian Yang is a professor of the School of Optics and Photonics, Beijing Institute of Technology. His focus is the application of virtual reality, medical image processing and computer vision. He received his Ph.D. at the School of Optics and Photonics, Beijing Institute of Technology, and was a Research Fellow at the Hospital for Sick Children, Canada, from 2007 to 2009.

    Alejandro F Frangi is a Professor of Biomedical Image Computing at the University of Leeds and affiliated to the School of Computing and School of Medicine, UK. He is also Director of the Center for Computational Imaging and Simulation Technologies in Biomedicine. Prof Frangi is a Fellow of IEEE.

    View full text