Multi-spectral video endoscopy system for the detection of cancerous tissue
Highlights
► We propose a multi-spectral video endoscopy system for cancer detection. ► Non-rigid movement is compensated by optical flow based image registration. ► A hyper-spectral classification model successfully predicts cancerous tissue regions. ► The proposed video endoscopy system allows overlays indicating cancerous regions.
Introduction
Video endoscopy combined with histopathological analysis of biopsies is currently the gold standard for preventive examinations and diagnosis of cancer and its precursors in otolaryngoscopy, gastroenteroscopy and colonoscopy (Zoller et al., 2000, Winawer et al., 1997, Schimao et al., 2000). Experienced experts achieve an excellent diagnosis performance in terms of sensitivity and specificity. However, one of these experienced domain experts is not always available and the investigation is conducted by less-experienced physicians. A video endoscope that semi-autonomously identifies cancerous and pre-cancerous tissue regions by augmenting the video stream with an overlay highlighting regions that require specific examination would help in these situations. This paper proposes such a system and discusses the results obtained for the detection of cancerous tissue regions.
Spectroscopic measurements of biopsies of different cancerous and pre-cancerous tissues proved that their spectral fingerprint provides valuable information for the diagnostic process (Sigurdsson et al., 2004, de Veld et al., 2005, Haka et al., 2005, Faigel, 2006). The early diagnosis and the extent of the cancerous tissue, in particular its margin to the healthy tissue, is crucial for the diagnosis and the therapy selection (Rubin, 1974, State et al., 1950, Seykora and Elder, 1996). Consequently, point spectroscopy alone is not sufficient, and a spatially resolved spectroscopy is required. In other words: a multi-spectral video endoscope that provides video data augmented with spatially resolved spectroscopic results will enable an early diagnosis of cancerous and pre-cancerous tissues. In the past multi-spectral image acquisition systems were of limited use for endoscopy due to (i) the necessary spatial scanning of push-broom approaches or (ii) the impractical long switching times of liquid crystal tunable filters. Recent technological advances in the field of tuneable filters, in particular the development of fast acousto-optical tunable filters (AOTF), made switching times below 1 ms feasible. Thus, AOTFs represent a suitable technology for the acquisition of multi-spectral image and video data with excellent spatial, spectral and temporal resolution.
In this paper, we propose a hyper-spectral imaging endoscope using a fast AOTF synchronised with a highly sensitive EMCCD 1 camera that allows the acquisition of hyper-spectral image and multi-spectral video data. A multi-variate classification applicable to both the hyper-spectral image and the multi-spectral video data enables the detection of cancerous tissue regions in endoscopic applications. An image registration ensures the spectral integrity of the acquired data also in case of non-rigid scene deformations that are common during endoscopic examinations. The acquired multi-spectral video data can be used for a live colour display and an overlay of a spatially resolved multi-variate classification result based on the spectral information available for each pixel. Such a system could be used during endoscopic investigations to provide the physician additional image-based information in parallel with the video stream to support the diagnostic and therapeutic decisions.
The paper is divided into the following sections: Related work in multi-spectral video acquisition and endoscopic cancer diagnosis is discussed in the next section. The hardware of the proposed hyper-spectral endoscopy system to acquire hyper-spectral image and multi-spectral video data is discussed in Section 3. This section also contains the description of the used calibration and pre-processing steps. The hyper-spectral image registration is explained in Section 4 and the multivariate classification approaches in Section 5. Section 6 explains how the hyper-spectral datasets were acquired and labeled by domain experts. The evaluation results of the image registration and the different classification approaches including their interpretation are reported in Section 7. In the final Section 8 the findings of the paper are summarised and the conclusions from the achieved results are drawn.
Section snippets
Multi-spectral video acquisition
Spectral imaging is the acquisition and processing of images with more than three colour planes enabling spatially resolved spectroscopy. These colour planes are usually denoted as spectral bands as the term colour plane has only a meaning for RGB2 or CMYK3 images. The term multi-spectral image refers to images with up to twelve bands while hyper-spectral images generally contain more than twelve, sometimes up to several hundred bands. Several
Demonstrator hardware
The demonstrator for multi-spectral video endoscopy was designed to fulfill two requirements: (i) acquisition of multi-spectral videos with up to 8 bands and 40 frames per second, and (ii) the acquisition of hyper-spectral images with 5 nm spectral resolution and 1 Mpixel spatial resolution. Liquid crystal tunable filters (LCTF) have a generally better blocking efficiency, but are with 50–150 ms switching time too slow for our purposes. AOTFs achieve switching times around 50 μs and are thus faster
Hyper-spectral image registration
An endoscope is usually guided manually and the sample may move during the capture of a hyper-spectral image relative to the camera. We use image registration techniques to remove any shift between consecutive image frames that may corrupt the spectral information and any subsequent hyper-spectral classification. The images are warped using an inverse mapping with bicubic interpolation. Two different registration techniques have been evaluated: (i) an affine transformation estimated from SIFT
Hyper-spectral classification
The hyper-spectral image datasets are subjected to a PCA6 to reduce the dimensionality to eight principal components (PC). All data sets are projected to the feature space constructed by their first eight PCs that preserve 99% of the total variance. We chose a QDC (Duda et al., 2001) to avoid overtraining and evaluate the potential of a multi-variate classification based on the information available in the spectra.
The questions we want to answer by an evaluation of
Tissue biopsies and expert annotations
The demonstrator has been evaluated during 36 measurements in clinical environment for otolaryngoscopic investigations at the Katharinenhospital (Stuttgart, Germany). Table 1 lists the measurements acquired with the demonstrator. The demonstrator was used to acquire hyper-spectral images (1004 × 1002 pixels, 51 bands from 400–650 nm with 5 nm FWHM) of biopsies taken after the investigations. The hyper-spectral images of the biopsies were acquired within 10–15 min after the excision. Within this
Registration and classification results
The results are divided into two parts: (i) the analysis of the multi-spectral image registration and (ii) the evaluation of the three hyper-spectral classification settings. Both are crucial for a video endoscopy setup to be capable of visualising video data augmented with an overlay indicating the classification result of consistent multi-spectral measurements as described in Section 1.
Conclusions
We have presented the hardware components and algorithmic blocks required for a multi-spectral video endoscope able to detect cancerous tissue regions. The system can acquire hyper-spectral images with 51 bands from 400–650 nm with a FWHM of 5 nm within 1.25 s, or multi-spectral videos with 8 bands (specific wavelenghts) at 5 multi-spectral image cubes per second.
The classification results achieved for otolaryngoscopic biopsies are encouraging. However, it was shown once more, that the spectral
Acknowledgements
The work described in this paper and the Competence Centre CTR is funded within the R&D Program COMET - Competence Centers for Excellent Technologies by the Federal Ministries of Transport, Innovation and Technology (BMVIT), of Economics and Labour (BMWA) and it is managed on their behalf by the Austrian Research Promotion Agency (FFG).
The Austrian provinces (Carinthia and Styria) provide additional funding.
References (50)
- et al.
Narrow-band imaging system with magnifying endoscopy for rectal amyloidosis
Gastrointest. Endosc.
(2008) - et al.
Real-time classification of polymers with NIR spectral imaging and blob analysis
Real Time Imaging
(2003) - et al.
Near infrared spectroscopy for determination of mycotoxins in cereals
Food Control
(2003) - et al.
An improved optical flow method for image registration with large-scale movements
Acta Automatica Sinica
(2008) - Arnold, T., De Biasio, M., Leitner, R., 2010. High-sensitivity hyper-spectral video endoscopy system for intra-surgery...
- Arnold, T., De Biasio, M., Leitner, R., 2011. Hyper-spectral video endoscope for intra-surgery tissue classification...
- Azaria, I., Goldshleger, N., Ben-Dor, E., Bar-Hamburger, R., 2009. Detection of cannabis plants by hyper-spectral...
- et al.
Color Image Processing
(2006) - et al.
Mapping minerals, amorphous materials, environmental materials, vegetation, water, ice, and snow, and other materials: the USGS tricorder algorithm
- De Biasio, M., Arnold, T., McGunnigle, G., Kraft, M., Leitner, R., Balthasar, D., Rehrmann, V., 2011. Detection of fire...
Near-infrared autofluorescence imaging for detection of cancer
J. Biomed. Optics
Clinical study for classification of benign, dysplastic, and malignant oral lesions using autofluorescence spectroscopy
J. Biomed. Optics
Segmentation of multi-spectral images using the combined classifier approach
Lasers Med. Sci.
Training data selection for cancer detection in multispectral endoscopy images
Proc. 21st International Conference on Pattern Recognition ICPR 2012
Pattern Classification
Endoscopic Oncology: Gastrointestinal Endoscopy and Cancer Management
Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography
SRI International, Communications of the ACM
Diagnosing breast cancer by using raman spectroscopy
Proc. Natl. Acad. Sci.
In vivo margin assessment during partial mastectomy breast surgery using raman spectroscopy
Cancer Res.
Prospective study of the frequency and size distribution of polyps missed by colonoscopy
J. Natl. Cancer Inst.
Snapshot hyperspectral imaging in ophthalmology
J. Biomed. Optics
Segmentation and classification of hyper-spectral skin data
Proceedings of the 31st Annual Conference of the German Classification Society
Initial vegetation species and senescence/stress indicator mapping in the san luis valley, colorado using imaging spectrometer data
Summaries of the Fifth Annual JPL Airborne Earth Science Workshop
Cited by (40)
Compressive Spectral Video Sensing using the Convolutional Sparse Coding framework CSC4D
2023, Journal of Visual Communication and Image RepresentationResearch on passive low-light-level optical fiber endoscope detection system
2021, OptikCitation Excerpt :It has the characteristics of high quantum efficiency and high signal-to-noise ratio [10–12]. Raimund Leitner et al. proposed a hyperspectral imaging endoscope system [13], which used a fast AOTF in conjunction with a highly sensitive EMCCD camera to detect cancerous tissue. Cao Xin et al. integrated the clinical endoscope with the emerging Cerenkov luminescence imaging (CLI) technology, and developed a endoscopic Cerenkov light imaging system (ECLI) [14].
A High-Speed Hyperspectral Laparoscopic Imaging System
2023, Progress in Biomedical Optics and Imaging - Proceedings of SPIEApplication of EMCCD in the Small-Optical Fiber Endoscope Imaging Detection System
2023, Proceedings of SPIE - The International Society for Optical EngineeringHyperspectral image compression method based on 3D Saab transform
2022, Beijing Hangkong Hangtian Daxue Xuebao/Journal of Beijing University of Aeronautics and Astronautics