Abstract
Today, photodynamic diagnostics is commonly used in endoscopic intervention of the urinary bladder. Excited by a narrow band illumination, fluorescence markers enhance the visual contrast between benign and malignant tissue. Since in this modality the endoscope must be moved close to the bladder wall to provide sufficiently exposed images, the field of view (FOV) of the endoscope is very limited. This impedes the navigation and the re-identifying of multi-focal tumors for the physician. Thus, an image providing a larger FOV, composed from single images is highly desired during the intervention for surgery assistance. Since endoscopic mosaicking in real-time is still an open issue, we introduce a new feature-based image mosaicking algorithm for fluorescence endoscopy. Using a multi-threaded software design, the extraction of SURF features, the matching and the image stitching are separated in single processing threads. In an optimization step we discuss the trade-off between feature repeatability and processing time. After adjusting an optimal thread synchronization, the optimal workload of each thread results in a fast and real-time capable computation of image mosaics. On a standard hardware platform our algorithm performs within the RealTimeFrame framework with an update rate of 8.17 frames per second at full input image resolution (780×576). Providing a fast growing image with an extended FOV during the intervention, the mosaic displayed on a second monitor promises high potential for surgery assistance.
Similar content being viewed by others
References
American Cancer Society: Cancer facts and figures 2009
Bagley DH, Huffman JL, Lyon ES (1985) Urologic endoscopy: a manual and atlas. Lippincott Williams and Wilkins, Philadelphia
Bay H, Ess A, Tuytelaars T, Van Gool L (2008) Surf: Speeded up robust features. Comput Vis Image Underst (CVIU) 110(3):346–359
Behrens A (2008) Creating panoramic images for bladder fluorescence endoscopy. Acta Polytech J Adv Eng 48(3):50–54
Behrens A, Bommes M, Stehle T, Gross S, Leonhardt S, Aach T (2010) Mosaickingalgorithmus zur schnellen Panoramabilderstellung in der Fluoreszenzendoskopie. In: Bildverarbeitung für die Medizin (BVM), vol 574. Springer, Berlin, pp 41–45
Behrens A, Bommes M, Stehle T, Gross S, Leonhardt S, Aach T (2010) A multi-threaded mosaicking algorithm for fast image composition of fluorescence bladder images. In: Wong KH, Miga MI (eds) Proc. SPIE medical imaging 2010: visualization, image-guided procedures, and modeling (San Diego, USA), vol 7625. SPIE, Bellingham, p 76252S
Behrens A, Stehle T, Gross S, Aach T (2009) Local and global panoramic imaging for fluorescence bladder endoscopy. In: Engineering in medicine and biology society, EMBC 2009. 31th annu int conf of the IEEE, Minneapolis. IEEE, New York, pp 6990–6993
Hamadou A Ben, Soussen C, Blondel W, Daul C, Wolf D (2009) Comparative study of image registration techniques for bladder video-endoscopy. In: SPIE European conference on biomedical optics—novel optical instrumentation for biomedical applications IV, vol 7371. SPIE, Bellingham
Bergen T, Ruthotto S, Munzenmayer C, Rupp S, Paulus D, Winter C (2009) Feature-based real-time endoscopic mosaicking. In: Proc 6th int symp image and signal processing and analysis (ISPA), pp 695–700
Brown M, Szeliski R, Winder S (2005) Multi-image matching using multi-scale oriented patches. In: Proc IEEE computer society conference on computer vision and pattern recognition (CVPR), vol 1, pp 510–517
Cao CG, Milgram P (2000) Disorientation in minimal access surgery: A case study. In: Proc IEA 2000/HFES 2000 congress, vol 4, pp 169–172
Cheng Z, Devarajan D, Radke RJ (2007) Determining vision graphs for distributed camera networks using feature digests. EURASIP J Appl Signal Process 2007(1):220–220
Fischler MA, Bolles RC (1981) Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395
Gross S, Behrens A, Stehle T (2009) Rapid development of video processing algorithms with RealTimeFrame. In: Proc biomedica, pp 217–220
Gross S, Stehle T (2008) RealTimeFrame—a real time processing framework for medical video sequences. Acta Polytech J Adv Eng 48(3):15–19
Halim I, Tavakkolizadeh A (2008) Notes: The next surgical revolution? Int J Surg 6(4):273–276
Hartley R, Kang SB (2007) Parameter-free radial distortion correction with center of distortion estimation. IEEE Trans Pattern Anal Mach Intell 29(8):1309–1321
Höller K, Penne J, Schneider A, Jahn J, Boronat JG, Wittenberg T, Feußner H, Hornegger J (2009) Endoscopic orientation correction. In: Medical image computing and computer-assisted intervention (MICCAI). LNCS, vol 5761. Springer, Berlin, pp 459–466
Hudson MA, Herr HW (1995) Carcinoma in situ of the bladder. J Urol 153:564–572
Hungerhuber E, Stepp H, Kriegmair M, Stief C, Hofstetter A, Hartmann A, Knuechel R, Karl A, Tritschler S, Zaak D (2007) Seven years’ experience with 5-aminolevulinic acid in detection of transitional cell carcinoma of the bladder. Urology 69(2):260–264
Kassouf W, Swanson D, Kamat AM, Leibovici D, Siefker-Radtke A, Munsell MF, Grossman HB, Dinney CP (2006) Partial cystectomy for muscle invasive urothelial carcinoma of the bladder: A contemporary review of the m. d. Anderson cancer center experience. J Urol 175(6):2058–2062
Konen W, Breiderhoff B, Scholz M (2007) Real-time image mosaic for endoscopic video sequences. In: Bildverarbeitung für die Medizin (BVM). Springer, Berlin
Kourogi M, Kurata T, Hoshino J, Muraoka Y (1999) Real-time image mosaicing from a video sequence. In: Proc IEEE international conference on image processing (ICIP), vol 4, pp 133–137
Lowe D (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60:91–110
Mann S, Picard R (1994) Virtual bellows: constructing high quality stills from video. In: Proc IEEE international conference image processing (ICIP), vol 1, pp 363–367
Matouschek E (1987) Urologisch-endoskopische Operationen. Schattauer, Stuttgart
Miranda-Luna R, Daul C, Blondel W, Hernandez-Mier Y, Wolf D, Guillemin F (2008) Mosaicing of bladder endoscopic image sequences: Distortion calibration and registration algorithm. IEEE Trans Biomed Eng 55(2):541–553
Miranda-Luna R, Hernandez-Mier Y, Daul C, Blondel W, Wolf D (2004) Mosaicing of medical video-endoscopic images: data quality improvement and algorithm testing. In: 1st int conf electrical and electronics engineering (ICEEE), pp 530–535
Mountney P, Yang GZ (2008) Soft tissue tracking for minimally invasive surgery: Learning local deformation online. In: Medical image computing and computer-assisted intervention (MICCAI). LNCS, vol 5242. Springer, Berlin, pp 364–372
Mountney P, Yang GZ (2009) Dynamic view expansion for minimal invasive surgery using simultaneous localization and mapping. In: Engineering in medicine and biology society, EMBC 2009. 31th annu int conf of the IEEE, pp 1184–1187
Olijnyk S, Mier YH, Blondel WM, Daul C, Wolf D, Bourg-Heckly G (2007) Combination of panoramic and fluorescence endoscopic images to obtain tumor spatial distribution information useful for bladder cancer detection. In: SPIE European conference on biomedical optics—novel optical instrumentation for biomedical applications III, vol 6631. SPIE, Bellingham, p 66310X
Orozco RE, Martin AA, Murphy WM (1994) Carcinoma in situ of the urinary bladder clues to host involvement in human carcinogenesis. Cancer 74(1):115–122
Rattner D, Kalloo A (2006) Asge/sages working group on natural orifice translumenal endoscopic surgery. Surg Endosc 20(2):329–333
Seshamani S, Lau WW, Hager GD (2006) Real-time endoscopic mosaicking. In: Medical image computing and computer-assisted intervention (MICCAI). LNCS, vol 4190. Springer, Berlin, pp 355–363
Stehle T, Behrens A, Aach T (2008) Enhancement of visual contrast in fluorescence endoscopy. In: IEEE international conference on multimedia and expo
Stehle T, Behrens A, Aach T (2008) Visual enhancement of facial tissue in endoscopy. In: SPIE medical imaging 2008: visualization, image guided procedures, and modeling, vol 6918. SPIE, Bellingham
Stehle T, Wulff J, Behrens A, Gross S, Aach T (2009) Denoising fluorescence endoscopy: A motion-compensated temporal recursive video filter with an optimal minimum mean square error parametrization. In: Proceedings of IEEE international symposium on biomedical imaging (ISBI), pp 314–317
Szeliski R (1996) Video mosaics for virtual environments. IEEE Comput Graph Appl 16(2):22–30
Szeliski R (2006) Image alignment and stitching: A tutorial. Tech rep MSR-TR-2004-92, Microsoft Research
Thormählen T, Broszio H, Meier PN (2002) Three-dimensional endoscopy. In: Falk symposium, medical imaging in gastroenterology and hepatology. Kluwer Academic, Dordrecht, p 124
Tomasi C, Kanade T (1991) Detection and tracking of point features. Tech rep CMU-CS-91-132. Carnegie Mellon University
Triggs B, Mclauchlan PF, Hartley RI, Fitzgibbon AW (2000) Bundle adjustment—a modern synthesis. In: Vision algorithms: theory and practice. LCNS, vol 1883. Springer, Berlin, pp 298–375
Wald D, Reeff M, Székely G, Cattin P, Paulus D (2005) Fließende Überblendung von Endoskopiebildern für die Erstellung eines Mosaiks. In: Bildverarbeitung für die Medizin (BVM). Springer, Berlin, pp 287–291
Wickham JEA (1987) The new surgery. Br Med J 295(6613):1581–1582
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Behrens, A., Bommes, M., Stehle, T. et al. Real-time image composition of bladder mosaics in fluorescence endoscopy. Comput Sci Res Dev 26, 51–64 (2011). https://doi.org/10.1007/s00450-010-0135-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00450-010-0135-z