Robust registration procedures for endoscopic imaging
Introduction
Image processing and image analysis play an important role in advanced surgery (CAS, computer aided surgery). Much work is devoted to 3D-reconstruction from CT-, MRI- or other volume data or to 3D-registration between different image modalities and the actual scene in the operating theatre. 3D-registration is an important prerequisite for navigation support systems which allow, for example, the surgeon to follow a predefined path during a surgical intervention.
Visual navigation in neuroendoscopy is a special kind of navigation employing only endoscopic images for navigational purposes. This technique can be utilized alone or in addition to conventional navigation systems based on magnetic resonance imaging (MRI) or computer tomography (CT) (Dorward et al., 1999, Rhode et al., 1998, Rhoten et al., 1997).
There is a large body of work in the area of virtual endoscopy (Bartz, 2005, Lemke et al., 2004) where radiological data (CT or MRI) are used to create virtual endoscopic views or movies, but those systems cannot react directly to observations made or situations happening in the operation theatre. An example is the case of red out, where a complete loss of vision in the endoscopic view due to bleeding occurs. With our system it is possible to give the surgeon precise visual aids in the endoscopic view to control a coagulation fibre (see Section 5.2) and stop the bleeding.
Digital image processing and image overlay within (real) endoscopic images, with the aim to provide the surgeon with navigational aids and/or measuring/tracking 3D-structures, has been addressed so far only by relatively few authors. Kosaka et al., 2000, Akatsuka et al., 2000 and later also Kawamata et al., 2002, Shahidi et al., 2002 describe augmented reality systems in which previously recorded CT- or MRI-data (e.g. 3D-data of a tumour) can be overlaid to the actual endoscopic image. Camera calibration and system registration (physical-to-image-space registration) are vital building blocks for those systems. Koppel et al., 2002, Koppel et al., 2004 describe real-time tracking in endoscopic images which is used to estimate the “up” vector of the camera and to adjust the display for the surgeon. Camera calibration and measurement of the endoscope 3D-position is not part of the system.
The main roadblocks which prevent a wider range of applications for image processing and 3D-measurements within endoscopic images lie in our opinion (a) in the difficulty to provide robust algorithms for calibration and extraction of useful 3D-information from multiple images of a moving camera, (b) in strong distortions caused by the wide-angle endoscope lens system and (c) in close-to-real-time requirements for any of the image processing tasks.
We developed a new and robust algorithm for topic (a), the calibration and physical-to-image-space registration (system registration) of the endoscope. In this paper we describe this algorithm in detail and compare it to other algorithms and connect it to the somewhat different standard registration approaches, with the aim to share this knowledge with a wider audience and to bring endoscopic registration to more routine use in the operating theatre.
Topics (b) and (c) have been addressed in Dey et al., 2002, Kawamata et al., 2002, Shahidi et al., 2002 as well as by us (Konen et al., 1997, Konen et al., 1998). We plan to describe more of our recent findings, laid out also in a patent application (Konen et al., 2006) in a follow-up paper (Konen et al., in preparation).
Medical image registration is a very active research field due to its broad applicability to many imaging modalities; a recent overview is given by Hill et al., 2001, Maintz and Viergever, 1998. The standard task of registration (or alignment) is to find the transformation T which transforms a set of points pi as well as possible into a corresponding set of points qi. In the case of rigid-body transformations the problem can be solved with methods from linear algebra, as it was first done by Green (1952), while Schoenemann (1966) was the first to use singular value decomposition (SVD) for this problem. Later Arun et al. (1987) rediscovered this method. Independently Farrell et al. (1966) provided a solution that guaranteed a proper rotation, an idea which was rediscovered by Umeyama (1991). If the set of points is large, it may be too complex to set up the correspondence mapping manually. An automated method is the iterative closest point (ICP) algorithm (Besl and McKay, 1992, Zhang, 1994), which can also be used to match a set of points to a free-form curve.
The problem of iterative algorithms is their starting value dependence which might result in a solution far away from the true solution. Koppel et al. (2002) describe a non-iterative, linear SVD-based solution, however for a somewhat different endoscope application where no positioning measurement system for the endoscope is used. With such a positioning system, the system registration contains 4 or 5 coordinate transformations, where 2 or 3 of them are unknown and have to be estimated simultaneously. The works of (Kosaka et al., 2000, Akatsuka et al., 2000, Kawamata et al., 2002) incorporate such a system registration, but do not describe which methods (iterative nonlinear vs. direct linear) are used. Shahidi et al. (2002) use a second tracking unit for a separately registered calibration pattern. Schwald and Seibert (2004) describe a linear method for solving the registration with four coordinate systems, which requires however a nonlinear quaternion mapping to ensure orthogonality of the rotation matrices. In this paper we describe a new direct linear method for system registration and compare it to other iterative possibilities, including a quaternion approach.
This paper is organized as follows: In Section 2 we will first present our system setup and formulate the general registration problem for endoscopic imaging. We will then develop in Section 3 different algorithms for solving the physical-to-image-space registration (some mathematical details are deferred to Appendix A Quaternions, Appendix B The structure of matrix C). Section 4 presents results both from simulations and real registration experiments to verify the robustness and usefulness of our method, while Section 5 discusses the applicability to other registration task and shows some endoscopy medical applications.
Section snippets
System setup
The rigid endoscope (outer diameter 5.9 mm, Camaert/Wolf GmbH) used in this work consists of a circular tube (6 mm diameter) where a colour CCD-camera at the rear end captures the image from the tip of the endoscope through a special lens system (distance tip–rear end: 380 mm). We developed a special device (see Konen et al. (1997) and Fig. 1) mounted on the shaft of the endoscope which holds three infrared LEDs (light-emitting diodes).
The positions of the LEDs are measured continuously by the
Methods
A robust solution of Eq. (6) which works well in practical surroundings is not an easy task due to the following factors
- •
many variables (2 × 12 = 24 variables, 9 rotation matrix elements and 3 translation vector elements in both DO and DC);
- •
nonlinear optimization, since the elements of DO and DC are coupled;
- •
constrained optimization problem, since there are only 2 × 6 = 12 independent degrees of freedom within the 24 variables.
We show in Section 3.1 how to decompose Eq. (6) from a nonlinear problem with
Comparing the algorithms
We want to compare the 3 different algorithms of Sections 3.2 Iterative nonlinear optimization, 3.3 Iterative quaternion optimization, 3.4 Direct linear algorithm with respect to the following factors:
- •
robustness against start value variation;
- •
robustness against measurement noise;
- •
quality improvement with more measurements;
- •
performance.
We name the three algorithms of Sections 3.2 Iterative nonlinear optimization, 3.3 Iterative quaternion optimization, 3.4 Direct linear algorithm as “iterative”,
Applicability for standard registration
Is the direct linear algorithm also of use for standard registration tasks?
In standard registration procedures (cf. Section 2.4) we have two sets of points x(i) and y(i), i = 1, …, n and seek a transformation which minimizes . If is drawn from the set of affine mappings, we have the case of Procrustes or rigid-body transformation, where the well-known algorithms of Schoenemann (1966) or Farrell et al. (1966), later rediscovered by Arun et al., 1987, Umeyama, 1991, provide a
Conclusion
We introduced a new method for the system registration of surgical devices, which provide 2D-projections of the 3D-world seen by the surgeon during interventions, such as endoscopes. The registration method is markerless, i.e. it does not need fiducial points (with known extrinsic 3D position in the operating theatre), but only observations of a passive calibration pattern with known (intrinsic) geometry. Shahidi et al. (2002) solve the system registration by placing a second tracker on the
Acknowledgements
We thank the anonymous reviewers as well as G. Plassmann and H. Westenberger for helpful revision comments.
References (33)
- et al.
A survey of medical image registration
Med. Image Anal.
(1998) - Akatsuka, Y., Shibasaki, T., Saito, A., Kosaka, A., Matsuzaki, H., Asano, T., Furuhashi, Y., 2000. Navigation system...
- et al.
Least-squares fitting of two 3D point sets
IEEE Trans. Pattern Anal. Mach. Intell.
(1987) Virtual endoscopy in research and clinical practice
Comp. Graph. Forum
(2005)- et al.
A method for registration of 3-D shapes
IEEE Trans. Pattern Anal. Mach. Intell.
(1992) - et al.
Automatic fusion of freehand endoscopic brain images to three-dimensional surfaces: creating stereoscopic panoramas
IEEE Trans. Med. Imaging
(2002) - et al.
Interactive image-guided neuroendoscopy: development and early clinical experience
Min. Invasive Neurosurg.
(1999) - et al.
A least squares estimate of satellite attitude, Solution 65-1
SIAM Rev.
(1966) The orthogonal approximation of an oblique structure in factor analysis
Psychometrika
(1952)- et al.
Medical image registration
Phys. Med. Biol.
(2001)