Elsevier

Medical Image Analysis

Volume 11, Issue 6, December 2007, Pages 526-539
Medical Image Analysis

Robust registration procedures for endoscopic imaging

https://doi.org/10.1016/j.media.2007.04.006Get rights and content

Abstract

This paper presents a robust algorithm for calibration and system registration of endoscopic imaging devices. The system registration allows us to map accurately each point in the world coordinate system into the endoscope image and vice versa to obtain the world line of sight for each image pixel.

The key point of our system is a robust linear algorithm based on singular value decomposition (SVD) for estimating simultaneously two unknown coordinate transformations. We show that our algorithm is superior in terms of robustness and computing efficiency to iterative procedures based on Levenberg–Marquardt optimization or on quaternion approaches. The algorithm does not require the calibration pattern to be tracked.

Experimental results and simulations verify the robustness and usefulness of our approach. They give an accuracy of less than 0.7 mm and a success rate >99%. We apply the calibrated endoscope to the neurosurgical relevant case of red out, where in spite of the complete loss of vision the surgeon gets visual aids in the endoscope image at the actual position, allowing him/her to manoeuvre a coagulation fibre into the right position.

Finally, we outline how our registration algorithm can be used also for standard registration applications (establish the mapping between two sets of points). We propose our algorithm as a linear, non-iterative algorithm also for projective transformations and for 2D–3D-mappings. Thus, it can be seen as a generalization of the well-known Umeyama registration algorithm.

Introduction

Image processing and image analysis play an important role in advanced surgery (CAS, computer aided surgery). Much work is devoted to 3D-reconstruction from CT-, MRI- or other volume data or to 3D-registration between different image modalities and the actual scene in the operating theatre. 3D-registration is an important prerequisite for navigation support systems which allow, for example, the surgeon to follow a predefined path during a surgical intervention.

Visual navigation in neuroendoscopy is a special kind of navigation employing only endoscopic images for navigational purposes. This technique can be utilized alone or in addition to conventional navigation systems based on magnetic resonance imaging (MRI) or computer tomography (CT) (Dorward et al., 1999, Rhode et al., 1998, Rhoten et al., 1997).

There is a large body of work in the area of virtual endoscopy (Bartz, 2005, Lemke et al., 2004) where radiological data (CT or MRI) are used to create virtual endoscopic views or movies, but those systems cannot react directly to observations made or situations happening in the operation theatre. An example is the case of red out, where a complete loss of vision in the endoscopic view due to bleeding occurs. With our system it is possible to give the surgeon precise visual aids in the endoscopic view to control a coagulation fibre (see Section 5.2) and stop the bleeding.

Digital image processing and image overlay within (real) endoscopic images, with the aim to provide the surgeon with navigational aids and/or measuring/tracking 3D-structures, has been addressed so far only by relatively few authors. Kosaka et al., 2000, Akatsuka et al., 2000 and later also Kawamata et al., 2002, Shahidi et al., 2002 describe augmented reality systems in which previously recorded CT- or MRI-data (e.g. 3D-data of a tumour) can be overlaid to the actual endoscopic image. Camera calibration and system registration (physical-to-image-space registration) are vital building blocks for those systems. Koppel et al., 2002, Koppel et al., 2004 describe real-time tracking in endoscopic images which is used to estimate the “up” vector of the camera and to adjust the display for the surgeon. Camera calibration and measurement of the endoscope 3D-position is not part of the system.

The main roadblocks which prevent a wider range of applications for image processing and 3D-measurements within endoscopic images lie in our opinion (a) in the difficulty to provide robust algorithms for calibration and extraction of useful 3D-information from multiple images of a moving camera, (b) in strong distortions caused by the wide-angle endoscope lens system and (c) in close-to-real-time requirements for any of the image processing tasks.

We developed a new and robust algorithm for topic (a), the calibration and physical-to-image-space registration (system registration) of the endoscope. In this paper we describe this algorithm in detail and compare it to other algorithms and connect it to the somewhat different standard registration approaches, with the aim to share this knowledge with a wider audience and to bring endoscopic registration to more routine use in the operating theatre.

Topics (b) and (c) have been addressed in Dey et al., 2002, Kawamata et al., 2002, Shahidi et al., 2002 as well as by us (Konen et al., 1997, Konen et al., 1998). We plan to describe more of our recent findings, laid out also in a patent application (Konen et al., 2006) in a follow-up paper (Konen et al., in preparation).

Medical image registration is a very active research field due to its broad applicability to many imaging modalities; a recent overview is given by Hill et al., 2001, Maintz and Viergever, 1998. The standard task of registration (or alignment) is to find the transformation T which transforms a set of points pi as well as possible into a corresponding set of points qi. In the case of rigid-body transformations the problem can be solved with methods from linear algebra, as it was first done by Green (1952), while Schoenemann (1966) was the first to use singular value decomposition (SVD) for this problem. Later Arun et al. (1987) rediscovered this method. Independently Farrell et al. (1966) provided a solution that guaranteed a proper rotation, an idea which was rediscovered by Umeyama (1991). If the set of points is large, it may be too complex to set up the correspondence mapping manually. An automated method is the iterative closest point (ICP) algorithm (Besl and McKay, 1992, Zhang, 1994), which can also be used to match a set of points to a free-form curve.

The problem of iterative algorithms is their starting value dependence which might result in a solution far away from the true solution. Koppel et al. (2002) describe a non-iterative, linear SVD-based solution, however for a somewhat different endoscope application where no positioning measurement system for the endoscope is used. With such a positioning system, the system registration contains 4 or 5 coordinate transformations, where 2 or 3 of them are unknown and have to be estimated simultaneously. The works of (Kosaka et al., 2000, Akatsuka et al., 2000, Kawamata et al., 2002) incorporate such a system registration, but do not describe which methods (iterative nonlinear vs. direct linear) are used. Shahidi et al. (2002) use a second tracking unit for a separately registered calibration pattern. Schwald and Seibert (2004) describe a linear method for solving the registration with four coordinate systems, which requires however a nonlinear quaternion mapping to ensure orthogonality of the rotation matrices. In this paper we describe a new direct linear method for system registration and compare it to other iterative possibilities, including a quaternion approach.

This paper is organized as follows: In Section 2 we will first present our system setup and formulate the general registration problem for endoscopic imaging. We will then develop in Section 3 different algorithms for solving the physical-to-image-space registration (some mathematical details are deferred to Appendix A Quaternions, Appendix B The structure of matrix C). Section 4 presents results both from simulations and real registration experiments to verify the robustness and usefulness of our method, while Section 5 discusses the applicability to other registration task and shows some endoscopy medical applications.

Section snippets

System setup

The rigid endoscope (outer diameter 5.9 mm, Camaert/Wolf GmbH) used in this work consists of a circular tube (6 mm diameter) where a colour CCD-camera at the rear end captures the image from the tip of the endoscope through a special lens system (distance tip–rear end: 380 mm). We developed a special device (see Konen et al. (1997) and Fig. 1) mounted on the shaft of the endoscope which holds three infrared LEDs (light-emitting diodes).

The positions of the LEDs are measured continuously by the

Methods

A robust solution of Eq. (6) which works well in practical surroundings is not an easy task due to the following factors

  • many variables (2 × 12 = 24 variables, 9 rotation matrix elements and 3 translation vector elements in both DO and DC);

  • nonlinear optimization, since the elements of DO and DC are coupled;

  • constrained optimization problem, since there are only 2 × 6 = 12 independent degrees of freedom within the 24 variables.

We show in Section 3.1 how to decompose Eq. (6) from a nonlinear problem with

Comparing the algorithms

We want to compare the 3 different algorithms of Sections 3.2 Iterative nonlinear optimization, 3.3 Iterative quaternion optimization, 3.4 Direct linear algorithm with respect to the following factors:

  • robustness against start value variation;

  • robustness against measurement noise;

  • quality improvement with more measurements;

  • performance.

We name the three algorithms of Sections 3.2 Iterative nonlinear optimization, 3.3 Iterative quaternion optimization, 3.4 Direct linear algorithm as “iterative”,

Applicability for standard registration

Is the direct linear algorithm also of use for standard registration tasks?

In standard registration procedures (cf. Section 2.4) we have two sets of points x(i) and y(i), i = 1, …, n and seek a transformation TT which minimizes iT(x(i))-y(i). If TT is drawn from the set of affine mappings, we have the case of Procrustes or rigid-body transformation, where the well-known algorithms of Schoenemann (1966) or Farrell et al. (1966), later rediscovered by Arun et al., 1987, Umeyama, 1991, provide a

Conclusion

We introduced a new method for the system registration of surgical devices, which provide 2D-projections of the 3D-world seen by the surgeon during interventions, such as endoscopes. The registration method is markerless, i.e. it does not need fiducial points (with known extrinsic 3D position in the operating theatre), but only observations of a passive calibration pattern with known (intrinsic) geometry. Shahidi et al. (2002) solve the system registration by placing a second tracker on the

Acknowledgements

We thank the anonymous reviewers as well as G. Plassmann and H. Westenberger for helpful revision comments.

References (33)

  • J.B.A. Maintz et al.

    A survey of medical image registration

    Med. Image Anal.

    (1998)
  • Akatsuka, Y., Shibasaki, T., Saito, A., Kosaka, A., Matsuzaki, H., Asano, T., Furuhashi, Y., 2000. Navigation system...
  • K.S. Arun et al.

    Least-squares fitting of two 3D point sets

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1987)
  • D. Bartz

    Virtual endoscopy in research and clinical practice

    Comp. Graph. Forum

    (2005)
  • P.J. Besl et al.

    A method for registration of 3-D shapes

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1992)
  • D. Dey et al.

    Automatic fusion of freehand endoscopic brain images to three-dimensional surfaces: creating stereoscopic panoramas

    IEEE Trans. Med. Imaging

    (2002)
  • N.L. Dorward et al.

    Interactive image-guided neuroendoscopy: development and early clinical experience

    Min. Invasive Neurosurg.

    (1999)
  • J.L. Farrell et al.

    A least squares estimate of satellite attitude, Solution 65-1

    SIAM Rev.

    (1966)
  • B.F. Green

    The orthogonal approximation of an oblique structure in factor analysis

    Psychometrika

    (1952)
  • D. Hill et al.

    Medical image registration

    Phys. Med. Biol.

    (2001)
  • T. Kawamata et al.

    Endoscopic augmented reality navigation for endonasal transsphenoidal surgery to treat pituitary tumors: technical note

    Neurosurgery

    (2002)
  • Konen, W., Scholz, M., Tombrock, S., Tölg, S., Brauckmann, M., Schwarz, A., Adams, L., 1997. An image-based navigation...
  • W. Konen et al.

    The VN-project: endoscopic image processing for neurosurgery

    Computer Aided Surg.

    (1998)
  • Konen, W., Scholz, M., Tombrock, S., 2006. Verfahren zur endoskopischen Navigation und zur Eichung von...
  • Konen, W, Scholz, M, Tombrock, S., in preparation. Robust real-time tracking and 3D-measurement in neuroendoscopic...
  • Koppel, D., Wang, Y.-F., Lee, H., 2002. Viewing enhancement in video-endoscopy. In: Proceedings of the Workshop on...
  • Cited by (0)

    View full text