A New Organ-Following Algorithm Based on Depth-Depth Matching and Simulated Annealing, and Its Experimental Evaluation

Abstract

Medical simulation technology has been rapidly developing, owing to the production of cheap high-performance computers and depth cameras. We are developing a surgery navigation system. In such a system, it is important to track a virtual organ model based on a real organ model. For this purpose, we should examine the coincidence of positions/orientations between two organs at any time. In order to achieve this, we compare depth images of virtual or real organ model. The former is calculated by the z-buffer on a graphics processing unit (GPU) board, while the latter is captured by a depth camera. Then, we search for the same position/orientation between real and virtual organ models by comparing the differences between two depth images. In this paper, we use simulated annealing (SA) to search for the same position/orientation.

1 Introduction

In a surgery navigation system, it is necessary to track a virtual organ model (which is displayed to users) against a real organ model, and to match the positions of the two organ models at any time (Fig. 1). Several papers [1,2,3,4] proposed typical matching methods. Independently, we are developing a navigation system for liver surgery [5,6,7].

Fig. 1.

(a) Polyhedral liver in STL format, and (b) 3D-printed plastic liver.

Our system captures the surface depth image of an organ by using a depth camera such as the Kinect v2, and loads depth-image data on a z-buffer on the PC. Graphics processing unit (GPU) board on the PC generates the surface image of the 3D virtual organ model, which is displayed to users, on another z-buffer on the GPU board. Figure 2 shows the two z-buffers in our system. To match the positions and orientations of the virtual and real organ (liver) models, we denote the position/orientation of a 3D virtual organ object by p = (x, y, z, φ, θ, ψ), where x, y, z are the coordinates in Euclidean 3D space, φ is the roll angle, θ is the pitch, and ψ is the yaw. We also denote p = (p1, p2, p3, p4, p5, p6). In this system, the position/orientation p is searched to decrease the differences between the depth images of the virtual and real liver models in order to match the depth images.

Fig. 2.

Position/orientation of virtual liver model is searched to decrease the difference of depth images of virtual and real liver models.

In this paper, we propose a new depth-depth matching algorithm based on the simulated annealing algorithm, which replaces the steepest descendent algorithm. The steepest descendent algorithm sometimes cannot escape from local minima in a search space with six degrees of freedom. Consequently, during surgery, the system fails to track the virtual organ model to the position/orientation of a real liver model.

Simulated annealing (SA) is a type of randomized search algorithm that was first proposed by S. Kirkpatrick, C. D. Gelatt, M. P. Vecchi et al. in 1983 [8]. This was fortunately rediscovered by V. Cerny in 1985 [9]. SA is a metaheuristic that finds global optimization in a large search space. Steepest descent is a simple search method. At each step, it selects the best neighbor of the current point that is stuck at a local optimum and often cannot find a global optimum. SA selects a neighbor probabilistically and finds the global optimum for a long enough time.

To overcome the drawbacks in our system, we use SA for matching two depth images. In Sect. 2, we describe our simulated annealing algorithm. Then, in Sect. 3, we evaluate the simulated annealing algorithm by changing several parameters, and consequently obtain some significant properties following a real 3D liver, using its virtual 3D liver model expressed by a polyhedron in an STL format. Then, in Sect. 4, we give some concluding remarks.

2 Initial Position/Orientation Identification Under Overlapping Ratio

In this study, we use two kinds of depth images. One is a depth image calculated by z-buffering of the GPU for a virtual model expressed as a polyhedron in STL format. The other is separate depth image captured by the Kinect v2 camera. By overlapping two kinds of depth images, a user can identify the degree of overlapping in real and virtual 3D livers (Fig. 3).

Fig. 3.

In our experiment, we evaluated our motion transcription algorithm by calculating the overlapping ratio (number of green pixels)/(number of blue pixels) × 100. This is how much a virtual 3D liver overlaps its corresponding real 3D liver during surgical navigation. (Color figure online)

By observing the colors on a PC monitor, the user can select several parameters related to the 3D CG (Computer Graphics) virtual environment according to the consistency between the real liver in the camera coordinate system and the virtual liver in the graphics coordinate system. In the image, blue and red pixels are expressed as pixels projected from virtual and real 3D livers along the vertical (Z) axis, respectively. In addition, yellow and green pixels are represented as pixels overlapping the virtual and real 3D livers in the XY plane and XYZ space, respectively. If depth differences between the real and virtual images are less than 1 cm at a pixel, the pixel color is changed from yellow to green. The reason is that many liver surgeons indicated that general liver surgery is not so precise, and that a 1 cm clearance is allowed in a real liver surgery.

In this image-based position/orientation adjustment system, the CG virtual world, captured artificially by the z-buffer of the GPU, should agree with the real world captured by a real depth camera [7]. Furthermore, we always calculate (the number of green pixels)/(the number of blue pixels) × 100 during a surgical operation. Using this ratio, we can understand how much a virtual 3D liver overlaps its corresponding 3D real liver (Fig. 3).

In the next section, we calculate the overlapping ratio during an experimental operation in a surgical room with shadow-less lamps. As a result, we can evaluate our motion transcription algorithm with zero, one, or two shadow-less lamps. The evaluation area is defined as a circle by which we can see the 3D liver through an occluding object. By changing the size of the circle, we can flexibly simulate many types of real liver surgeries.

3 Simulated Annealing Algorithm

During surgery, our system continues to capture depth images of a liver at fixed sampling intervals of Δt. Let t0, t1, t2, …. be the capture times. Then, ti + 1 − ti = Δt. The surgery system must finish matching two depth images within a sampling interval of Δt.

We denote the position/orientation of a 3D virtual organ object by p = (x, y, z, φ, θ, ψ), where x, y, z are the coordinates in Euclidean 3D space, φ is the roll angle, θ is the pitch, and ψ is the yaw. We denote also p = (p1, p2, p3, p4, p5, p6). In this system, the position/orientation p is searched to decrease the difference between two depth images of virtual and real liver models to match the depth images. Below is the SA search algorithm used in our system.

The function random(0,1) returns a random value from zero to one, and the function fitness(p) is the difference between the two depth images in this algorithm. The parameter D is half the width of the area, which is a six-dimensional hypercube with width 2D, in which a new position/orientation p’ is randomly generated. If D increases, the algorithm can choose a more distant neighbor to avoid falling into a local minimum.

4 Experimental Results Obtained by the Proposed Algorithm

Using our new algorithm, we conducted several experiments with/without shadow-less lamps. In this section, we consider cases in which a surgeon rotationally or translationally operates a real 3D lever printed by our 3D printer. This lever is based on an STL polyhedron converted from a segmented DICOM (Digital Imaging and Communication in Medicine) of a patient by a CT (Computed Tomography) scanner. Using our simulated annealing algorithm (explained in the previous section), the virtual liver formatted by the STL polyhedral model automatically follows the real liver perfectly.

In this study, compared with our previous studies [5,6,7], many realistic experiments are conducted not in a laboratory but in a medical surgical operation room. The overall layout of the experimental equipment is shown in Figs. 4 and 5. In order to move the liver, an acrylic plate of 25 cm in length, 25 cm in width, and 2 cm in thickness was placed on the surgical bed, and a real object liver model was placed on the top. The printed 3D liver was created by moving the acrylic plate rotationally or translationally instead of moving the actual liver model. In succession, in order to consider several types of occlusions by the human body against a 3D liver, the liver is covered by a cardboard with a hole of 10 cm in diameter. The surface of this corrugated board was colored in light orange using a coloring spray to make it look like normal human skin. Meanwhile, our Kinect v2 camera can change the distance from the 3D real liver. The Kinect is placed horizontally with respect to the operating table at a height of 84 cm from the bottom of the actual liver model. The Kinect is fixed by attaching metal fittings to a metal rod.

Fig. 4.

In our navigation study, we evaluated our motion transcription algorithm in a real surgical room with zero, one, and two shadow-less lamps. In our navigation, a 3D liver is always obstructed by a cardboard box with a circle. Through the circle, the camera can capture a part of the 3D liver. In another experiment, we evaluated the motion transcription algorithm by changing the size of the circle.

Fig. 5.

Our surgical navigation is used in the actual operation room of a hospital. Capturing an organ such as a liver is stable because the camera is completely fixed to the surgical bed.

By using the matching algorithm based on real and virtual depth images against real and virtual livers (explained in the next section), initial positions/orientations between the real and virtual livers are coincident with each other. Then, a real liver model placed on an acrylic board was covered with corrugated cardboard and subjected to parallel movement and rotational movement within a corrugated cardboard box. Before the movement, the actual liver model was installed so that it could be seen through the entire hole of the cardboard. Thus, the range visible from the hole of the cardboard remained at least half after the translation movement. The offset value was 10 mm for the real and virtual depth images against the 3D real liver and its polyhedron in STL format in our surgical navigator. The amount of movement was 5 cm along the Y-axis during parallel movement, and the rotation movement was 45º around the Z-axis. Each coordinate axis is shown in Fig. 4.

First, we explain several differences between our current and previous experiments [5,6,7]. The main difference is that we used a real surgical room with one or two shadow-less lamps (Figs. 4 and 5). A minor difference is that a 3D real liver is occluded by a cardboard with a hole of 10 cm in diameter. All previous experimental results were obtained in our research room using daylight and/or fluorescent lamps for a 3D liver which is not covered by any another object. In the following paragraphs, for pairs of translation and rotation movements of the 3D liver with some occlusion, we introduce experiments without a lamp, with one lamp, and with two lamps.

4.1 Surgical Experience Without a Lamp

In the surgical operation room without any shadow-less light, several experiments were carried out. These are shown in Fig. 6. Rotating and translating a 3D real liver is recorded using Kinect Studio in several types of windows. As shown in Fig. 6, the black window indicates a real liver in the real world, and the white window is the virtual liver in the virtual world. The bottom-right window is an RGB color image that corresponds to the depth image. All experiments were performed using our simulated annealing algorithm based on data taken in the black and white windows. The accuracy of the algorithm is evaluated by the overlapping ratio from the initial position/orientation to the target orientation (Fig. 7).

Fig. 6.

Actual red liver obstructed by a box with a circle window, followed by its virtual blue liver formed by STL polyhedron in surgical operation room without a shadow-less lamp. In each figure, the black window indicates a real liver in the real world, and the white window is its virtual liver in the virtual world. The bottom-right window is the RGB color image that corresponds to the depth image. (a), (b) Liver is rotationally moved before and after surgical operation, respectively. (c), (d) Liver is translationally moved before and after surgical operation, respectively (Color figure online).

Fig. 7.

Our simulated annealing algorithm. Overlapping ratio is changed during surgical operation if a human moves a 3D liver rotationally and translationally. In a surgical room, there is no shadow-less lamp. (a), (b) Δt = 500 and D = 1. (c), (d) Δt = 400 and D = 1.

Fig. 8.

A real red liver obstructed by a box with a circle window, followed by its virtual blue liver formed by STL polyhedron in a surgical operation room with one shadow-less lamp. In each figure, the black window indicates a real liver in the real world, and the white window is its virtual liver in the virtual world. Bottom-right window is the RGB color image that corresponds to the depth image (a), (b) Liver is rotationally moved before and after a surgical operation, respectively. (c), (d) Liver is translationally moved before and after a surgical operation, respectively.

After the real and virtual livers were initially overlapped using our software (explained in the previous section), a human operated the real 3D liver rotationally. Before rotating (Fig. 6(a)), the overlapping ratio was 99% in Fig. 7(a). When moving rotationally (Fig. 6(b)), the overlapping ratio was always maintained at least 96%. Then, after movement, the ratio became 100% in Fig. 7(a). The graph shows the transition of the overlapping ratio from the start to the end during a surgical operation. With regard to the simulated annealing method liver navigator during traveling, there was no significant decrease in the overlapping ratio.

After the real and virtual livers initially overlapped each other using our software explained in the previous section, a human operator controlled the real 3D liver translationally (Fig. 6(c), (d)). Unfortunately, compared with the rotational movement, the overlapping ratio from the start to the end was not as effective, especially in a middle of the interval (Fig. 7(b)). The main reason is that a real 3D liver moves out of the circle that the Kinect camera can capture. However in a real surgery, such a situation is rare. Even in such a difficult situation, the ratio finally becomes over 80% after the operation.

As mentioned in the previous section, our simulated annealing algorithm has several parameters in which search space and speed can be flexibly changed. In this study, we add two parameters: Δt and D. In our algorithm, Δt leads the size of the search space in the while loop, and D leads the size of the movement within the search space. If Δt becomes larger, the search space becomes wider. If D becomes larger, the density of the search decreases. As illustrated in Fig. 7(c) and (d), we describe the overlapping ratio during a surgical operation rotationally and translationally, respectively, when Δt is changed from 500 to 400 in our algorithm. Furthermore, in Fig. 9 (c) and (d), we describe the overlapping ratio during a surgical operation rotationally and translationally, respectively, when D is changed from 1 to 2 in our algorithm. The difference in the overlapping ratio is quite small, even though two parameters are changed. As a result, we can see that our simulated annealing algorithm achieves stable motion when tracking a real 3D liver to its virtual 3D liver.

Fig. 9.

Our simulated annealing algorithm. Overlapping ratio is changed during surgical operation if a human moves a 3D liver rotationally and translationally. In the surgical room, there is no shadowless lamp. (a), (b) Δt = 500 and D = 1. (c), (d) Δt = 500 and D = 2.

4.2 Surgical Experience with One Lamp

In a surgical room with one shadow-less light, we conducted several experiments, as illustrated in Fig. 8. The translating and rotating movements of the real liver were taken with Kinect Studio. The accuracy as evaluated by using the overlapping ratio from the start to the end of the rotation and translation movements (Fig. 10). After the real and virtual livers initially overlapped each other in our software (explained in the previous section), a human operator controlled the real 3D liver rotationally. Before rotating (Fig. 8(a)), the overlapping ratio was 100%, as shown in Fig. 10(a). When moved (Fig. 8(b)), the overlapping ratio was always higher than 95%; and after movement, it became 98%, as shown in Fig. 10(a). The figure is a graph showing the transition of the overlapping ratio from the end of the initial alignment to the end of the movement. With regard to the simulated annealing method liver navigator during traveling, there was no significant decrease in the overlapping ratio.

Fig. 10.

Our simulated annealing algorithm. Overlapping ratio is changed during surgical operation if a human moves a 3D liver rotationally and translationally. In the surgical room, we used one shadow-less lamp. (a), (b) Δt = 500 and D = 1. (c), (d) Δt = 400 and D = 1.

After the real and virtual livers initially overlapped each other in our software (explained in the previous section), a human operator controlled the real 3D liver translationally (Fig. 8(c), (d)). Unfortunately, compared with the rotational movement, the overlapping ratio from the start to the end is not as effective (Fig. 10(b)). However, compared with the case that did not use a lamp, even in a middle of an interval, the damage was quite

Moreover, as illustrated in Fig. 10(c) and (d), the overlapping ratio changed during a human operation when Δt was changed from 500 to 400 in our algorithm.

In this case, a human operator moved the real 3D liver rotationally and translationally. However, the difference between the two parameters with regard to the overlapping ratio is quite small (Fig. 11(c) and (d)). Meanwhile, using our algorithm, we described how the overlapping ratio changed during a human operation if D was changed from 1 to 2. The difference between the two parameters with regard to the overlapping ratio is also small (Fig. 10(c), (d) and Fig. 11(c), (d)). As a result, our simulated annealing algorithm achieves stable motion when tracking a real 3D liver to its virtual 3D liver.

Fig. 11.

Our simulated annealing algorithm. Overlapping ratio is changed during surgical operation if a human moves a 3D liver rotationally and translationally. In the surgical room, we used one shadow-less lamp. (a), (b) Δt = 500 and D = 1. (c), (d) Δt = 400 and D = 1.

4.3 Surgical Experience with Two Lamps

In this section, we first evaluate our simulated annealing algorithm for a 3D real liver rotated by a human. With the help of the algorithm, the motion of the real liver is coincident with that of its virtual liver. Before moving rotationally, the overlapping ratio is 100%, as shown in Fig. 12(a). When moved (Fig. 12(b)), the overlapping ratio is always maintained by high scores and also the variation is quite small. Then, after movement, the ratio became 100%, as shown in Fig. 13(a). This resulted in an increase in the overlapping ratio after the initial position alignment was complete.

Fig. 12.

A real red liver obstructed by a box with a circle window, followed by its virtual blue liver formed by STL polyhedron in a surgical operation room with two shadow-less lamps. (a), (b) Liver is rotationally moved before and after a surgical operation, respectively. (c), (d) Liver is translationally moved before and after a surgical operation, respectively. (Color figure online)

Fig. 13.

Our simulated annealing algorithm. Overlapping ratio is changed during surgical operation if a human moves a 3D liver rotationally and translationally. In the surgical room, we used two shadow-less lamps. (a), (b) Δt = 500 and D = 1. (c), (d) Δt = 400 and D = 1.

Figure 13(a) is a graph showing the transition of the overlapping ratio from the end of the initial alignment to the end of the movement. A decrease in the overlapping ratio is observed for each movement, and the overlapping ratio decreased by approximately 10% in the middle, but after the movement, the overlapping ratio showed a higher coincidence ratio than after the completion of the initial position adjustment. From the beginning, it showed a considerably high concordance rate.

A human controlled a real liver model translationally, and then its virtual liver followed the real liver model. As shown in Fig. 13(b), the ratio is as high as 100% before moving (Fig. 12(c)). When moved translationally (Fig. 12(d)), the overlapping ratio fluctuated over 80%; and after the movement, it became 90%, as shown in Fig. 13(b). Although the overlapping ratio decreases, the result always shows a high overlapping ratio that exceeds 80%. Figure 13(b) is a graph showing the transition of the overlapping ratio from the end of the initial alignment to the end of the movement. The result shows that the overlapping ratio is low on average compared with when the shadow-less light is not lit.

Moreover, as illustrated in Fig. 13(c) and (d), the overlapping ratio changed during human operation if Δt was changed from 500 to 400 in our algorithm. In this case, a human operator controlled the real 3D liver rotationally and translationally. The difference concerning the overlapping ratio is quite small (Fig. 13(c) and (d)). On the other hand, the overlapping ratio is changed during human operation if D is changed from 1 to 2 in our algorithm. The difference in the overlapping ratio is also small (Fig. 14(c) and (d)). As a result, our simulated annealing algorithm achieves stable motion when tracking a real 3D liver to its virtual 3D liver.

Fig. 14.

Our simulated annealing algorithm. Overlapping ratio is changed during surgical operation if a human moves a 3D liver rotationally and translationally. In the surgical room, we used two shadow-less lamps. (a), (b) Δt = 500 and D = 1. (c), (d) Δt = 400 and D = 1.

5 Conclusions

In this study, we proposed a simulated annealing algorithm and checked its usefulness by conducting several kinds of experiments in a real surgical room with shadow-less lamps. In addition, to simulate a practical surgery operation, we printed a 3D liver using a 3D printer based on segmented DICOM data captured for a patient by a CT scanner. Then, we put the 3D liver on an acrylic board of a surgical bed, which was covered with corrugated cardboard and subjected to translational movement and rotational movement within a corrugated cardboard box. By changing the circle size, we could create many types of occlusions. According to many experimental results of the simulated annealing algorithm, whose parameters were flexibly changed, the algorithm was stable for the motion transcription method from a real 3D liver to its virtual 3D liver. Therefore, this algorithm can be used in our liver surgical navigation.