Algorithm Experimental Evaluation for an Occluded Liver with/without Shadow-Less Lamps and Invisible Light Filter in a Surgical Room

Abstract

In this paper, we investigate our proposed motion transcription algorithm that helps develop a virtual liver model from a real liver; the virtual liver is designed using the STL-polyhedron in an experimental surgical room. If we do not use any shadow-less lamps in the room, the algorithm correctly copies the translational/rotational motions from the real liver to the virtual liver. However, if we use one or two shadow-less lamps during the surgery, the copy quality decreases significantly, and consequently, our surgical navigation is sometime dammed. To overcome this problem, we attempted to overlap the shadow-less lamps with light-blocking filters. The purpose of using the light-blocking filter is to eliminate the unsuitable wavelength of shadow-less light for our camera system, the Microsoft Kinect v2. It is equipped with three types of cameras: an RGB camera, an infrared camera, and an infrared laser projector. The Microsoft Kinect v2 detects the infrared rays reflected from the object in front of it, estimates all the depth distances within all the pixels, and tracks 3D objects with several shapes. As a result, using one of the two light blocking filters, the camera Microsoft Kinect v2 captures all depths at all pixels stably on a 3D liver model, and consequently, our motion transcription algorithm plays an active role in the experimental procedure.

1 Introduction

In the past two or more decades, many fast and robust approaches have been developed for surface registration [1,2,3,4,5,6,7]. However, these approaches adopt three-dimensional (3D) point-point matching and irregular (x, y, z-axes) 3D matching. These approaches are quite time consuming, and consequently, they are not suitable for surgical navigation in real time. To overcome this drawback, we propose an approach where the movements of a real liver in an operating room are replicated by the movements of a virtual liver simulated on a PC. This motion transfer function is important for developing a surgical navigation system.

Our algorithm involves two types of parallel processing: depth difference calculation for all pixels by z-buffering with a graphics processing unit (GPU) and selecting the best neighbor from a large number of neighbors using the multicores on a GPU. In the former procedure, we use two-dimensional (2D) depth-depth matching between a real depth image in the real world and its virtual depth image, which is developed automatically by z-buffering in the virtual world. This type of matching comprises regularly arranged one-dimensional (z-axis = depth) matching. Therefore, all the matches for a vast number of pixels can be calculated in parallel by z-buffering on the GPU. In the latter procedure, we perform a rapid search for the best neighbor whose difference between real and virtual depth images is to be the minimum/medium/average/maximum. The real depth image is captured by the camera, and the virtual depth image is calculated by the z-buffering. The best neighbor is selected within a large number of neighbors based on six degrees of freedom (DOFs) that determine the translational/rotational movements in space in parallel using the multicores on the GPU. Thus, we successively find a position/posture that is most likely to exist based on various differences between the depth image captured by a Kinect v2 in the real world and the image captured by the GPU in the z-buffer in the virtual world. To eliminate the local minima, we use two types of randomization processes in the steepest descent method.

A practical system for surgical navigation requires several basic functions. Therefore, we are now developing several types of software, hardware, sensing equipment, and control schema. In this paper, firstly, we explain our navigation system with many software/hardware/sensing items in Sect. 2. Then, we test our algorithm for a liver occluded by a cover in a real surgical room both with and without the shadow-less lamps and/or two light blocking filters in Sect. 3. Thereafter, in Sect. 4, we conclude by presenting the types of experimental results obtained.

2 Our Motion Transcription Algorithm

From the past three years, we have been developing a surgical navigation system [8,9,10]. In this system, we pay attention to an automatic machine tracking a real organ by using its virtual organ model as shown in Fig. 1. For this purpose, we theoretically and practically investigated the two-dimensional (2D) depth-depth matching between a real depth image in a real environment, which is captured by a depth camera, such as Kinect v2, and its virtual depth image in the graphical environment, which is captured automatically by z-buffering of the GPU (Fig. 2).

Fig. 1.

(a) A polyhedral liver with STL format, and (b) its printed plastic liver.

Fig. 2.

By matching the real and virtual depth images, we can find a better position/orientation of virtual liver against that of the real liver.

Our algorithm performs a rapid search for the best neighbor, where the difference between the real and virtual depths is the minimum/medium/average/maximum compared with a large number of neighbors based on six degrees of freedom (DOFs) translation/rotation movements in space in parallel using the multicores on the GPU (Fig. 3). Thus, we successively find a position/posture that is most likely to exist based on various differences between the depth image captured by a Kinect v2 in the real world and the image captured by the GPU in the z-buffer in the virtual world. To eliminate the local minima, we use two types of randomization processes in the steepest descent method (Fig. 4).

Fig. 3.

In our algorithm, we randomly select a set of pixels whose number is N in each image, and evaluate the minimum, median, or average of difference distribution between their master and slave depths. Furthermore, we select the minimum, median, or average of evaluation values in many images whose number is M. The kinds of randomizations escape from a local minimum among motion space in our 2D depth-depth matching algorithms.

Fig. 4.

(a) The flowchart of our depth-depth matching algorithm, (b) 24 neighbor candidates whose dimension is 6 represented by 3 translational degree-of-freedoms and 3 rotational ones.

Finally, in our navigation study, a 3D computer-generated (CG) virtual environment that is controlled by the OpenGL platform on the GPU is theoretically adjusted for a 3D real camera environment, which is controlled by the Kinect SDK and Kinect Studio API. Thus, the virtual 3D coordinate system is roughly consistent with the real 3D world. Furthermore, a user can make out the differences between the depth images of the real and virtual livers in the 3D real and virtual environments, according to various color types used in the Kinect v1 [10] (Fig. 5).

Fig. 5.

In our experiment, we evaluate our motion transcription algorithm by the calculation of overlapping ratio (the number of green pixels)/(the number of blue pixels) * 100. This refers to the extent to which a virtual 3D liver overlaps its corresponding real 3D liver during a surgical navigation (Color figure online).

While observing the colors, a user can select several parameters related to the 3D CG virtual environment according to the consistency between the real liver in the camera coordinate system and the virtual liver in the graphics coordinate system. In the image, the blue and red pixels represent the pixels projected from the virtual and real 3D livers, respectively. Furthermore, the yellow and green pixels represent the pixels overlapping the virtual and real 3D livers in the XY plane and the XYZ space, respectively. If depth differences between the real and virtual images are less than 1 cm at a pixel, the pixel color is changed from yellow to green. Many liver surgeons said that an error of 1 cm is acceptable during a liver surgery. In this image-based position/orientation adjustment system, the CG virtual world captured artificially by the z-buffer of the GPU should agree with the real world captured by a real depth camera [10].

Furthermore, by calculating (the number of green pixels)/(the number of blue pixels) * 100, we can evaluate the ratio of a 3D real liver that overlaps its corresponding 3D virtual liver (Fig. 5). In the following section, by using the overlapping ratio during an experimental operation in a surgical room, we can evaluate our motion transcription algorithm with no, one, two shadow-less lamps, as well as without them, and with invisible light filters. It is quite obvious that the evaluation area is defined as a circle in which we can see the 3D liver through an occluding object.

3 Our Experimental Results for an Occluded Liver with/Without Shadow-Less Lamps and Invisible Light Filter

In this section, many realistic experiments are conducted, not in the laboratory, but in the medical surgical operation room. The overall layout of the experimental equipment is shown in Fig. 6. In order to move the liver, an acrylic plate that is 25 cm in length, 25 cm in width, and 2 cm in thickness was placed on the operating table, and a real object liver model was placed on the top of the plate. Migration of the liver was carried out by moving the acrylic plate instead of moving the actual liver model itself. Moreover, for experiments involving occlusion, a cardboard sheet containing a hole of 10 cm in diameter was placed on the real liver model. The surface of this corrugated board was colored in light orange color using a coloring spray to make it look like human skin. Kinect, which was attached to a vertically movable robot, was able to change the distance from the real liver model according to the situation. Kinect was placed horizontally with respect to the operating table at a height of 84 cm from the bottom of the actual liver model. The camera system was fixed by attaching metal fittings to a metal rod. The metal rod-fixed Kinect was attached to the robot. The distance from the robot to Kinect was 32 cm.

Fig. 6.

In our navigation study, we evaluate our motion transcription algorithm in a real surgical room with two shadow less lamps covered by two kinds of invisible light filters. In our navigation, a 3D liver is always obstructed by a cardboard box with a circle. Through the circle, the camera can capture a part of the 3D liver. In another experiment, we evaluate the motion transcription algorithm by changing the size of circle.

In the experimental procedure, first, the initial positions of the real liver model and the virtual liver model were matched. Then, the real liver model that was placed on an acrylic board was covered with corrugated cardboard and was subjected to translational and rotational movement within the corrugated cardboard box. Prior to the movement, the actual liver model was installed, so that it could be seen from the whole hole of the cardboard. It was done to ensure that the range visible from the hole of the cardboard did not become less than half after the movement of the model. The offset value was 10 mm for real and virtual depth images against the 3D real liver and its polyhedron with the STL format in our surgical navigator. The model was moved by 5 cm in the y-axis direction during the parallel movement, and it was rotated 45° around the z-axis center during the rotational movement. Each coordinate axis is shown in Fig. 6.

First, we explain the differences between this experiment and our previous experiments [8,9,10]. The main difference is the use of a real surgical room with one or two shadow-less lamps (Fig. 6). All the previous experimental results were obtained in our research room under daylight and/or fluorescent lamps for a 3D liver opened perfectly (which is not occluded by any other object).

In the following paragraphs, we successively discuss the following: observation without any lamp, observation with one lamp, observation with two lamps, observation with two lamps covered by an infrared shielding filter TS6080, which is manufactured by SyberLeps Co, and the observation with two lamps covered by an invisible light filter SL999 developed by Nexfil Co. for pairs of translational and rotational movements of the 3D liver (Fig. 7).

Fig. 7.

(a) Light transmittance and several specs of SL999, (b) Light transmittance and several specs of TS6080.

3.1 Surgical Observation Without Any Lamp

The experiment was carried out in the surgical operation room without any shadow-less light.. The state of translating and rotating the 3D real liver model was captured with Kinect Studio and the experiments were performed using the data taken. Accuracy evaluation was performed using the overlapping ratio from the time of initial position adjustment to the time of parallel movement and after rotational movement (Fig. 8).

Fig. 8.

A real red liver obstructed by a box with a circle window is followed by its virtual blue liver formed by STL polyhedron in a surgical operation room with one shadow-less lamp. (a), (b) The liver is rotationally moved before and after a surgical operation, respectively. (c), (d) The liver is translationally moved before and after a surgical operation, respectively.

The real liver model and the virtual liver model were interlocked, and it was confirmed that it moves in the same way while rotating and translating. Before moving (Fig. 8(a)), as shown in Fig. 10(a), the overlapping ratio was 99.5%, which is considerably high. When moved (Fig. 8(b)), the overlapping ratio was affected, and after movement it became 95.4%, as shown in Fig. 8(b). Figure 10(a) is a graph showing the transition of the overlapping ratio from the end of the initial alignment to the end of the movement. No significant change in the coincidence ratio was calculated from the end of the initial alignment until after the movement. In our steepest descent method, although the liver navigator also showed a decrease in the overlapping ratio during the translational movement, there was no drop in the overlapping ratio during the rotational movement.

It was confirmed that when the real liver model was moved, the virtual liver model moved in parallel in the same way. Before the movement, as shown in Fig. 8(c), the overlapping ratio was 98.6%, which was considerably high. When the model was moved, as shown in Fig. 8(d), the overlapping ratio decreased, and after the movement, it became 88.3%, as shown in Fig. 10(b). Although the decrease in the overlapping ratio is seen when compared with the same ratio before the movement, the result shows a high overlapping ratio that exceeds 80%. Figure 10(b) is a graph showing the transition of the overlapping ratio from the end of the initial alignment to the end of the movement. During the movement, the overlapping ratio dropped sharply by approximately two times. It was approximately 30% lower than the initial alignment, and a maximum reduction of approximately 60% was observed. However, the overlapping ratio stabilized and improved at each elapsed time interval, and after the movement, it showed a high value that approached the overlapping ratio after the completion of the initial position adjustment.

3.2 Surgical Observation with One Lamp

In this part, we use the operating room with one shadow-less lamp for conducting several experiments. As compared with the no lighting case, our algorithm’s performance decreases, unfortunately, due to the influence of some unsuitable wavelengths emitted by the shadow-less lamp (Fig. 9).

Fig. 9.

A real red liver obstructed by a box with a circle window, is followed by its virtual blue liver formed by STL polyhedron in a surgical operation room without any shadow less lamp. (a), (b) The liver is rotated before and after a surgical operation, respectively. (c), (d) The liver is translationally moved before and after a surgical operation, respectively (Color figure online).

The real liver model and the virtual liver model were interlocked, and it was confirmed that it moves in the same way when rotating and translating. Prior to the movement, as illustrated in Fig. 9(a), a very high coincidence ratio of 98.2% was observed. During movement, the overlapping ratio fluctuated, and after movement, as shown in Fig. 9(b), it was reduced to 77.3%, as shown in Fig. 10(c). It is a graph showing the transition of the overlapping ratio from the end of the initial alignment to the end of the movement. A decrease in the overlapping ratio was observed every time it was moved, and it decreased to 50% at the maximum. However, as the movement ended, the overlapping ratio increased, showing an overlapping ratio exceeding 70%.

Fig. 10.

(a), (b) The overlapping ratio is changed during a surgical operation if a human moves a 3D liver rotationally and translationally, respectively, in a surgical room without any shadow-less lamp. (c),(d) The overlapping ratio is changed during a surgical operation if a human moves a 3D liver rotationally and translationally, respectively, in a surgical room with one shadow-less lamp.

It was confirmed that when the real liver model was moved, the virtual liver model moved in parallel, in the same way (Fig. 9(c), (d)). Prior to the movement, as shown in Fig. 10(d), a very high coincidence ratio of 100% was observed. During movement, the coincidence ratio fluctuated, and after movement, it was reduced to 83.4%, as shown in Fig. 10(d). Although the decrease in the overlapping ratio is seen while comparing it with the same ratio before the movement, the result shows a high overlapping ratio exceeding 80%. Figure 10(d) is a graph showing the transition of the overlapping ratio from the end of the initial alignment to the end of the movement. Although there was no significant decrease in the overlapping ratio as there was in the case of no lighting, a decrease by approximately 15% in the overlapping ratio was observed. No increase was observed after the decrease in the overlapping ratio, and the overlapping ratio after the movement was 83.4%. The difference was not very significant when compared with the state when the shadow-less light was not used.

3.3 Surgical Observation with Two Lamps

A real liver model is moved rotationally, and then its virtual liver model follows the real liver following our motion transcription algorithm. Prior to the rotational movement, as shown in Fig. 11(a), the overlapping ratio is 95.9%, which is considerably high. When moved (Fig. 11(b)), a variation in the overlapping ratio is observed, and after the movement, it increases to 97.6%, as shown in Fig. 13(a). It can be observed that there is an increase in the overlapping ratio after the initial position alignment is complete. Figure 13(a) is a graph showing the transition of the overlapping ratio from the end of the initial alignment to the end of the movement. A decrease in the overlapping ratio is observed for each movement, and the overlapping ratio decreased by approximately 20%. However, after the movement, the overlapping ratio showed a higher coincidence ratio when compared with the coincidence ratio after the completion of the initial position adjustment. From the beginning of the experiment, it showed a considerably high concordance rate.

Fig. 11.

A real red liver obstructed by a box with a circle window, is followed by its virtual blue liver formed by STL polyhedron in a surgical operation room with two shadow less lamps. (a), (b) The liver is rotated before and after a surgical operation, respectively. (c), (d) The liver is translationally moved before and after a surgical operation, respectively (Color figure online).

In succession, a human controls a real liver model translationally, and then its virtual liver model follows the real liver model. As shown in Fig. 11(c), the overlapping ratio is 90.3% before moving, which is considerably high. However, it showed a slightly lower overlapping ratio when compared to when no shadow light was used or when one shadow light was used. When moved (Fig. 11(d)), the overlapping ratio fluctuated, and after the movement, it was reduced to 82.9%, as shown in Fig. 13(b). Although the decrease in the overlapping ratio is seen in comparison with the ratio before the movement, the result shows a high overlapping ratio exceeding 80%. Figure 13(b) is a graph showing the transition of the overlapping ratio from the end of the initial alignment to the end of the movement. The result shows that the overlapping ratio is low on an average as compared with when the shadow-less light is not used.

3.4 Surgical Translational Observation with Two Lamps Covered by an Invisible Light Filter SL999 Constructed by Nexfil Co.

The rotational movement is observed by following a real liver model by its virtual liver model. Figure 12(a) shows the real 3D liver before the movement. In the first stage, the real and virtual livers coincide with each other as 99.9% overlapping ratio is observed. Thereafter, when a human controls the real liver translationally, the overlapping rate fluctuated. If the movement is finished as illustrated in Fig. 12(b), the overlapping ratio decreased to 98.8%, as shown in Fig. 13(c). The overlapping ratio did not decrease even after the completion of the initial position adjustment. Figure 13(c) is a graph showing the transition of the overlapping ratio from the end of the initial alignment to the end of the movement. A decrease in the overlapping ratio was observed every time the model was moved, and at times, it reduced to less than 90%. After the movement, the result showed a high coincidence ratio after the completion of the initial position adjustment. The overlapping ratio was high even during the movement.

Fig. 12.

A real red liver obstructed by a box with a circle window, is followed by its virtual blue liver formed by STL polyhedron in a surgical operation room with two shadow less lamps covered by an infrared shielding filter SL999. (a), (b) The liver is rotated before and after a surgical operation, respectively. (c), (d) The liver is translationally moved before and after a surgical operation, respectively (Color figure online).

Fig. 13.

(a), (b) The overlapping ratio is changed during a surgical operation if a human moves a 3D liver rotationally and translationally, in a surgical room with two shadow less lamps. (c), (d) The overlapping ratio is changed during a surgical operation if a human moves a 3D liver rotationally and translationally, in a surgical room with two shadow less lamps covered by an invisible light filter SL999 developed by Nexfil Co.

Similar to the previous experiments, a human controls the real liver translationally. Thereafter, by using our motion transcription algorithm, its virtual liver model follows the real liver model stably. Prior to the movement, as shown in Fig. 12(c), the overlapping ratio was 98.9%, which was considerably high. It showed an overlapping ratio of approximately 8% higher as compared with the case when two shadow-less lights were used. When moved (Fig. 12(d)), a variation in the overlapping ratio was observed, and after the movement, it became 89.3%, as shown in Fig. 13(d). Although a decrease in the overlapping ratio is seen when compared with the same ratio before the movement, the result showed a high overlapping ratio, which exceeded 80%, after the movement. Figure 13(d) is a graph showing the transition of the overlapping ratio from the end of the initial alignment to the end of the movement. Although a sharp decrease in the overlapping ratio was observed before the end of the movement, the overlapping ratio increased to nearly 90% when the movement was over.

3.5 Surgical Rotational Observation with Two Lamps Covered by an Infrared Shielding Filter TS6080 Made in SyberLeps Co

A real liver model is rotationally manipulated by a human, to study the rotational movement. In succession, our algorithm always leads its virtual liver model so that the virtual liver follows the real liver. Prior to the rotational movement (when the initial state is described in Fig. 14(a), the overlapping ratio was 98.8%, which was very high. When moved, the overlapping ratio changes. After the movement (Fig. 14(b)), the ratio decreased to 94.4%, as shown in Fig. 15(c). The variation in the overlapping ratio is considerably small, since the initial alignment was completed. Figure 15(c) is a graph showing the transition of the overlapping ratio from the end of the initial alignment to the end of the movement. A decrease in the overlapping ratio was observed each time the model was moved. However, it did not fall below 90%, indicating an overall high overlapping ratio. This result showed a high overlapping ratio after the initial position is adjusted.

Fig. 14.

A real red liver obstructed by a box with a circle window, is followed by its virtual blue liver formed by STL polyhedron in a surgical operation room with two shadow-less lamps covered by an infrared shielding filter TS6080. (a), (b) The liver is rotated before and after a surgical operation, respectively. (c), (d) The liver is translationally moved before and after a surgical operation, respectively (Color figure online).

Fig. 15.

(a), (b) The overlapping ratio is changed during a surgical operation if a human moves a 3D liver rotationally and translationally, respectively, in a surgical room with two shadow less lamps. (c), (d) The overlapping ratio is changed during a surgical operation if a human moves a 3D liver rotationally and translationally, respectively, in a surgical room with two shadow less lamps covered by an infrared shielding filter TS6080 made in SyberLeps Co.

During the translational movement of a real liver model, its virtual liver model, which is supported by our algorithm, follows the real liver model precisely. As illustrated in Fig. 14(c), it is as high as 97.8% before moving. When moved (Fig. 14(d)), a variation in the overlapping ratio is observed, and after the movement, it is reduced to 90.5%, as shown in Fig. 15(d). Although a decrease in the overlapping ratio is observed as compared with the same ratio before the movement, the result showed a high overlapping ratio, which exceeded 90%, even after the movement. Figure 15(d) is a graph showing the transition of the overlapping ratio during the movement. The overlapping ratio decreased to 80%, but when the movement finished, the overlapping ratio increased to approximately 90%. The fluctuation in the overlapping ratio during movement is less than when SL999 was used, but the fluctuation is consistent.

4 Conclusions

In this paper, we carefully evaluate our liver following algorithm during a liver surgery with some artificial occlusion in a surgical room with at most two shadow-less lamps covered by two types of infrared shielding filters, SL999 and TS6080. Our motion transcription algorithm uses a sequence of depth images captured practically within a circle obstructed by a cardboard box. Here, the real depth images are strongly affected by the shadow-less lamps. In general, the light spectrum of shadow-less lamp should be between 350 and 1000 nm. Therefore, in this research, we locate an infrared shielding-type filter in the front of the shadow-less lamps to cut the infrared radiation (invisible light) over 900 ns. Thereafter, the following performance of our depth-depth matching algorithm is evaluated by using only the visible radiation. As a result, the performance of experimental following a real liver by its virtual liver with some occlusion is aggressively improved by using two types of infrared shielding filters SL999 and TS6080S.