1 Introduction

In transnasal endoscopic surgery, to remove pituitary tumors, an endoscope and surgical instruments are inserted into the nasal cavity, and an incision is made in the sphenoidal sinus (paranasal sinus) and Turkish saddle (sella turcica), so that removal of the tumor can be performed. This type of surgery is less invasive for the patient than a craniotomy, but to make the incision to remove the tumor based on imagery from an endoscope, the surgeon must use highly skilled techniques. Thus, a CT or MRI is used in advance to acquire an image surrounding the affected area in the patient for intraoperative navigation, but understanding the structure of the area can be difficult because the image differs from the operating field in that it is 2D. There are many kinds of surgical navigation systems that have been proposed. They are almost used many present 2D images acquired in advance using a CT, MRI, etc., and the surgeon must attempt to understand the position of the tumor indirectly. In addition, although navigation systems for superimposing 3D models on the operating field have been proposed, there has been little application to the pituitary gland [1,2,3,4].

Therefore, in this research, to reduce the burden on the surgeon, we have investigated a support navigation system that superimposes information related to the vicinity of the affected area on the operating field. In this paper, we analyze the method for estimating the position and orientation of the endoscopic camera. To verify the examined method, a mock endoscope model based on actual transnasal endoscopy was created, and we report on the results of a prototype system we implemented that uses the camera tip position estimation method using the mock endoscope and markers.

2 Camera Positioning and Orientation Estimation Method

To estimate the positioning and orientation of the tip of the endoscopic camera inserted into the patient’s body during surgery, this method places markers opposite the endoscopic camera, and an external camera is used to estimate the position and orientation. To estimate the tip of the camera’s position and orientation from the acquired position and orientation of the markers, markers for recognizing the camera tip are temporarily prepared, and a vector between the markers is calculated. Using this calculated vector, the positioning and orientation of the moving camera are calculated [5, 6]. This work is fundamentally performed prior to surgery. However, even if the position/orientation of the markers shift for some reason, it is possible to perform readjustment and calibration during surgery. A general explanation of this process is shown in Fig. 1.

Fig. 1.
figure 1

Process flow for the method.

Full size image

The external coordinate system will be represented as \( \Sigma _{c} \), and the marker coordinate system will be \( \Sigma _{k} \). The markers attached to the endoscope camera in advance will be \( M_{camera} \), and the temporary markers for recognizing the tip of the endoscopic camera will be \( M_{temp} \). The camera tip is placed at the origin of \( M_{temp} \), and the positioning and orientation of each marker are measured. The position and orientation of the \( M_{camera} \) markers measured with \( \Sigma _{c} \) are \( P_{camera}^{c} \) and \( R_{camera}^{c} \), and the positions of the markers \( M_{temp} \) are taken as \( P_{temp}^{c} \). The formula for calculating the value for the relative vector from the end of the endoscope to the tip, \( P_{rel}^{c} \), is

$$ \begin{array}{*{20}c} {\varvec{P}_{{\varvec{rel}}}^{\varvec{c}} = \varvec{P}_{{\varvec{camera}}}^{\varvec{c}} - \varvec{P}_{{\varvec{temp}}}^{\varvec{c}} } \\ \end{array} $$
(1)

Further, because the relative vector’s value is for \( \Sigma _{c} \), it must be converted to \( \Sigma _{k} \). The formula for this is

$$ \begin{array}{*{20}c} {\varvec{P}_{{\varvec{rel}}}^{\varvec{k}} = \left( {\varvec{R}_{{\varvec{camera}}}^{\varvec{c}} } \right)^{ - 1} \cdot \,\varvec{P}_{{\varvec{rel}}}^{\varvec{c}} } \\ \end{array} $$
(2)

If the tip position of the endoscope camera moving during the operation is \( P_{tip}^{c} \), then the formula for calculating its value is

$$ \begin{array}{*{20}c} {\varvec{P}_{{\varvec{tip}}}^{\varvec{c}} = \varvec{R}_{{\varvec{camera}}}^{\varvec{c}} \cdot \varvec{P}_{{\varvec{rel}}}^{\varvec{k}} + \varvec{P}_{{\varvec{camera}}}^{\varvec{c}} } \\ \end{array} $$
(3)

Additionally, several markers are simultaneously recognized by the external camera, and the estimated values are used as an average from this plurality of estimated tip positions. Moreover, as shown in Fig. 2, the markers used are simple ArUco markers [7, 8].

Fig. 2.
figure 2

ArUco marker

Full size image

3 Prototype System

To estimate the position of the endoscopic camera using the external camera, we begin by making a mock endoscope with a small-scale camera attached based on an endoscope actually used in surgery. The mock endoscope is equipped with a camera, and markers are attached to the end of the endoscope. Using these markers attached to the mock endoscope, the position of the camera tip can be estimated.

3.1 Mock Endoscope

Based on endoscopes actually used in surgery, a mock endoscope model was created as shown in Fig. 3. A camera was built into the mock endoscope model, and as shown in Fig. 4, a base was designed to allow for markers to be set; it was equipped with regular hexagonal markers on the end. To account for any camera orientation, as shown in Fig. 5, the mock endoscope was created with six different forms of ArUco markers used for the portion with the markers installed. The size of the mock endoscope can be seen in Fig. 6.

Fig. 3.
figure 3

Endoscope model.

Full size image
Fig. 4.
figure 4

Model for marker installation.

Full size image
Fig. 5.
figure 5

Mock endoscope.

Full size image
Fig. 6.
figure 6

Mock endoscope size.

Full size image

3.2 External Camera

To recognize the attached markers on the mock endoscope, this system uses a camera with the specifications listed in Table 1. The main body of the camera can be seen in Fig. 7.

Table 1. External camera specifications.
Full size table
Fig. 7.
figure 7

External camera for marker recognition.

Full size image

3.3 Estimating the Tip Position of the Endoscopic Camera

To estimate any position of the endoscopic camera, this system calibrates the tip position beforehand. As shown in Fig. 8, first, the marker at the end of the camera installed on the mock endoscope and the temporary marker at the tip of the camera position are prepared in advance. The values for the position and orientation of these markers are recognized by the external camera, and a direction vector is calculated from the end marker to the tip marker. This work is performed for the six types of markers affixed to the end. After that, the markers affixed to the camera tip position are removed. Based on the data obtained from calibration, the camera tip’s position can be estimated for any position and orientation of the mock endoscope. In addition, when several end markers are recognized simultaneously, the average value is acquired.

Fig. 8.
figure 8

Estimation for tip position of endoscopic camera setup.

Full size image

4 Preliminary Experiment

We conducted a preliminary evaluation experiment to test the accuracy of marker recognition using the prototype system. The constructed experimental environment is shown in Fig. 9. The external camera was installed 95 cm above the surface of the desk with the shooting surface parallel to the ground, and using this external camera, the coordinate values of the markers were read. The camera tip position estimation was then calculated based on the measurements of those numerical values. For the measurement procedure, the mock endoscope was moved from the initial position 5 cm with respect to the x-axis, y-axis, and z-axis, with one set of position measurements made for each centimeter, resulting in ten sets of measurements. Additionally, to move the mock endoscope, a parallel, moving measurement stage was used. The actual experimental environment can be seen in Fig. 10.

Fig. 9.
figure 9

Experimental environment conditions.

Full size image
Fig. 10.
figure 10

Actual experimental environment.

Full size image

The experimental results of the position measurement error can be seen in Fig. 11. The average error for each position was less than 1.8 mm. Further, the largest errors observed were 3.26 mm on the x-axis, 2.42 mm on the y-axis, and 2.16 mm on the z-axis. Figure 12 shows the orientation measurement error results. The average measurement error for orientation was less than 3°. The greatest errors observed were 2.78° for the x-axis, 3.87° for the y-axis, and 4.46° for the z-axis.

Fig. 11.
figure 11

Results for position measurement errors.

Full size image
Fig. 12.
figure 12

Results for orientation measurement errors.

Full size image

Using the graphs created from the experimental results, constant accuracy was confirmed for both axis movement and axis rotation. However, depending on the rotation angle, there were cases in which recognized markers were unstable. In particular, these cases were often seen when the external camera and markers became parallel. From this observation, we can consider the decrease in the accuracy of tip position estimation in cases where the external camera and markers are parallel.

Moreover, as an experiment for verifying the marker accuracy, the results for fixed markers and position measurements performed for 15 s can be seen in Figs. 13, 14 and 15. The horizontal axis of the figures shows the number of measurements. From these results, it was discovered that the measurement results changed by approximately 1 mm for each axis approximately 7 s from the start of the measurement. Based on this information, we could confirm the limits of the measurement errors for this prototype system.

Fig. 13.
figure 13

Measurement for fixed position of marker over time (x-axis).

Full size image
Fig. 14.
figure 14

Measurement for fixed position of marker over time (y-axis).

Full size image
Fig. 15.
figure 15

Measurement for fixed position of marker over time (z-axis).

Full size image

5 Conclusion

In this research, we investigated an estimation method for the tip position and orientation of an endoscopic camera for developing a surgical navigation system, and we prototyped a useable system. In basic experiments using ArUco markers and a mock endoscope, we could measure the error range and confirm a useful consistency. Regarding future topics, we hope to further improve the accuracy of this system and, based on the measurement values for the positioning and orientation of the camera tip, investigate a function that would allow the superimposition of a 3D model onto a camera image showing the area of the operation.