Keywords

1 Introduction

Endoscopic surgery has been popular since the 1990s with number of successful cases and several researches have focused on its variety of cases and procedures [1]. Endoscopic surgery is performed using special forceps and endoscope inserting through small incisions to treat diseases. Owing to its low intrusiveness and dressing effect after the surgery compared with conventional open surgery, the demand of endoscopic surgery has significantly increased [2, 3].

The focus of the related research then shifted gradually to its safety and operability issues. According to Isaka [4], at a certain level of difficulty in the case of surgery, the intrusiveness for the endoscopic surgery exceeds that for open surgery. Difficult surgical procedures using endoscopy can be performed by a few skilled endoscopic surgeons and this problem related to technology transfer causes endoscopic surgery cases to be delayed to replace open surgery cases [4].

One of the causes for increasing the degree of difficulty in endoscopic surgery is that both hands of the operator and assistant have to be completely engaged in forceps throughout the operation because of security and maintenance of the surgical field once constructed. Consequently, when they communicate with each other discussing treatments such as which area of the body to resect, it is not possible to physically point at the image of the inside of the body projected on a monitor. Therefore, the operator and assistant have to share information associated with specific locations for treatment on endoscopic images projected on the monitor verbally. This problem means that accurate and effective communication is not fully achieved between the two, and, in the worst case, this may create the potential risk of a medical accident.

To avoid such miscommunication problems, we have developed an application for AR-based communication supporting system during endoscopic surgery. The application uses a see-through head mounted AR device, for visualization of graphical information superimposed with the monitor showing endoscopic images. The application allows the users to draw free lines and shapes on the projected endoscopic images using head motions additionally with the function of voice recognition. The drawn images are shared by both the operator and the assistant instantly; thus, the operator can provide instructions to the assistant on the surgical operation, such as the method of excising the affected area, without releasing his/her hand from the forceps.

When this system is practically employed in the surgical operation room environment, it is desirable to confirm an input accuracy that allows the lines and shapes to be drawn easily as intended by operators. In the field of HCI, quantitative evaluation for such input methods has been modeled using the steering law, a performance model for trajectory-based testing paradigms. Several studies have been conducted to compare existing input devices, such as mice and tablets, in the task of adjusting to activation using steering law, and it has been demonstrated that this law was useful as an indicator in comparing such performance [5]. As far as it is concerned, research regarding input device using head motions evaluated by steering law was, however, insufficient to apply the knowledge to our applications. Therefore, we conducted the experiment to compare and evaluate the input performance using the head motions with direct hand motions and indirect mouse input motions using steering law. The objective of this paper is to show the results of the experiment and discuss them, especially in terms of effectiveness of the communication supporting system during endoscopic surgery used for the operating room settings.

2 The Proposed System

We constructed the system using Microsoft HoloLens as a hardware platform for wearable AR and built a corresponding image drawing application, “Drawing” as well. As an application of Microsoft HoloLens in the medical field, position confirmation of blood vessels by superimposing and displaying a virtual model on a patient during operation [6], support of cancer removal surgery by near infrared light using AR technology [7], and practical training before surgery by holographicizing patient data and the surgical site in an operation room space [8].

Head motion and voice input were adopted as the operation methods of this application, thus the operators can perform hands-free communication with assistants, and they do not have to move their hands away from the surgical tool during communication. Using “Drawing”, as shown in Fig. 1, it was possible to clearly convey the excision site from the in-vivo image on the monitor. In “Drawing”, a line was drawn according to the motion of the cursor displayed on the object surface in a direction straight from the operator’s head, and drawing was started and completed through speech input. In addition, the deletion of drawn lines, change of line thickness, and change of line color were also performed through voice input.

Fig. 1.
figure 1

The proposed system. Blue curved line at the center of the monitor was drawn by the operator for indicating to the assistant where and how to put a scalpel. The ring-shaped icon indicates where the operator is currently looking. (Color figure online)

Full size image

3 Methods

3.1 Participants

Five male university students were recruited as participants for the experiment. Their ages ranged from 21 to 23 years. One of the participants had experience using head motion control devices.

3.2 Experimental Setup

The experimental setup and layout is as shown in Fig. 2. A test pattern for performing the steering task was projected on the screen. The distance from the participant to the screen was 1.5 [m], which was based on the distance from the operator to the monitor in the actual endoscopic surgery environment. The size of the projected image was 94.4 [cm] long and 53.2 [cm] wide. In the actual operating room environment, the operators perform surgery in a standing position, and they use not only head motions but also the whole body if they want to control cursors for drawing objects precisely on the screen. However, in the present experiment, we asked them to be seated to restrict them to only head motion so that we could compare the results with other types of motion such as hand and mouse motions.

Fig. 2.
figure 2

Experimental environment. Participant uses a laser pointer to move the light along with a test pattern. For head condition, the AR-head mounted device was equipped on the participant’s head, instead.

Full size image

In this experiment, we compared the quality of the drawing operation strategies under the three input conditions, i.e., direct head motion, direct hand motion, and indirect mouse motion. For the mouse condition, participants control the mouse by looking at an icon projected on the screen. For the hand condition, participants extended their arms, held the laser pointer, and operated the laser while looking at the projected screen (see Fig. 2). For the head condition, we used a head mounted device with a laser pointer attached as shown in Fig. 3. The switch of the laser pointer was maintained in the ON state, thus always emitting light. Wearing this device, the participant was able to operate the pointer according to the head motion. The cursor in this experiment was set to the same shape and size as the laser pointer.

Fig. 3.
figure 3

Device used for the experiment

Full size image

According to previous research [5, 9, 10], the steering task was performed using two types of objects, linear and circular as shown in Fig. 4. Indices of difficulty in the steering task with a linear object can be expressed by Eq. (1), and the Indices of difficulty in the steering task with a circular object can be expressed by Eq. (2).

Fig. 4.
figure 4

Types of object for steering task

Full size image
$$ ID = \frac{A}{W} $$
(1)
$$ ID = \frac{A}{W} = \frac{2\pi R}{W} $$
(2)

In these equations, A represents the length of the object, W represents the width of the object, and R represents the radius of the circular object. Table 1 summarizes the task conditions. Six levels of different IDs ranging from 5.7 to 26.7 were tested in this study.

Table 1. Test patterns with their features in the study.
Full size table

3.3 Procedure

Participants were asked to be seated and were informed about the experiment before their informed consent was obtained. Subsequently, participants repeated the task patterns 6 and 12 ten times in a practice session to eliminate initial learning effect. This practice session was completed under each condition of mouse, hand, and head. After the practice session, the trial was started. The participant moved the pointer from the start position to the goal position as quickly and accurately as possible while preventing the trajectory from protruding to the test pattern projected on the screen. When the pointer reached the goal position, the next test pattern was displayed. When the test pattern was a line object, participants were asked to use a pointer to trace the object from left to right, and when the test pattern was a circular object, they were asked to trace it in a clockwise direction. In this manner, the participants repeated the trials until all test patterns were completed. If the pointer went out of the test pattern while the participant was in tracking operation, the trial was discarded and he had to try again from the starting position. The number of error trials was counted and error rates (i.e., number of error trials divided by total trials) were analyzed. Task patterns were displayed in a random order. The time elapsed as the pointer moved from the start position to the end position was recorded as \( MT \). These series of operations were performed under the three conditions of mouse, finger, and head, for 144 trials of 12 trials (2 objects × 6 IDs), including each for four repetitions.

4 Results

Figures 5 and 6 show the relationship between \( ID \) and \( MT \) for linear and circular objects.

Fig. 5.
figure 5

The relationship between \( ID \) and \( MT \) for linear objects

Full size image
Fig. 6
figure 6

The relationship between \( ID \) and \( MT \) for circular objects

Full size image

Linear regression equations were obtained for each condition, and they are summarized in Table 2.

Table 2. Linear regression equations and their coefficient of determination for each condition.
Full size table

The coefficient of determination exceeded 0.97 even at the lowest task conditions, thus it was discovered that the task performances using the various input motions could be well described through the steering law. Table 3 shows \( IPs \) and error rates.

Table 3. \( IPs \) and error rates for each condition.
Full size table

It was observed that the case of head condition demonstrated relatively lower performance than other cases according to the \( IPs \) and error rates.

5 Discussion

In this experiment, performance for the drawing task operated using head was compared with operation using mouse and hand, respectively, and the steering law was applied to evaluate the input method for the communication support during endoscopic surgery. Consequently, \( IP \) increased in the order of Hand, Mouse, and Head conditions.

The Hand condition demonstrated better performance than the Head condition, and there may be a unique advantage of direct manipulation by the distance between the operating position and the screen position. In the case of the Hand condition, the participant stretched their arms and performed drawing operations, so that the distance between the screen and the hand was much closer than in the Head condition. Assuming that the length of the participant’s arm was 70 [cm], the distance from the position of the laser pointer to the screen was approximately 80 [cm]. Consequently, the displacement of the pointer on the screen was 2.62 [cm] if the participant tilted the laser pointer by 1 [°], whereas in the case of the Hand condition, the displacement was reduced to 1.40 [cm] for the same degree of tilt angle. Thus, the difference of pointer-screen distance might have resulted in a difference of \( IP \).

In this experiment, it was evident that the Mouse condition, where the arm could be placed on the desk, was more stable than the Hand and Head conditions. Because the pointer position on the screen did not change unless the participant moved the mouse, the participant no longer needed to continuously control the position of the pointer. Because of the trade-off between speed and accuracy, a strategy that prioritizes speed under Hand conditions also appeared in the trend of error rates. It was inferred that the error rate at the Hand and Head conditions was higher than that at the mouse condition due to the difference in the operation strategy.

The error rate under the Mouse condition was 0.833% and 6.667% for a linear and circular object, respectively, whereas Accot and Zhai [5] reported that the error rate for Mouse condition was 9% and 14% for a linear and a circular object, respectively. The difference in error rate was unlikely to be owing to the difference in \( ID \), because the range of \( IDs \) in this experiment was 5.7–26.7, which was fairly close to the range of \( IDs \) by Accot and Zhai [5], demonstrating 3.6–28.6. Therefore, although no distinct cause has been observed so far, it was speculated that differences in mouse precision may be affecting the difference in error rates.

The effectiveness of our application “Drawing” as an endoscopic surgery support system can be evaluated to a certain extent based on comments obtained from the endoscopic surgeons. According to them, it is desirable for the line to be drawn within ±1 [cm] of the assumed resection site projected on the screen. When this condition is applied to the steering task in this study, it corresponds to \( W = 2 \) [cm], for the steering task model. Under this condition, the ID is 10 when a quarter circle of 40 cm in circumference is drawn on the screen. From the regression line obtained in this experiment, it can be observed that there is a difference of approximately 1.5 [s] in movement time between head movement conditions and other conditions.

In this experiment, under the head condition, the operation control using the lower limb was not permitted. However, in practice, the operator is standing and predicts that the accuracy of the operation may be improved using additional control by lower limb motions. The disadvantage of the distance between the pointer and the screen at the time of the head condition also suggested that the operation at the head condition may not be as difficult as the difference in \( IP \) actually observed. Although stress may occur compared with the hand and mouse conditions, it is considered that this system has potential as an endoscopic surgery support system.