Keywords

1 Introduction

Minimally invasive surgery (MIS) has advantages over open surgery which include smaller incisions, tissue sparing techniques, shorter hospital stay, and quicker recovery times. However, MIS remains challenging for new and experienced surgeons because complex surgical tasks are completed with surgical instruments that have limited dexterity [2]. These challenges are mitigated with assistance from Robot-assisted surgery (RAS), whereby a surgeon controls dexterous surgical instruments to perform MIS procedures more efficiently. However, functional outcomes of the above-mentioned paradigm depend on the surgeon’s proficiency and training, which varies between individuals. Such performance variations tend to increase complication rates in general surgeries [8]. Autonomous RAS is one emerging solution with the potential to minimize inconsistency between surgeons, and reduce procedure complications [12].

Thus far, autonomous RAS research has mostly focused on automating surgical tasks on static anatomy via pre-planning methods and without considering in situ tissue deformation and motion [6, 11]. Implementing a similar workflow in deformable tissue surgery would be risky, as interactions with the tissue induce unpredictable changes in position, orientation, and deformation during surgery. An ideal autonomous robotic system needs to identify these changes, and regularly update the surgical path to complete the task. Since changes in deformable tissue are unpredictable, implementing accurate path planning remains challenging. Several path planning strategies have been proposed for deformable tissue in autonomous robotic surgery. Schulman et al. implemented a non-rigid transformation that updates motion trajectories for autonomous suturing on a deformable target [10]. Yet, the proposed method did not involve initial planning for placing suture points on the target. Moreover, Shademan et al. demonstrated an in vivo supervised autonomous suture routine for deformable tissue [12] based on tracking and linear suture path planning between biocompatible near-infrared (NIR) markers placed on the tissue. This strategy breaks down if the contour of the tissue surface is not a linear shape (Fig. 1e), since 3D information from the tissue edge is not considered. To plan a path using surface features, Le et al. implemented a 3D shortest path planning method into an autonomous robotic system to achieve 3D tumor resection [3]. Nevertheless, the method was not fully automated and repeating such paradigm to plan a 3D path is time consuming.

Fig. 1.
figure 1

(a) Experimental testbed, (b) dual camera system, (c) NIR camera image of the vaginal cuff phantom, (d) overlayed marker positions on the point cloud, and (e) comparison between linear (i.e. blue box) and 3D path planning (i.e. green point) on an uneven surface. (Color figure online)

To address the drawbacks mentioned above, we present a novel 3D path planning algorithm using bio-compatible NIR markers and a non-rigid registration technique. NIR markers have strong signal penetration, and are suitable for intra-operative robot guidance since their high signal to noise ratio (SNR) allows them to be readily detected when obstructed by blood and tissue [1]. The proposed algorithm first utilizes locations of the NIR markers to calculate and place planned suture points on the edge of a vaginal cuff. Since the planned suture points are placed virtually and can not be tracked after the tissue is deformed, a non-rigid registration technique is used to update the new suture plan. More specifically, the algorithm incorporates (i) continuous 3D location detection of NIR markers placed on deformable tissue before the procedure, (ii) generating a uniform and consistent suture placement plan using 3D path planning based on locations of the NIR markers, and (iii) updating the remaining suture plan after each completed stitch using a non-rigid registration technique to account for tissue deformation during anastomosis. We implement this novel path planning algorithm with the Smart Tissue Autonomous Robot (STAR) to perform semi-autonomous anastomosis on synthetic vaginal cuff phantoms. We experimentally demonstrate the accuracy and consistency, and compare the results against manual laparoscopic suturing performed by an experienced surgeon.

2 Methods

2.1 Testbed

The experimental testbed is shown in Fig. 1a. The testbed includes a robotic laparoscopic suturing tool mounted on a 7-DOF KUKA Med lightweight arm (KUKA AG, Augsburg Germany), and a dual camera system to provide visual feedback. The dual camera imaging system (Fig. 1b) consists of a Realsense D415 RGBD camera (Intel Corp., Santa Clara, California) to detect 3D tissue surface information and a 845 nm ± 55 nm 2D NIR camera (Basler Inc., Exton, PA) for detecting the NIR markers. Both camera coordinate systems are registered onto the robot coordinate systems using standard hand-eye calibration with a calibration rod and a checkerboard. A light source with 760 nm high power light-emitting diode (North Coast Technical Inc., Chesterland, OH) was used to excite the fluorophore, enabling the markers to be visualized with the NIR camera. The dual camera setup extracts the 3D position of the biocompatible NIR markers by ray tracing the NIR marker positions (Fig. 1b) via a co-registered point cloud from the 3D camera (as shown in Fig. 1c and d) [1]. The 3D location of the markers are used later in a suture planning strategy, described in Sect. 2.3.

A multi-axis suturing tool was used by STAR to perform the semi-autonomous suturing tasks in this study (Fig. 1a). The structure and mechanism of the tool is detailed in Section III [7]. The tool is modified from the commercial Proxisure suturing device (Ethicon Inc. Somerville, NJ, United States) and incorporates a multi axis motor pack to independently control the motions of the tool (e.g., pitch, roll, and needle drive). Each motor includes an encoder for precise positioning using EPOS2 controllers (Maxon Motors, Sachseln, Switzerland), and are integrated using a controller area network (CAN). 2-0 polyester Ethibond suture was used with the suturing tool for this experiment.

Fig. 2.
figure 2

The autonomous control loop.

2.2 Surgical Task and Evaluation Criteria

An expert surgeon and STAR were asked to perform anastomosis consisting of a knot and running stitches in synthetic vaginal cuff tissue (3-Dmed, Ohio, United States). Synthetic tissue, with 5 cm diameter and 5 mm wall thickness, was chosen because it is specifically designed for anastomosis training in vaginal cuff closure, is easier to maintain and analyze than ex vivo tissue, and is reproducible. The test sample was secured within a surgical ring using two stay sutures from the side and two alligator clips from the bottom to simulate attachment of the vaginal cuff to surrounding tissue. NIR markers were manually placed on the surface of the tissue edge using a syringe, prior to starting the semi-autonomous robotic anastomosis. Both manual and semi-autonomous robotic anastomoses were compared on (i) task completion time, (ii) suture spacing (i.e. the distance between consecutive stitches), and (iii) bite size (i.e. distance between where a stitch enters into tissue and the tissue edge). The latter two measures are related to post surgical complications including dehiscence and infection [5]. T-test and Levene’s test statistic were used to compare the equality of averages and variances, respectively, in suture spacing and bite size across modalities.

2.3 Control System

The block diagram of the autonomous robot controller is shown in Fig. 2. In the control loop, a dual camera system obtains the point cloud and the 3D coordinates of the NIR markers in real-time via a ray tracing technique detailed in [1]. The suture planning strategy, developed in this paper, utilizes the locations of NIR markers and performs 3D path planning via non-rigid registration to update suture points (later described in Sect. 2.3). The suture points are used in a high-level suturing logic and task planner that plans a sequence of robot motions to complete suturing sub-steps such as knot and running stitches on the planned points. The high-level suturing logic and task planner is similar to the algorithms developed and tested in [4, 7] and hence details are not repeated here. A low-level controller based on Fast Research Interface (FRI) [9] and Kinematics and Dynamics Library (KDL) in Open Robot Control Systems (OROCOS) [13] guarantees the motion control of the robot to perform each subtask. More specifically, the low-level controller solves the inverse kinematics of the robot motion to perform subtasks, such as reaching the target suturing point and tensioning the suture, and also generates smooth task-space and joints-space motion trajectories for the robot to follow.

Fig. 3.
figure 3

The pipeline for suturing planning strategies. (a) Suture plan initialization: determines equally spaced suture points \(\mathbf {s}_1\) using 3D path planning with NIR markers \(\mathbf {m}_1\) on circular tissue edge, and (b) suture plan update strategy: updates a new suture plan \(\mathbf {s}_k\) based on the previous suture plan \(\mathbf {s}_{k-1}\), current markers \(\mathbf {m}_k\), and previous markers \(\mathbf {m}_{k-1}\) via non-rigid registration technique.

Suture Planning Strategies. The pipeline of suture planning strategies developed in this paper is shown in Fig. 3 which includes (i) a suture plan initialization strategy (i.e. Fig. 3a), and (ii) a suture plan update strategy (i.e. Fig. 3b). The initialization strategy determines the location of suture points before placing any stitch on the test sample while the suture plan update strategy updates the suture point after placing each stitch and repeats it until the end of the suturing task. The two planning strategies are detailed in the following.

Suture Plan Initialization: For this step, the goal is to determine a set of initial suturing points based on the location of the NIR markers on the circular edge of target tissue. To this aim, we let the operator select the NIR markers via a user interface and then an autonomous suture path planner equally distributes a set of suture points on the upper and lower halves of the circular tissue edge as follows.

Denote \(\mathbf {m}_k\) as the augmented vector of the 3D position of NIR markers \(\mathbf {m}_{i_k} \in \mathcal {R}^3\) at the planning stage k, where \(i\in \{1,2...\text {M}\}\) and M is the total number of markers. Similarly, denote \(\mathbf {s}_k\) as the augmented vector of the 3D position of suture points \(\mathbf {s}_{j_k} \in \mathcal {R}^3\), where \(j\in \{1,2...\text {N}\}\) and N is the total number of suture points. In the first stage of the suture planning (i.e. \(k=1\)), the path planner places identical suture points with even spacing on both the upper and lower halves of the circular tissue edge (Fig. 3a). A human operator first selects NIR markers via mouse clicks (Fig. 1c), and the dual camera system extracts the 3D position of the selected markers \(\mathbf {m}_1\) in Fig. 3a using the method detailed in [1]. Next, the path planner utilizes a 3D path planning method to calculate a 3D path between the NIR markers on the tissue edge. The 3D path planning method used in this paper is based on the 3D path planning that finds a shortest 3D path between two different positions on a point cloud [3]. The calculated 3D paths are grouped, to represent the upper and lower paths on the tissue edge (i.e. green lines in Fig. 3a). Suture points \(\mathbf {s}_1\) are then distributed with equal spacing using the total length l of each grouped path (i.e. breaking l into \(n+1\) equal sections). We determine the positions of \(\mathbf {s}_{1_j}\) as a point on the path that is the closest point to the location \(\frac{j}{n+1}l\) from the start point of the 3D path (e.g. at \(\frac{1}{4}l\), \(\frac{2}{4}l\), and \(\frac{3}{4}l\) for \(n=3\)). In our experiment, six suture points (i.e. \(n=6\)) were placed on each of the tissue edges (i.e. pink points in Fig. 3a), and the 3D position of the suture points \(\mathbf {s}_k\) were passed to the high-level suturing logic and task planner to perform the corresponding suturing sub-tasks (e.g. reaching the tissue, firing a stitch, tensioning the suture, etc.).

Suture Plan Update Strategy: The goal of this step is to estimate the new position of the initial planned suture points after each completed stitch (Fig. 3b). We use the 3D position of NIR markers as landmarks after the tissue deformation, resulting from the placement of stitch and the tensioning of tissue, with the thin-plate spline robust point matching (TPS-RPM) algorithm [10] to obtain an updated position of the suturing points according to what follows.

At the planning stage k (i.e. planning for the \(k^{\text {th}}\) suture, \(k>1\)), the planner uses the TPS-RPM algorithm to find a warping function f that estimates a non-rigid transformation between the current NIR marker positions (i.e. \(\mathbf {m}_k\)) and their corresponding positions at the previous planning stage (i.e. \(\mathbf {m}_{k-1}\)). To this aim, the algorithm iteratively solves the following least-square problem

$$\begin{aligned} \underset{f}{\text {min}}~\sum _{i=1}^{\text {M}}\Vert \mathbf {y}_{i_{k}} - f(\mathbf {m}_{i_{k-1}})\Vert ^2+\lambda {T}\Vert Lf\Vert ^2 \end{aligned}$$
(1)

where f is a non-rigid transformation that maps \(\mathbf {m}_{k-1}\) to \(\mathbf {y}_k\). More specifically, \(\mathbf {y}_k\) is encoded with a correspondence matrix and \(\mathbf {m}_{k}\), and is the augmented vector of newly estimated positions \(\mathbf {y}_{i_k} \in \mathcal {R}^3\) at the planning stage k, where \(i\in \{1,2...\text {M}\}\). In (1), \(\Vert Lf\Vert ^2\) is a TPS smoothness constraint preventing arbitrary mappings, \(\lambda \) controls the weighting of the smoothness constraint, and T is a annealing parameter that reduces gradually every iteration. The algorithm iterates to solve for (1), as \(\mathbf {y}_k\) become closer to \(\mathbf {m}_k\), and update the warping function f until T reaches \(T_{final}\) and terminates the algorithm. Once the iteration is terminated, the planner utilizes f to transform \(\mathbf {s}_{k-1}\) to \(\mathbf {s}_k\). Finally, the suture points \(\mathbf {s}_k\) are passed to the high-level suturing logic and task planner.

Table 1. Comparison of the results.
Fig. 4.
figure 4

Results of anastomosis. (a) Semi-autonomous, and (b) manual.

3 Experiment and Result

Using the vaginal cuff phantom, four experiments were conducted for the manual suturing method and two experiments for the semi-autonomous robotic suturing. The manual suturing experiments were completed by an expert surgeon using a laparoscopic trainer. For the semi-autonomous suturing, the control and planning algorithm detailed earlier in Sect. 2.3 were implemented via the testbed in Fig. 1a. During the semi-autonomous suturing, since the robotic suturing is performed by a single robotic arm, a surgical assistant uses a laparoscopic needle driver to manage the excess length of the suture thread. The experiment data detailed in Sect. 2.2 were recorded and evaluated.

The results of the manual and robotic anastomoses are summarized in Table 1. Representative results through semi-autonomous and manual suturing are shown in Fig. 4. In Table 1, the average task completion time for the semi-autonomous method is 299.0 s longer than manual method. More specific, STAR performed slower in completing both knot and stitch than the surgeon (i.e. 56 s longer in the knot, and 35.2 s longer in per stitch, \(p<0.001\)). The average suturing spacing obtained by STAR is statistically l.27 mm smaller than the manual, \(p < 0.05\). Moreover, variance of suture spacing is statistically less for STAR than human (\(p < 0.05\)) which indicates that STAR placed the running stitches more uniformly (2.6 times better) compared to the human surgeon. Regarding the bite size, STAR produced statistically 2.98 mm larger average bite size than manual, \(p < 0.001\). The variance of the bite size of STAR is statistically smaller than human (\(p < 0.05\)), which means STAR is more consistent in bite depth (2.4 times better) compared to manual surgery.

4 Discussion and Conclusions

The results indicate that the STAR was more consistent in bite depth and suturing spacing. However, in terms of the closure completeness, shown in Fig. 4, manual anastomosis was better than semi-autonomous anastomosis. Two possible causes lead to such a difference. First, two knots, a first and final knot, were tied for manual anastomosis but only a first knot was tied for semi-autonomous anastomosis. Second, lack of experience on handling the suture thread on tying a knot affects the closure completeness in semi-autonomous anastomosis. We will incorporate functionality for tying a final knot as well as practicing more on the task of knot tying to improve the closure completeness in the future.

We present a novel suturing planning strategy that advances STAR system to perform semi-autonomous anastomosis on deformable tissue. Specifically, a 3D path planning algorithm and a non-rigid registration algorithm are used during the planning process to achieve the suturing task. Moreover, NIR markers are placed on test sample as tracking landmarks with using our imaging system to function suture planning strategies. The experiments demonstrate that our proposed strategy can achieve a better consistency in suture spacing and bite size compared to the manual laparoscopic method. Future work will include improving the speed of the suturing procedures, evaluating proposed planning strategy on actual tissue, and integrating laparoscopic constrains to proceed to in vivo anastomoses tests.