Safety-aware robotic steering of a flexible endoscope for nasotracheal intubation

https://doi.org/10.1016/j.bspc.2022.104504Get rights and content

Highlights

  • A safety-aware robot control framework for steering an endoscope is presented.

  • Lumen center detection is performed to provide guidance for endoscope steering.

  • Relative depth from the endoscope tip to the lumen surface is estimated.

Abstract

Automation in robotic endoscopy surgery is an important but challenging research topic. The flexible deformation of the endoscope makes it difficult for surgeons to steer it, especially in a narrow workspace. The safety of endoscope steering has been a major concern in a robotic endoscope. This study presents a safety-aware control framework to enable a robotic endotracheal intubation system (RNIS) to automatically steer the flexible endoscope during nasotracheal intubation (NTI). With the endoscopic image as feedback, we used the fusion of image intensity and optical flow measurements to detect the lumen center. A velocity-based orientation controller is designed to adjust the orientation of the endoscope tip to track the lumen center. To ensure surgical safety, the relative depth from the endoscope tip to the lumen surface is calculated from endoscopic images. With the relative depth as feedback, the feed motion of the endoscope is controlled to prevent the collision between the endoscope tip and its surrounding tissue. The position and orientation of the endoscope tip can be adjusted by the RNIS during NTI. Finally, extensive experiments based on a training manikin were performed. The results indicate that the proposed control framework enables the RNIS to successfully steer the flexible endoscope along the upper respiratory tract with high safety. The performance of lumen center detection is good according to the annotation of experienced users.

Introduction

Nasotracheal intubation (NTI) is one of the most popular procedures in anesthesia and it is considered the gold standard for securing the airway of patients [1], [2]. In NTI, a flexible endoscope is inserted into the trachea from the nasal cavity to maintain an open airway of the patient. Typically, the surgeon holds and steers the endoscope by hand while observing the images. To smoothly move along the respiratory tract, the position and orientation of the endoscope need to be accurately controlled so that the lumen center is in the center of the endoscope view. Fig. 1(a) shows the anatomy of the upper respiratory tract, which consists of four key landmarks, namely the nasal cavity, throat, glottis, and trachea. The endoscopic intubation path is along the midline of the respiratory tract and is indicated by a dashed red line. However, the flexible deformation of the endoscope makes it difficult for surgeons to steer it, especially in a narrow workspace. Moreover, the endoscopes only provide 2D images but not depth images, which may lead to collisions between the endoscope with tissues and the loss of the lumen center. Hence, apart from stable tracking of the lumen center, another major concern in the robotic endoscope is safety. Improper operation of the endoscope may lead to serious complications, such as bleeding and asphyxia [3]. According to an emergency department report [4], the success rates of NTI can be as low as 70% of the time at the first attempt. Therefore, there is an urgent need for safety steering of the flexible endoscope during NTI.

Perceiving environment from endoscopic images is a critical aspect of autonomous endoscope steering [5], [6]. Robotic steering of the flexible endoscope requires the lumen center as guidance. There are some methods to recognize and detect anatomical features from endoscopic images [7], [8]. For instance, to guide the manual operation of the endoscope, Boehler et al. [9] trained an SVM classifier with Haar features to detect anatomical features including the glottis and the trachea. However, their work only provides a rough estimation of the location of the lumen center with a bounding box. Some works extract image features, such as image intensity [10] and optical flow [11], to generate navigation. Bell et al. [12] employed an optical flow method to estimate scene motion and localize the lumen center for robotic colonoscopy. The optical flow-based methods have high computational efficiency and can be used for real-time robot control. However, the computation of optical flow is susceptible to illumination and motion blur, so its reliability is poor. Reilink et al. [13], [14] detected the lumen center based on image intensity analysis. The optical flow-based and intensity-based methods have their advantages and disadvantages. Typically, the optical flow-based method is sensitive to the image texture and light condition. The intensity-based method cannot deal with occlusion. On the other hand, Jia et al. [15] employed a deep neural network for polyp detection. The detected polyp was used as the approaching target of the wireless capsule endoscope. However, their method required a large number of samples for network training to ensure network performance. To achieve the robust detection of the lumen center, this work proposes a weighted fusion strategy to effectively utilize the measurements of the optical flow and image intensity.

To ensure surgical safety, the collision between the endoscope tip and its surrounding tissue should be avoided. Therefore, the depth from the endoscope tip to the surrounding tissue needs to be monitored during NTI [16], [17]. However, due to the size limitation of the endoscope, additional sensors, such as the stereo camera, cannot be easily placed in the endoscope, making the calculation of the actual depth challenging [18]. Typically, the endoscope outputs 2D images for visualization through a monocular camera, which is placed at its tip. In the field of computing vision, multi-view stereo methods [19], such as structure from motion (SfM) [20] and simultaneous localization and mapping (SLAM) [21], have been used to estimate depth information from monocular images. However, due to the low texture of endoscopic images, the accuracy of the SfM or SLAM method is severely affected. Moreover, these methods are not designed for feedback in the robot control loop. Some recent works used deep learning methods for depth estimation from monocular images [22], [23]. Because acquiring ground truth depth maps for network training is challenging, it is difficult to apply deep learning methods to practical robot control [24], the distance between the light source and the tissue is approximately inversely proportional to the light intensity. Based on this, we propose the estimation of the relative depth from the endoscope tip to the surrounding tissue based on image intensity analysis. The relative depth is then used as feedback for a robot to prevent collision between the endoscope tip and its surrounding tissue.

This paper aims to enable a robot to automatically steer the flexible endoscope along the human upper respiratory tract without colliding with the lumen surface. Field-of-view control and collision avoidance are two critical technical aspects of automatic robotic endoscopy. For this purpose, a safety-aware control framework is proposed. The location of the lumen center and the relative depth from the endoscope tip to the lumen surface are first calculated from the RGB image. Using visual feedback, the endoscope is controlled to track the lumen center for better intervention and visualization. Additionally, With the estimated relative depth, the feed motion of the endoscope is controlled for collision avoidance and thus improve surgical safety. The performance of the proposed approaches was demonstrated with experiments. The contributions of this work can be summarized as follows:

  • A safety–aware control framework is proposed to enable a robot to automatically steer the flexible endoscope along the human upper respiratory tract without colliding with the lumen surface. The position and orientation of the endoscope are adjusted with visual feedback.

  • A lumen center detection is designed to find the location of the lumen center in 2D endoscopic images. Multiple features are extracted to improve the detection performance.

  • The relative depth from the endoscope tip to the lumen surface is estimated from 2D images to provide feedback for collision avoidance control. The proposed method does not require knowledge of anatomy and extra sensors.

The rest of this letter is organized as follows. In Section 2, the visual navigation and control strategies are introduced. Three key computations, i.e., lumen center detection, relation depth estimation, and visual-feedback control, are described in detail. Experimental results are presented in Section 3. Finally, Section 4 concludes this work.

Section snippets

Methods

This section presents the individual components of the safety–aware control framework that enable the RNIS to steer the flexible endoscope along the upper respiratory tract during NTI. The RNIS and the overall workflow are first introduced in Section 2.1, while the methods of lumen center detection and relative depth estimation are presented in Sections 2.2 Lumen center detection based on weighted fusion, 2.3 Relative depth estimation. Section 2.4 introduces the visual-feedback control

Experiment

This section begins with a performance evaluation of the interactive trajectory lumen center detection and relative depth estimation. Then, using real-world experiments, the effectiveness of the proposed control framework is demonstrated. The experimental results and their analysis are provided.

Conclusion

In this paper, a safety–aware control framework was introduced to enable the robotic endotracheal intubation system (RNIS) to automatically steer the flexible endoscope along the upper respiratory tract during nasotracheal intubation (NTI). Two control requirements of the endoscope, i.e., lumen center tracking and collision avoidance, were accommodated through the proposed methods. The location of the lumen center and the relative depth of the endoscope to the lumen surface were calculated from

CRediT authorship contribution statement

Zhen Deng: Conceptualization, Methodology, Software, Validation. Peijie Jiang: Methodology, Writing – original draft, Supervision. Yuxin Guo: Visualization, Investigation. Shengzhan Zhang: Data curation, Software, Validation. Ying Hu: Data curation, Writing – review & editing. Xiaochun Zheng: Writing – review & editing. Bingwei He: Data curation, Writing – review.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This research was partly supported by the National Natural Science Foundation of China (No. 62003089), Major scientific research projects of health in Fujian Province, China (No. 2021ZD01003) and the Natural Science Foundation of Fujian Province, China (No. 2020J01455).

References (33)

  • MinJ.K. et al.

    Overview of deep learning in gastrointestinal endoscopy

    Gut Liver

    (2019)
  • Van der StapN. et al.

    The use of the focus of expansion for automated steering of flexible endoscopes

  • BellC.S. et al.

    Six DOF motion estimation for teleoperated flexible endoscopes using optical flow: A comparative study

  • ReilinkR. et al.

    Image-based flexible endoscope steering

  • ReilinkR. et al.

    Three-dimensional pose reconstruction of flexible instruments from endoscopic images

  • BoehlerQ. et al.

    REALITI: A robotic endoscope automated via laryngeal imaging for tracheal intubation

    IEEE Trans. Med. Robot. Bionics

    (2020)
  • Cited by (4)

    • Design and validation of a novel tracheal intubation robot

      2023, 2023 IEEE International Conference on Robotics and Biomimetics, ROBIO 2023
    • An Original Design of Robotic-Assisted Flexible Nasotracheal Intubation System

      2023, 2023 IEEE International Conference on Robotics and Biomimetics, ROBIO 2023
    • Eliciting context-oriented NFR constraints and conflicts in robotic systems

      2023, Innovations in Systems and Software Engineering

    This research was partly supported by the National Natural Science Foundation of China (No. 62003089), Fujian Provincial Health Technology Project (No. 2021ZD01003) and the Natural Science Foundation of Fujian Province (No. 2020J01455).

    View full text