Skip to main content

Advertisement

Springer Nature Link
Log in

A CNN-based prototype method of unstructured surgical state perception and navigation for an endovascular surgery robot

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

Performance of robot-assisted endovascular surgery (ES) remains highly dependent on an individual surgeon’s skills, due to common adoption of master-slave robotic structure. Surgeons’ skill modeling and unstructured surgical state perception pose prohibitive challenges for an autonomous ES robot. In this paper, a novel convolutional neural network (CNN)-based framework is proposed to address these challenges for navigation of an ES robot based on surgeons’ skill learning. An operating action probability estimator is proposed by integrating a two-dimensional CNN, with which the features of a surgical state image are extracted and then directly mapped to the action probability. A one-dimensional CNN with multi-input is developed to recognize the guide wire operating force condition. An eye-hand collaborative servoing algorithm is proposed to combine the outputs of these two networks and to control the robot under a closed-loop architecture. A real-world ES robot is employed for data collection and task performance evaluation in laboratory condition. Compared with the state of the art, the CNN-based method shows its capability of adapting to different situations and achieves similar success rate and average operating time. Robotic operation performs similar operating trajectory and maintains similar level of operating force with manual operation. The CNN-based method can be easily extended to many other surgical robots.

A surgeon’s guide wire operating skills in endovascular surgery (ES) is learned by the proposed CNN-based method. Then, the learned model is used for autonomous control of a ES robot with surgical state input (images and operating force).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Jenny DJohn JVDD, Paul B (2010) Current technology on minimally invasive surgery and interventional techniques. IEEE International Conference on Instrumentation Control and Automation. https://doi.org/10.1109/ICA.2011.6130118

  2. Hedyeh RT, Christopher JP, Yang GZ (2013) Current and emerging robot-assisted endovascular catheterization technologies: a review. Ann Biomed Eng 42:697–715. https://doi.org/10.1007/s10439-013-0946-8

    Article  Google Scholar 

  3. Yu S, Shuxiang G, Xuanchun Y, Linshuai Z, Yu W, Hirata H, Ishihara H (2017) Design and performance evaluation of a haptic interface based on MR fluids for endovascular tele-surgery. Microsyst Technol 24(10):909–918. https://doi.org/10.1007/s00542-017-3404-y

    Article  Google Scholar 

  4. Yu S, Shuxiang G, Xuanchun Y, Linshuai Z, Yu W, Hirata H, Ishihara H, Tamiya T (2018) Performance evaluation of a robot-assisted catheter operating system with haptic feedback. Biomed Microdevices 20:50. https://doi.org/10.1007/s10544-018-0294-4

    Article  Google Scholar 

  5. Xuanchun Y, Shuxinag G, Nan X, Tamiya T, Hirata H, Ishihara H (2016) Safety operation consciousness realization of a MR fluids-based novel haptic interface for teleoperated catheter minimally invasive neuro surgery. IEEE/ASME Trans Mechatronics 21:1043–1054. https://doi.org/10.1109/TMECH.2015.2489219

    Article  Google Scholar 

  6. Zhou C, Xie L, Shen X, Luo M, Wu Z, Gu L (2015) Cardiovascular interventional surgery virtual training platform and its preliminary evaluation. Int J Med Robotics Comput Assist Surg 11:375–387. https://doi.org/10.1002/rcs.1627

    Article  Google Scholar 

  7. Xianqiang B, Shuxiang G, Nan X, Youxiang L, Cheng Y, Yuhua J (2018) A cooperation of catheters and guide-wires-based novel remote-controlled endovascular interventional robot. Biomed Microdevices 20:20. https://doi.org/10.1007/s10544-018-0261-0

    Article  Google Scholar 

  8. Xianqiang B, Shuxiang G, Nan X, Li Y, Shi L (2018) Compensatory force measurement and multimodal force feedback for remote-controlled vascular interventional robot. Biomed Microdevices 20:74. https://doi.org/10.1007/s10544-018-0318-0

    Article  CAS  Google Scholar 

  9. Jian G, Shuxiang G, Lin S, Peng W (2016) Design and performance evaluation of a novel robotic catheter system for endovascular interventional surgery. Microsyst Technol 22:2167–2176. https://doi.org/10.1007/s00542-015-2659-4

    Article  Google Scholar 

  10. Yang X, Wang H, Sun L, Hongnian Y (2015) Operation and force analysis of the guide wire in a minimally invasive. Chin J Mech Eng 28:249–257. https://doi.org/10.3901/cjme.2014.1229.181

    Article  Google Scholar 

  11. Wang K, Chen B, Xu X (2016) Design and control method of surgical robot for endovascular intervention operation. ROBIO. https://doi.org/10.1109/ROBIO.2016.7866331

  12. Chaonan Z, Shuxiang G, Nan X, Jiaqing W, Youxiang L, Yuhua J (2018) Transverse microvibration-based guide wire drag reduction evaluation for vascular interventional application. Biomed Microdevices 20:69. https://doi.org/10.1007/s10544-018-0315-3

    Article  Google Scholar 

  13. Yi L, Shuxiang G, Hirata H, Ishihara H, Tamiya T (2018) Development of a powered variable-stiffness exoskeleton device for elbow rehabilitation. Biomed Microdevices 20:64. https://doi.org/10.1007/s10544-018-0312-6

    Article  Google Scholar 

  14. Kim Y, Cheng S, Diakite M, Gullapalli R, Simard J, Desai J (2017) Toward the development of a flexible mesoscale MRI-compatible neurosurgical continuum robot. IEEE Trans Robot 33:1386–1397. https://doi.org/10.1109/tro.2017.2719035

    Article  PubMed  PubMed Central  Google Scholar 

  15. Kim Y, Cheng S, Desai J (2017) Active stiffness tuning of a spring-based continuum robot for MRI-guided neurosurgery. IEEE Trans Robot 34:18–28. https://doi.org/10.1109/TRO.2017.2750692

    Article  PubMed  PubMed Central  Google Scholar 

  16. Jin G, Shuxiang G, Maoxun L, Tamiya T (2018) A marker-based contactless catheter-sensing method to detect surgeons’ operations for catheterization training systems. Biomed Microdevices 20:76. https://doi.org/10.1007/s10544-018-0321-5

    Article  CAS  Google Scholar 

  17. Shuxiang G, Yuan W, Nan X, Youxiang L, Yuhua J (2018) Study on real-time force feedback with a master-slave interventional surgical robotic system. Biomed Microdevices 20(37). https://doi.org/10.1007/s10544-018-0278-4

  18. Linshuai Z, Shuxiang G, Huadong Yu YS (2017) Performance evaluation of a strain-gauge force sensor for a haptic robot-assisted catheter operating system. Microsyst Technol 23:5041–5050. https://doi.org/10.1007/s00542-017-3380-2

    Article  Google Scholar 

  19. Jian G, Shuxiang G, Yang Y (2016) Design and characteristics evaluation of a novel teleoperated robotic catheterization system with force feedback for endovascular interventional surgery. Biomed Microdevices 18:76. https://doi.org/10.1007/s10544-016-0100-0

    Article  Google Scholar 

  20. Vandini A, Bergeles C, Glocker B, Giataganas P, Guangzhong Y (2017) Unified tracking and shape estimation for concentric tube robots. IEEE Trans Robot 33:901–915 1109/TRO.2017.2690977

    Article  Google Scholar 

  21. Jian G, Xiaolaing J, Shuxiang G (2018) Study of the operational safety of an endovascular interventional surgical robotic system. Micromachines 9:119. https://doi.org/10.3390/mi9030119

    Article  Google Scholar 

  22. Linshuai Z, Shuxaing G, Huadong Y, Yu S, Tamiya T, Hirata H, Ishihara H (2018) Design and performance evaluation of collision protection-based safety operation for a haptic robot-assisted catheter operating system. Biomed Microdevices 20:22. https://doi.org/10.1007/s10544-018-0266-8

    Article  Google Scholar 

  23. Murali A, Sen S, Kehoe B, Garg A, McFarland S, Patil S, Boyd WD, Lim S, Abbeel P, Goldberg K (2015) Learning by observation for surgical subtasks: multilateral cutting of 3D viscoelastic and 2D Orthotropic Tissue Phantoms. ICRA. https://doi.org/10.1109/ICRA.2015.7139344

  24. Shademan A, Decker RS, Opfermann JD, Leonard S, Krieger A, Kim PCW (2018) Supervised autonomous robotic soft tissue surgery. Sci Transl Med 8:337ra64. https://doi.org/10.1126/scitranslmed.aad9398

    Article  Google Scholar 

  25. Xianqiang B, Shuxiang G, Nan X, Youxiang L, Cheng Y, Rui S, Jinxin C, Yuhua J, Xinke L, Keyun L (2018) Operation evaluation in-human of a novel remote-controlled endovascular interventional robot. Biomed Microdevices 20:34. https://doi.org/10.1007/s10544-018-0277-5

    Article  CAS  Google Scholar 

  26. Yan Z, Shuxiang G, Nan X, Yuxin W, Youxiang L, Yuhua J (2018) Operating force information on-line acquisition of a novel slave manipulator for vessel interventional surgery. Biomed Microdevices 20(33). https://doi.org/10.1007/s10544-018-0275-7

  27. Hui L, Jianxin Y, Huchen L, Guangdong T (2018) Acquiring and sharing tacit knowledge based on interval 2-tuple linguistic assessments and extended fuzzy petrinets. Int J Uncertain Fuzz 26:43–65. https://doi.org/10.1142/S0218488518500034

    Article  Google Scholar 

  28. Kassahun Y, Yu B, Tibebu A, Stoyanov D, Giannarou S, Metzen J, Poorten E (2016) Surgical robotics beyond enhanced dexterity instrumentation: a survey of machine learning techniques and their role in intelligent and autonomous surgical actions. Int J Comput Ass Rad 11:847–847. https://doi.org/10.1007/s11548-015-1340-9

    Article  Google Scholar 

  29. Mayer H, Gomez F, Wierstra D, Nagy I (2006) A system for robotic heart surgery that learns to tie knots using recurrent neural networks. IROS 22(13):543–548. https://doi.org/10.1163/156855308X360604

    Article  Google Scholar 

  30. Seita D, Krishnan S, Fox R, McKinley S, Canny J, Goldberg K (2018) Fast and reliable autonomous surgical debridement with cable-driven robots using a two-phase calibration procedure. ICRA. https://doi.org/10.1109/ICRA.2018.8460583

  31. Kassahun Y, Yu B, Poorten E (2013) Learning catheter-aorta interaction model using joint probability densities. Joint Workshop on New Technologies for CRAS. https://lirias.kuleuven.be/handle/123456789/415899. Accessed 12 Nov 2016

  32. Hedyeh RT, Jindong L, Su-Lin L, Colin B, Guangzhong Y (2013) Learning-based modeling of endovascular navigation for collaborative robotic catheterization. Med Image Comput Comput Assist Interv 16:369–377. https://doi.org/10.1007/978-3-642-40763-5_46

  33. Hedyeh RT, Jindong L, Christopher JP, Colin B, Guangzhong Y (2014) Hierarchical HMM based learning of navigation primitives for cooperative robotic endovascular catheterization. Med Image Comput Comput Assist Interv 17:496–503. https://doi.org/10.1007/978-3-319-10404-162

    Article  Google Scholar 

  34. Jeremy DB, Conor EOB, Sarah CL, Kristoffel RD, David IL, Katherine JK (2017) Using contact forces and robot arm accelerations to automatically rate surgeon skill at peg transfer. IEEE Trans Biomed Eng 64:2263–2675. https://doi.org/10.1109/TBME.2016.2634861

    Article  Google Scholar 

  35. Wenqiang C, Jindong L, Hedyeh RT, Celia R, Colin B, Guangzhong Y (2018) Learning-based endovascular navigation through the use of non-rigid registration for collaborative robotic catheterization. Int J Comput Assist Radiol Surg 13:855–864. https://doi.org/10.1007/s11548-018-1743-5

    Article  Google Scholar 

  36. Olga R, Jia D, Hao S, Jonathan K, Sanjeev S, Sean M, Zhiheng H, Andrej K, Aditya K, Michael B, Alexander CB, Feifei L (2015) ImageNet large scale visual recognition challenge. Int J Comput Vis 115:211–252. https://doi.org/10.1007/s11263-015-0816-y

    Article  Google Scholar 

  37. David S, Aja H, Chris JM, Arthur G, Laurent S, George van den D, Julian S, Ioannis A, Veda P, Marc L, Sander D, Dominik G, John N, Nal K, Ilya S, Timothy L, Madeleine L, Koray K, Thore G, Demis H (2016) Mastering the game of Go with deep neural networks and tree search. Nature 529:484–489. https://doi.org/10.1038/nature16961

    Article  CAS  Google Scholar 

  38. Volodymyr M, Koray K, David S, Alex G, Ioannis A, Daan W, Martin R (2013) Playing Atari with deep reinforcement learning. NIPS Deep Learning Workshop, arXiv:1312.5602

  39. Qi D, Hao C, Lequan Y, Jing Q, Pheng Ann H (2017) Multilevel contextual 3-D CNNs for false positive reduction in pulmonary nodule detection. IEEE Trans Biomed Eng 64:1558–1567. https://doi.org/10.1109/TBME.2016.2613502

    Article  Google Scholar 

  40. Sergey L, Chelsea F, Trevor D, Pieter A (2016) End-to-end training of deep visuomotor policies. J Mach Learn Res 17:1–40. https://doi.org/10.1016/j.dam.2015.09.011

    Article  Google Scholar 

  41. Roberto C, Andrew O, Dinesh J, Justin L, Wenzhen Y, Jitendra M, Edward HA, Sergey L (2017) More than a feeling: learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters 3:3300–3307. https://doi.org/10.1109/LRA.2018.2852779

    Article  Google Scholar 

  42. Sergey L, Peter P, Alex K, Julian I, Deirdre Q (2017) Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int J Robot Res 37:421–436. https://doi.org/10.1177/0278364917710318

    Article  Google Scholar 

  43. Fangyi Z, Jürgen L, Michael M, Ben U, Peter C (2015) Towards vision-based deep reinforcement learning for robotic motion control. In: Australasian Conference on Robotics and Automation, https://doi.org/10.1016/j.seppur.2006.09.006

  44. Mingxin J, Rong L, Jian J, Binjie Q (2017) Extracting contrast-filled vessels in X-ray angiography by graduated RPCA with motion coherency constraint. Pattern Recogn 63:653–666. https://doi.org/10.1016/j.patcog.2016.09.042

    Article  Google Scholar 

  45. Binjie Q, Mingxin J, Dongdong H, Yisong L, Qiegen L, Yueqi Z, Song D, Jun Z, Baowei F (2019) Accurate vessel extraction via tensor completion of background layer in X-ray coronary angiograms. Pattern Recognit 87:38–54. https://doi.org/10.1016/j.patcog.2018.09.015

    Article  Google Scholar 

  46. Pierre A, Ihor S, Daniel R, Wiro JN, Adriaan M, Theo VW (2017) A hidden Markov model for 3D catheter tip tracking with 2D X-ray catheterization sequence and 3D rotational angiography. IEEE Trans Med Imaging 36:757–768. https://doi.org/10.1016/0.1109/TMI.2016.2625811

    Article  Google Scholar 

  47. Li W, Xiaoliang X, Guilin B, Zengguang H, Xiaoran C, Pusit P (2017) Guide-wire detection using region proposal network for X-ray image-guided navigation. IEEE IJCNN:3169–3175. https://doi.org/10.1109/IJCNN.2017.7966251

  48. Christian S, Vincent V, Sergey I, Jonathon S, Zbigniew W (2016) Rethinking the inception architecture for computer vision. IEEE CVPR:2818–2826. https://doi.org/10.1109/CVPR.2016.308

  49. Nasr-Esfahania E, Karimia N, Jafaria MH, Soroushmehrbc SMR, Samaviabc S, Nallamothuc BK, Najarianbc K (2018) Segmentation of vessels in angiograms using convolutional neural networks. Biomedical Signal Processing and Control 40:240–251. https://doi.org/10.1016/j.bspc.2017.09.012

    Article  Google Scholar 

  50. Kragic D From active perception to deep learning. Sci Robot 3:eaav1778 http://robotics.sciencemag.org/. Accessed 8 Dec 2016

  51. Guangzhong Y, James C, Kevin C, Eric D, James D, Pierre ED, Nobuhiko H, Peter K, Sylvain M, Rajni VP, Veronica JS, Russell HT (2017) Medical robotics—regulatory, ethical, and legal considerations for increasing levels of autonomy. Sci. Robot. 2. https://doi.org/10.1126/scirobotics.aam8638

  52. Pierre BR, Maura P, Hawkeye K, Guangzhong Y (2016) Hubot: a three state human-robot collaborative framework for bimanual surgical tasks based on learned models. ICRA. https://doi.org/10.1109/ICRA.2016.7487198

Download references

Funding

This research is partly supported by the National High-tech R&D Program (863 Program) of China (No. 2015AA043202) and National Key Research and Development Program of China (2017YFB1304401).

Author information

Authors and Affiliations

  1. Key Laboratory of Convergence Biomedical Engineering System and Healthcare Technology, The Ministry of Industry and Information Technology, School of Life Science, Beijing Institute of Technology, No.5, Zhongguancun South Street, Haidian District, Beijing, 100081, China

    Yan Zhao, Shuxiang Guo, Yuxin Wang, Jinxin Cui, Youchun Ma, Yuwen Zeng, Liwei Shi & Nan Xiao

  2. Faculty of Engineering, Kagawa University, 2217-20 Hayashi-cho, Takamatsu, Kagawa, 760-8521, Japan

    Shuxiang Guo

  3. Department of Interventional Neuroradiology, Beijing Neurosurgical Institute and Beijing Tiantan Hospital, Capital Medical University, Beijing, 100050, China

    Xinke Liu, Yuhua Jiang & Youxinag Li

Authors
  1. Yan Zhao
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Shuxiang Guo
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Yuxin Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Jinxin Cui
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Youchun Ma
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Yuwen Zeng
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Xinke Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  8. Yuhua Jiang
    View author publications

    You can also search for this author inPubMed Google Scholar

  9. Youxinag Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  10. Liwei Shi
    View author publications

    You can also search for this author inPubMed Google Scholar

  11. Nan Xiao
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding authors

Correspondence to Shuxiang Guo, Liwei Shi or Nan Xiao.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, Y., Guo, S., Wang, Y. et al. A CNN-based prototype method of unstructured surgical state perception and navigation for an endovascular surgery robot. Med Biol Eng Comput 57, 1875–1887 (2019). https://doi.org/10.1007/s11517-019-02002-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-019-02002-0

Keywords