Skip to main content

Advertisement

SpringerLink
Log in

Robot-assisted flexible needle insertion using universal distributional deep reinforcement learning

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Flexible needle insertion is an important minimally invasive surgery approach for biopsy and radio-frequency ablation. This approach can minimize intraoperative trauma and improve postoperative recovery. We propose a new path planning framework using multi-goal deep reinforcement learning to overcome the difficulties in uncertain needle–tissue interactions and enhance the robustness of robot-assisted insertion process.

Methods

This framework utilizes a new algorithm called universal distributional Q-learning (UDQL) to learn a stable steering policy and perform risk management by visualizing the learned Q-value distribution. To further improve the robustness, universal value function approximation is leveraged in the training process of UDQL to maximize generalization and connect to diagnosis by adapting fast re-planning and transfer learning.

Results

Computer simulation and phantom experimental results show our proposed framework can securely steer flexible needles with high insertion accuracy and robustness. The framework also improves robustness by providing distribution information to clinicians for diagnosis and decision making during surgery.

Conclusions

Compared with previous methods, the proposed framework can perform multi-target needle insertion through single insertion point qunder continuous state space model with higher accuracy and robustness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Hamad GG, Curet M (2010) Minimally invasive surgery. Am J Surg 199(2):263–265

    Article  Google Scholar 

  2. Tan X, Chng C-B, Ye S, Lim K-B, Chui C-K (2019) Robot-assisted training in laparoscopy using deep reinforcement learning. IEEE Robot Autom Lett 4(2):485–492

    Article  Google Scholar 

  3. Hiraki T, Kamegawa T, Matsuno T, Sakurai J, Kirita Y, Matsuura R, Yamaguchi T, Sasaki T, Mitsuhashi T, Komaki T (2017) Robotically driven ct-guided needle insertion: preliminary results in phantom and animal experiments. Radiology 285(2):454–461

    Article  Google Scholar 

  4. Abolhassani N, Patel R, Moallem M (2007) Needle insertion into soft tissue: a survey. Med Eng Phys 29(4):413–431

    Article  Google Scholar 

  5. Schaul T, Horgan D, Gregor K, Silver D (2015a) Universal value function approximators. In: International conference on machine learning. pp 1312–1320

  6. Tan X, Yu P, Lim K-B, Chui C-K (2018) Robust path planning for flexible needle insertion using Markov decision processes. Int J Comput Assist Radiol Surg 13(9):1439–1451

    Article  Google Scholar 

  7. Duan B, Wen R, Chng C-B, Wang W, Liu P, Qin J, Peneyra JL, Chang SK-Y, Heng P-A, Chui C-K (2015) Image-guided robotic system for radiofrequency ablation of large liver tumor with single incision. In: 2015 12th International conference on ubiquitous robots and ambient intelligence (URAI). IEEE, pp 284–289

  8. DiMaio SP, Salcudean SE (2005) Needle steering and motion planning in soft tissues. IEEE Trans Biomed Eng 52(6):965–974

    Article  Google Scholar 

  9. Taylor RH, Menciassi A, Fichtinger G, Fiorini P, Dario P (2016) Medical robotics and computer-integrated surgery. In: Siciliano B, Khatib O (eds) Springer handbook of robotics. Springer, Cham, pp 1657–1684

    Chapter  Google Scholar 

  10. Liu P, Qin J, Duan B, Wang Q, Tan X, Zhao B, Jonnathan PL, Chui C-K, Heng P-A (2018) Overlapping radiofrequency ablation planning and robot-assisted needle insertion for large liver tumors. Int J Med Robot Comput Assist Surg 15:e1952

    Article  Google Scholar 

  11. Chatelain P, Krupa A, Navab N (2015) 3d ultrasound-guided robotic steering of a flexible needle via visual servoing. In: IEEE international conference on robotics and automation, ICRA’15

  12. Alterovitz R, Siméon T, Goldberg KY (2007) The stochastic motion roadmap: a sampling framework for planning with Markov motion uncertainty. In: Robotics: science and systems, vol 3, pp 233–241

  13. Alterovitz R, Branicky M, Goldberg K (2008) Motion planning under uncertainty for image-guided medical needle steering. Int J Robot Res 27(11–12):1361–1374

    Article  Google Scholar 

  14. Morar A, Moldoveanu F, Gröller E (2012) Image segmentation based on active contours without edges. In: 2012 IEEE 8th international conference on intelligent computer communication and processing. IEEE, pp 213–220

  15. Chen X, Nguyen BP, Chui C-K, Ong S-H (2016) Automated brain tumor segmentation using kernel dictionary learning and superpixel-level features. In: 2016 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, pp 002547–002552

  16. Sutton RS, Barto AG (1998) Introduction to reinforcement learning, vol 135. MIT Press, Cambridge

    Google Scholar 

  17. Deng L, Yu D (2014) Deep learning: methods and applications. Found Trends® Signal Process 7(3–4):197–387

    Article  Google Scholar 

  18. Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, Hubert T, Baker L, Lai M, Bolton A (2017) Mastering the game of go without human knowledge. Nature 550(7676):354

    Article  CAS  Google Scholar 

  19. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv:1707.06347

  20. Fu YB, Chui CK, Teo CL (2013) Liver tissue characterization from uniaxial stress–strain data using probabilistic and inverse finite element methods. J Mech Behav Biomed Mater 20:105–112

    Article  CAS  Google Scholar 

  21. Fu YB, Chui CK (2014) Modelling and simulation of porcine liver tissue indentation using finite element method and uniaxial stress–strain data. J Biomech 47(10):2430–2435

    Article  CAS  Google Scholar 

  22. Qu C, Mannor S, Xu H (2018) Nonlinear distributional gradient temporal-difference learning. arXiv:1805.07732

  23. Bellemare MG, Dabney W, Munos R (2017) A distributional perspective on reinforcement learning. arXiv:1707.06887

  24. Andrychowicz M, Wolski F, Ray A, Schneider J, Fong R, Welinder P, McGrew B, Tobin J, Abbeel OP, Zaremba W (2017) Hindsight experience replay. In: Advances in neural information processing systems, pp 5048–5058

  25. Schaul T, Quan J, Antonoglou I, Silver D (2015b) Prioritized experience replay. arXiv:1511.05952

  26. Tamar A, Di Castro D, Mannor S (2016) Learning the variance of the reward-to-go. J Mach Learn Res 17(1):361–396

    Google Scholar 

  27. Yang L, Wen R, Qin J, Chui C-K, Lim K-B, Chang SK-Y (2010) A robotic system for overlapping radiofrequency ablation in large tumor treatment. IEEE/ASME Trans Mechatron 15(6):887–897

    Article  Google Scholar 

  28. Tan X, Chng C-B, Duan B, Ho Y, Wen R, Chen X, Lim K-B, Chui C-K (2017) Cognitive engine for robot-assisted radio-frequency ablation system. Acta Polytech Hung 14(1):129–145

    Google Scholar 

  29. Tan X, Chng C-B, Duan B, Ho Y, Wen R, Chen X, Lim K-B, Chui C-K (2016) Design and implementation of a patient-specific cognitive engine for robotic needle insertion. In: 2016 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, pp 000560–000565

  30. Van Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double Q-learning. In: AAAI, vol 16, pp 2094–2100

  31. Wasserstein RL, Lazar NA (2016) The asa’s statement on p values: context, process, and purpose. Am Stat 70(2):129–133

    Article  Google Scholar 

  32. Leong F, Huang W-H, Chui C-K (2013) Modeling and analysis of coagulated liver tissue and its interaction with a scalpel blade. Med Biol Eng Comput 51(6):687–695

    Article  Google Scholar 

  33. Tokuda J, Song S-E, Fischer GS, Iordachita II, Seifabadi R, Cho NB, Tuncali K, Fichtinger G, Tempany CM, Hata N (2012) Preclinical evaluation of an MRI-compatible pneumatic robot for angulated needle placement in transperineal prostate interventions. Int J Comput Assist Radiol Surg 7(6):949–957

    Article  Google Scholar 

  34. Krieger A, Susil RC, Fichtinger G, Atalar E, Whitcomb LL (2004) Design of a novel MRI compatible manipulator for image guided prostate intervention. In: IEEE international conference on robotics and automation, proceedings on ICRA’04, vol 1. IEEE, pp 377–382

  35. Schouten MG, Ansems J, Renema WKJ, Bosboom D, Scheenen TWJ, Fütterer JJ (2010) The accuracy and safety aspects of a novel robotic needle guide manipulator to perform transrectal prostate biopsies. Med Phys 37(9):4744–4750

    Article  Google Scholar 

Download references

Acknowledgements

The last author would like to acknowledge the contribution of A/Prof Stephen Chang of Mount Elizabeth Hospital, Singapore for his input on surgeries and medical education.

Funding

The research and development of the prototype Image-guide Radio-frequency Ablation Surgical System was supported in parts by Research Grants from Singapore Agency of Science and Technology (A*Star) and Ministry of Education, Singapore respectively.

Author information

Authors and Affiliations

  1. Department of Mechanical Engineering, National University of Singapore, Singapore, 117575, Singapore

    Xiaoyu Tan, Yonggu Lee, Chin-Boon Chng, Kah-Bin Lim & Chee-Kong Chui

Authors
  1. Xiaoyu Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yonggu Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chin-Boon Chng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kah-Bin Lim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chee-Kong Chui
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoyu Tan.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Human and animal rights

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tan, X., Lee, Y., Chng, CB. et al. Robot-assisted flexible needle insertion using universal distributional deep reinforcement learning. Int J CARS 15, 341–349 (2020). https://doi.org/10.1007/s11548-019-02098-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-019-02098-7

Keywords

Search

Navigation