Skip to main content
SpringerLink
Log in

Transfer of learned dynamics between different surgical robots and operative configurations

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Using the da Vinci Research Kit (dVRK), we propose and experimentally demonstrate transfer learning (Xfer) of dynamics between different configurations and robots distributed around the world. This can extend recent research using neural networks to estimate the dynamics of the patient side manipulator (PSM) to provide accurate external end-effector force estimation, by adapting it to different robots and instruments, and in different configurations, with additional forces applied on the instruments as they pass through the trocar.

Methods

The goal of the learned models is to predict internal joint torques during robot motion. First, exhaustive training is performed during free-space (FS) motion, using several configurations to include gravity effects. Second, to adapt to different setups, a limited amount of training data is collected and then the neural network is updated through Xfer.

Results

Xfer can adapt a FS network trained on one robot, in one configuration, with a particular instrument, to provide comparable joint torque estimation for a different robot, in a different configuration, using a different instrument, and inserted through a trocar. The robustness of this approach is demonstrated with multiple PSMs (sampled from the dVRK community), instruments, configurations and trocar ports.

Conclusion

Xfer provides significant improvements in prediction errors without the need for complete training from scratch and is robust over a wide range of robots, kinematic configurations, surgical instruments, and patient-specific setups.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Hollerbach J, Khalil W, Gautier M (2008) Model identification. Springer Handbook of Robotics, Springer, Berlin Heidelberg, pp. 321–344

  2. Khosla PK, Kanade T (1985) Parameter identification of robot dynamics. In: Conference on decision and control (CDC). IEEE, pp 1754–1760

  3. Smith AC, Hashtrudi-Zaad K (2005) Application of neural networks in inverse dynamics based contact force estimation. In: Conference on control applications (CCA). IEEE, pp 1021–1026

  4. Kazanzides P, Chen Z, Deguet A, Fischer GS, Taylor RH, DiMaio SP (2014) An open-source research kit for the da Vinci® surgical system. In: IEEE International conference on robotics and automation (ICRA). Hong Kong, China, pp 6434–6439

  5. D’Ettorre C, Mariani A, Stilli A, Rodriguez y Baena F, Valdastri P, Deguet A, Kazanzides P, Taylor RH, Fischer GS, DiMaio SP, Menciassi A, Stoyanov D (2021) Accelerating surgical robotics research: a review of 10 years with the da Vinci Research Kit. IEEE Robot Autom Mag 6:66

  6. Hannaford B, Rosen J, Friedman DW, King H, Roan P, Cheng L, Glozman D, Ma J, Kosari SN, White L (2012) Raven-II: an open platform for surgical robotics research. IEEE Trans Biomed Eng 60(4):954–959

    Article  Google Scholar 

  7. Mahvash M, Gwilliam J, Agarwal R, Vagvolgyi B, Su L.-M, Yuh DD, Okamura AM (2008) Force-feedback surgical teleoperator: controller design and palpation experiments. In: 2008 Symposium on haptic interfaces for virtual environment and teleoperator systems. IEEE, pp 465–471

  8. Fontanelli GA, Ficuciello F, Villani L, Siciliano B (2017) Modelling and identification of the da Vinci research kit robotic arms. In: IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 1464–1469

  9. Sang H, Yun J, Monfaredi R, Wilson E, Fooladi H, Cleary K (2017) External force estimation and implementation in robotically assisted minimally invasive surgery. Int J Med Robot Comp Assist Surg 13(2):1824

    Article  Google Scholar 

  10. Piqué F, Boushaki MN, Brancadoro M, De Momi E, Menciassi A (2019) Dynamic modeling of the a Vinci Research Kit arm for the estimation of interaction wrench. In: International symposium on medical robotics (ISMR). IEEE, pp 1–7

  11. Wang Y, Gondokaryono R, Munawar A, Fischer GS (2019) A convex optimization-based dynamic model identification package for the da Vinci Research Kit. IEEE Robot Autom Lett 4(4):3657–3664

    Article  Google Scholar 

  12. Haghighipanah M, Miyasaka M, Hannaford B (2017) Utilizing elasticity of cable-driven surgical robot to estimate cable tension and external force. IEEE Robot Autom Lett 2(3):1593–1600

    Article  Google Scholar 

  13. Miyasaka M, Haghighipanah M, Li Y, Hannaford B (2016) Hysteresis model of longitudinally loaded cable for cable driven robots and identification of the parameters. In: International conference on robotics and automation (ICRA). IEEE, pp 4051–4057

  14. Li Y, Miyasaka M, Haghighipanah M, Cheng L, Hannaford B (2016) Dynamic modeling of cable driven elongated surgical instruments for sensorless grip force estimation. In: International conference on robotics and automation (ICRA). IEEE, pp 4128–4134

  15. Guillaume P, Pintelon R, Schoukens J (1996) Accurate estimation of multivariable frequency response functions. IFAC Proc 29(1):4351–4356

    Google Scholar 

  16. Östring M, Gunnarsson S, Norrlöf M (2003) Closed-loop identification of an industrial robot containing flexibilities. Control Eng Pract 11(3):291–300

    Article  Google Scholar 

  17. Wernholt E (2004) On multivariable and nonlinear identification of industrial robots. PhD thesis, Linköping University, Linköping, Sweden

  18. Karakasoglu A, Sudharsanan SI, Sundareshan MK (1993) Identification and decentralized adaptive control using dynamical neural networks with application to robotic manipulators. IEEE Trans Neural Netw 4(6):919–930

    Article  CAS  Google Scholar 

  19. Pham D, Oh S (1994) Adaptive control of a robot using neural networks. Robotica 12(6):553–561

    Article  Google Scholar 

  20. Su H, Qi W, Hu Y, Sandoval J, Zhang L, Schmirander Y, Chen G, Aliverti A, Knoll A, Ferrigno G, De Momi E (2019) Towards model-free tool dynamic identification and calibration using multi-layer neural network. Sensors 19(17):3636

    Article  Google Scholar 

  21. de Gea Fernández J, Yu B, Bargsten V, Zipper M, Sprengel H (2020) Design, modelling and control of novel series-elastic actuators for industrial robots. In: Actuators, vol 9. Multidisciplinary Digital Publishing Institute, p 6

  22. Bargsten V, de Gea Fernandez J, Kassahun Y (2016) Experimental robot inverse dynamics identification using classical and machine learning techniques. In: International symposium on robotics (ISR). VDE, pp 1–6

  23. Shareef Z, Mohammadi P, Steil J (2016) Improving the inverse dynamics model of the KUKA LWR IV+ using independent joint learning. IFAC-PapersOnLine 49(21):507–512

    Article  Google Scholar 

  24. Lu J, Behbood V, Hao P, Zuo H, Xue S, Zhang G (2015) Transfer learning using computational intelligence: a survey. Knowl Based Syst 80:14–23

    Article  Google Scholar 

  25. Yilmaz N, Wu JY, Kazanzides P, Tumerdem U (2020) Neural network based inverse dynamics identification and external force estimation on the da Vinci Research Kit. In: International conference on robotics and automation (ICRA). IEEE, pp 1387–1393

  26. Wu JY, Yilmaz N, Tumerdem U, Kazanzides P (2021) Robot force estimation with learned intraoperative correction. In: 2021 International symposium on medical robotics (ISMR). IEEE, pp 1–7

Download references

Acknowledgements

This work was supported in part by NSF OISE 1927354. Haoying (Jack) Zhou from Worcester Polytechnic Institute (PI: G. Fischer), Scotty Chung from Wake Forest University (PI: P. Brown), Tamás D. Nagy from Óbuda University (PI: T. Haidegger), Solene Dietsch from University College London (PI: D. Stoyanov), and Andrea Mariani from Sant’Anna School of Advanced Studies (PI: A. Menciassi) provided training data for this study.

Author information

Authors and Affiliations

  1. Department of Mechanical Engineering, Marmara University, Istanbul, 34722, Turkey

    Nural Yilmaz & Ugur Tumerdem

  2. Department of Computer Science, Johns Hopkins University, Baltimore, MD, 21218, USA

    Nural Yilmaz, Jintan Zhang & Peter Kazanzides

Authors
  1. Nural Yilmaz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jintan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Peter Kazanzides
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ugur Tumerdem
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nural Yilmaz.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Human and animal participants

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yilmaz, N., Zhang, J., Kazanzides, P. et al. Transfer of learned dynamics between different surgical robots and operative configurations. Int J CARS 17, 903–910 (2022). https://doi.org/10.1007/s11548-022-02601-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-022-02601-7

Keywords

Search

Navigation