Skip to main content
Springer Nature Link
Log in

Vision-based hand–eye calibration for robot-assisted minimally invasive surgery

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

The knowledge of laparoscope vision can greatly improve the surgical operation room (OR) efficiency. For the vision-based computer-assisted surgery, the hand–eye calibration establishes the coordinate relationship between laparoscope and robot slave arm. While significant advances have been made for hand–eye calibration in recent years, efficient algorithm for minimally invasive surgical robot is still a major challenge. Removing the external calibration object in abdominal environment to estimate the hand–eye transformation is still a critical problem.

Methods

We propose a novel hand–eye calibration algorithm to tackle the problem which relies purely on surgical instrument already in the operating scenario for robot-assisted minimally invasive surgery (RMIS). Our model is formed by the geometry information of the surgical instrument and the remote center-of-motion (RCM) constraint. We also enhance the algorithm with stereo laparoscope model.

Results

Promising validation of synthetic simulation and experimental surgical robot system have been conducted to evaluate the proposed method. We report results that the proposed method can exhibit the hand–eye calibration without calibration object.

Conclusion

Vision-based hand–eye calibration is developed. We demonstrate the feasibility to perform hand–eye calibration by taking advantage of the components of surgical robot system, leading to the efficiency of surgical OR.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

References

  1. Freschi C, Ferrari V, Melfi F, Ferrari M, Mosca F, Cuschieri A (2013) Technical review of the da Vinci surgical telemanipulator. Int J Med Robot Comput Assist Surg 9(4):396–406

    Article  CAS  Google Scholar 

  2. Su H, Shuai Li, Jagadesh M, Bascetta L, Ferrigno G, De ME (2019) Manipulability optimization control of a serial redundant robot for robot-assisted minimally invasive surgery. In: 2019 IEEE international conference on robotics and automation (ICRA). IEEE, pp 1323–1328

  3. Kassahun Y, Yu B, Tibebu AT, Stoyanov D, Giannarou S, Metzen JH, Poorten EV (2016) Surgical robotics beyond enhanced dexterity instrumentation: a survey of machine learning techniques and their role in intelligent and autonomous surgical actions. Int J Comput Assist Radiol Surg 11(4):553–568

    Article  Google Scholar 

  4. Kehoe B, Kahn G, Mahler J, Kim J, Lee A, Lee A, Nakagawa K, Patil S, Boyd WD, Abbeel P, Goldberg K (2013) Autonomous multilateral debridement with the Raven surgical robot. In: 2013 IEEE international conference on robotics and automation (ICRA). IEEE, pp 1432–1439

  5. Murali A, Sen S, Kehoe B, Garg A, Goldberg K (2015) Learning by observation for surgical subtasks: multilateral cutting of 3D viscoelastic and 2D orthotropic tissue phantoms. In: 2015 IEEE international conference on robotics and automation (ICRA). IEEE, pp 1202–1209

  6. Thananjeyan B, Garg A, Krishnan S, Chen C, Goldberg K (2017) Multilateral surgical pattern cutting in 2D orthotropic gauze with deep reinforcement learning policies for tensioning. In: 2017 IEEE international conference on robotics and automation (ICRA). IEEE, pp 2371–2378

  7. Allan M, Ourselin S, Hawkes DJ, Kelly JD, Stoyanov D (2018) 3-D pose estimation of articulated instruments in robotic minimally invasive surgery. IEEE Trans Med Imaging 37(5):1204–1213

    Article  CAS  Google Scholar 

  8. Allan M, Chang PL, Ourselin S, Hawkes DJ, Sridhar A, Kelly J, Stoyanov D (2015) Image based surgical instrument pose estimation with multi-class labelling and optical flow. In: 2015 international conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 331-338

  9. Du X, Kurmann T, Chang PL, Allan M, Ourselin S, Sznitman R, Stoyanov D (2018) Articulated multi-instrument 2-D pose estimation using fully convolutional networks. IEEE Trans Med Imaging 37(5):1276–1287

    Article  Google Scholar 

  10. Li W, Dong M, Lu N, Lou X, Sun P (2018) Simultaneous robot–world and hand–eye calibration without a calibration object. Sensors 18(11):3949

    Article  Google Scholar 

  11. Park FC, Martin BJ (1994) Robot sensor calibration: solving AX = XB on the Euclidean group. IEEE Trans Robot Autom 10(5):717–721

    Article  Google Scholar 

  12. Andreff N, Horaud R, Espiau B (1999) On-line hand–eye calibration. In: Second international conference on 3-D digital imaging and modeling (Cat. No. PR00062). IEEE, pp 430–436

  13. Horaud R, Dornaika F (1995) Hand–eye calibration. Int J Robot Res 14(3):195–210

    Article  Google Scholar 

  14. Daniilidis K (1999) Hand–eye calibration using dual quaternions. Int J Robot Res 18(3):286–298

    Article  Google Scholar 

  15. Zhang Z, Zhang L, Yang GZ (2017) A computationally efficient method for hand–eye calibration. Int J Comput Assist Radiol Surg 12(10):1775–1787

    Article  Google Scholar 

  16. Thompson S, Stoyanov D, Schneider C, Gurusamy K, Ourselin S, Davidson B, Clarkson MJ (2016) Hand–eye calibration for rigid laparoscopes using an invariant point. Int J Comput Assist Radiol Surg 11(6):1071–1080

    Article  Google Scholar 

  17. Morgan I, Jayarathne U, Rankin A, Peters TM, Chen EC (2017) Hand–eye calibration for surgical cameras: a procrustean perspective-n-point solution. Int J Comput Assist Radiol Surg 12(7):1141–1149

    Article  Google Scholar 

  18. Malti A, Barreto J P (2010) Robust hand–eye calibration for computer aided medical endoscopy. In: 2010 IEEE international conference on robotics and automation. IEEE, pp 5543–5549

  19. Pachtrachai K, Vasconcelos F, Dwyer G, Hailes S, Stoyanov D (2019) Hand–eye calibration with a remote centre of motion. IEEE Robot Autom Lett 4(4):3121–3128

    Article  Google Scholar 

  20. Andreff N, Horaud R, Espiau B (2001) Robot hand–eye calibration using structure-from-motion. Int J Robot Res 20(3):228–248

    Article  Google Scholar 

  21. Pachtrachai K, Allan M, Pawar V, Hailes S, Stoyanov D (2016) Hand–eye calibration for robotic assisted minimally invasive surgery without a calibration object. In: 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 2485–2491

  22. Strobl KH, Hirzinger G (2006) Optimal hand–eye calibration. In: 2006 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 4647–4653

  23. Malti A (2013) Hand–eye calibration with epipolar constraints: application to endoscopy. Robot Autonom Syst 61(2):161–169

    Article  Google Scholar 

  24. Malti A, Barreto JP (2013) Hand–eye and radial distortion calibration for rigid endoscopes. Int J Med Robot Comput Assist Surg 9(4):441–454

    Article  Google Scholar 

  25. Wang Z, Liu Z, Ma Q, Cheng A, Liu YH, Kim S, Taylor RH (2017) Vision-based calibration of dual RCM-based robot arms in human-robot collaborative minimally invasive surgery. IEEE Robot Autom Lett 3(2):672–679

    Article  Google Scholar 

  26. Pachtrachai K, Vasconcelos F, Chadebecq F, Chadebecq F, Allan M, Hailes S, Pawar V, Stoyanov D (2018) Adjoint transformation algorithm for hand–eye calibration with applications in robotic assisted surgery. Ann Biomed Eng 46(10):1606–1620

    Article  Google Scholar 

  27. Corke P (2017) Robotics vision and control: fundamental algorithms In MATLAB® second. Springer, Berlin

    Book  Google Scholar 

  28. Bouget D, Allan M, Stoyanov D, Jannin P (2017) Vision-based and marker-less surgical tool detection and tracking: a review of the literature. Med Image Anal 35:633–654

    Article  Google Scholar 

  29. Kaehler A, Bradski GR (2016) Learning OpenCV 3. O’Reilly Media, Sebastopol

    Google Scholar 

  30. Stockman George C (2001) Computer vision. Prentice Hall, Upper Saddle River

    Google Scholar 

Download references

Acknowledgements

This study was supported by Foundation for Innovative Research Groups of the National Natural Science Foundation of China (Grant No. 51521003) and National Science Foundation of China (Grant No. 61803341), State Key Laboratory of Robotics and Systems (Grant No. SKLRS202009B).

Author information

Authors and Affiliations

  1. State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin, China

    Yanwen Sun, Bo Pan, Yongchen Guo & Yili Fu

  2. School of Mechanical Engineering and Automation, Zhejiang Sci‐Tech University, Hangzhou, China

    Guojun Niu

Authors
  1. Yanwen Sun
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Bo Pan
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Yongchen Guo
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Yili Fu
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Guojun Niu
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Bo Pan.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

This article does not contain patient data.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, Y., Pan, B., Guo, Y. et al. Vision-based hand–eye calibration for robot-assisted minimally invasive surgery. Int J CARS 15, 2061–2069 (2020). https://doi.org/10.1007/s11548-020-02245-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-020-02245-5

Keywords