Skip to main content
SpringerLink
Log in

A gradient-based neural network accelerated for vision-based control of an RCM-constrained surgical endoscope robot

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

This paper presents an accelerated gradient-based neural network (GNN) to achieve visual servoing of a surgical endoscope robot. A KUKA LWR 4+ robot with seven joints is used to serve as an endoscope holder. Kinematic mapping is established between the joint space of the robot and the image space of the camera. For surgical applications, the motions of the KUKA robot are constrained with respect to a remote-center-of-motion (RCM) point. Meanwhile, each joint of the KUKA robot has its own physical limits (e.g., joint-angle and joint velocity limits) that cannot be violated. By taking into account the kinematic equation, RCM constraints and physical limits, a control scheme possessing a quadratic programming (QP) formulation is constructed. To solve the QP problem, an inverse-free GNN model is accelerated to be finite-time convergent using a powerful activation function. Mathematical derivations of the accelerated GNN model and theoretical proofs relevant to the finite-time convergence are detailed. Comparative validations are conducted with the superior convergence performance of the accelerated GNN model substantiated. The effectiveness of the proposed GNN solution for vision-based control of the surgical endoscope is verified with RCM constraints and physical limits respected simultaneously.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Zhang Y, Li W, Yu M, Wu H, Li J (2013) Encoder based online motion planning and feedback control of redundant manipulators. Control Eng Pract 21:1277–1289

    Article  Google Scholar 

  2. Elshabasy MM, Mohamed KT, Ata AA (2017) Power optimization of planar redundant manipulator moving along constrained-end trajectory using hybrid techniques. Alex Eng J 56:439–447

    Article  Google Scholar 

  3. Zhang Y, Yan X, Chen D, Guo D, Li W (2016) QP-based refined manipulability-maximizing scheme for coordinated motion planning and control of physically constrained wheeled mobile redundant manipulators. Nonlinear Dyn 85:245–261

    Article  MathSciNet  Google Scholar 

  4. Hägele M, Nilsson K, Pires JN, Bischoff R (2016) Industrial robotics. Springer International Publishing, Cham, pp 1385–1422

    Google Scholar 

  5. Weede O, Mönnich H, Müller B, Wörn H (2011) An intelligent and autonomous endoscopic guidance system for minimally invasive surgery. In: Proceedings of IEEE international conference on robotics and automation, pp. 5762–5768

  6. García-Martínez A, Lled’® L D, Badesa F J, García N, Sabater-Navarro J M (2014) Integration of heterogeneous robotic systems in a surgical scenario. Proceedings of IEEE RAS/EMBS International conference on biomedical robotics and biomechatronics. pp. 24–27

  7. Kuo CH, Dai JS, Dasgupta P (2012) Kinematic design considerations for minimally invasive surgical robots: an overview. Int J Med Robot Comput Assist Surg 8:127–145

    Article  Google Scholar 

  8. Guo D, Xu F, Yan L (2018) New pseudoinverse-based path-planning scheme with PID characteristic for redundant robot manipulators in the presence of noise. IEEE Trans Control Syst Technol 26:2008–2019

    Article  Google Scholar 

  9. Guo D, Xu F, Yan L, Nie Z, Shao H (2018) A new noise-tolerant obstacle avoidance scheme for motion planning of redundant robot manipulators. Front Neurorobot 12:51

    Article  Google Scholar 

  10. Zhao H, Kolathaya S, Ames A D (2014) Quadratic programming and impedance control for transfemoral prosthesis. Proceedings of IEEE international conference on robotics and automation. pp. 1341–1347

  11. Farshidian F, Jelavić E, Winkler A W, Buchli J (2017) Robust whole-body motion control of legged robots. Proceedings of IEEE/RSJ international conference on intelligent robots and systems. pp. 4589–4596

  12. Flacco F, De Luca A, Khatib O (2015) Control of redundant robots under hard joint constraints: saturation in the null space. IEEE Trans Robot 31:637–654

    Article  Google Scholar 

  13. Truma Y (2014) Linear programming: theory, algorithms and applications. Nova Science Publishers, New York

    Google Scholar 

  14. Zhang Y, Yi C (2011) Zhang neural networks and neural-dynamic method. Nova Science Publishers, New York

    Google Scholar 

  15. Li W (2018) A recurrent neural network with explicitly definable convergence time for solving time-variant linear matrix equations. IEEE Trans Ind Inform 14:5289–5298

    Article  Google Scholar 

  16. Li W, Su Z, Tan Z (2019) A variable-gain finite-time convergent recurrent neural network for time-variant quadratic programming with unknown noises endured. IEEE Trans Ind Inform 15:5330–5340

    Article  Google Scholar 

  17. Xiao L (2019) A finite-time convergent Zhang neural network and its application to real-time matrix square root finding. Neural Comput Appl 31:793–800

    Article  Google Scholar 

  18. Shi Y, Jin L, Li S, Qiang J (2020) Proposing, developing and verification of a novel discrete-time zeroing neural network for solving future augmented Sylvester matrix equation. J Frankl Inst 357:3636–3655

    Article  MathSciNet  Google Scholar 

  19. Leung Y, Chen K-Z, Jiao Y-C, Gao X-B, Leung KS (2001) A new gradient-based neural network for solving linear and quadratic programming problems. IEEE Trans Neural Netw 12:1074–1083

    Article  Google Scholar 

  20. Han Q, Liao L-Z, Qi H, Qi L (2001) Stability analysis of gradient-based neural networks for optimization problems. J Global Optim 19:363–381

    Article  MathSciNet  Google Scholar 

  21. Boyd S, Vandenberghe L (2004) Convex optimization. Cambridge University Press, Cambridge

    Book  Google Scholar 

  22. Zhang Z, Zheng L, Yu J, Li Y, Yu Z (2017) Three recurrent neural networks and three numerical methods for solving a repetitive motion planning scheme of redundant robot manipulators. IEEE/ASME Trans Mechatron 22:1423–1434

    Article  Google Scholar 

  23. Liao-McPherson D, Huang M, Kolmanovsky I (2019) A regularized and smoothed Fischer-Burmeister method for quadratic programming with applications to model predictive control. IEEE Trans Autom Control 64:2937–2944

    Article  MathSciNet  Google Scholar 

  24. Li S, Li Y, Wang Z (2013) A class of finite-time dual neural networks for solving quadratic programming problems and its k-winners-take-all application. Neural Netw 39:27–39

    Article  Google Scholar 

  25. Su H, Hu Y, Karimi HR, Knoll A, Ferrigno G, Momi ED (2020) Improved recurrent neural network-based manipulator control with remote center of motion constraints: experimental results. Neural Netw 131:291–299

    Article  Google Scholar 

  26. Khan AH, Li S, Cao X (2021) Tracking control of redundant manipulator under active remote center-of-motion constraints: an RNN-based metaheuristic approach. Sci China Inform Sci 64:132203

    Article  MathSciNet  Google Scholar 

  27. Li W, Chiu PWY, Li Z (2020) An accelerated finite-time convergent neural network for visual servoing of a flexible surgical endoscope with physical and RCM constraints. IEEE Trans Neural Netw Learn Syst 31:5272–5284

    Article  MathSciNet  Google Scholar 

  28. Zhang X, Li W, Ng W Y, Huang Y, Xian Y, Chiu P W Y, Li Z (2021) An autonomous robotic flexible endoscope system with a DNA-inspired continuum mechanism. Proceedings of international conference on robotics and automation (ICRA), Accepted

  29. Su H, Sandoval J, Vieyres P, Poisson G, Ferrigno G, Momi ED (2018) Safety-enhanced collaborative framework for tele-operated minimally invasive surgery using a 7-DoF torque-controlled robot. Int J Contr Autom Syst 16:2915–2923

    Article  Google Scholar 

  30. Corke P (2011) Robotics, vision and control: fundamental algorithms in MATLAB. Springer, Berlin

    Book  Google Scholar 

  31. Marinho M M, Harada K, Mitsuishi M (2017) Comparison of remote center-of-motion generation algorithms. Proceedings of IEEE/SICE international symposium on system integration. pp. 668–673

  32. Zhang Y, Wang J (2002) A dual neural network for convex quadratic programming subject to linear equality and inequality constraints. Phys Lett A 298:271–278

    Article  MathSciNet  Google Scholar 

  33. Piersanti G (2012) The macroeconomic theory of exchange rate crises. Oxford University Press, Oxford

    Book  Google Scholar 

  34. Li W, Xiao L, Liao B (2020) A finite-time convergent and noise-rejection recurrent neural network and its discretization for dynamic nonlinear equations solving. IEEE Trans Cybern 50:3195–3207

    Article  Google Scholar 

  35. Gallier J (2020) Geometric methods and applications: for computer science and engineering. Springer, New York

    MATH  Google Scholar 

  36. Miao P, Shen Y, Xia X (2014) Finite time dual neural networks with a tunable activation function for solving quadratic programming problems and its application. Neurocomputing 143:80–89

    Article  Google Scholar 

  37. Freese M, Singh S, Ozaki F, Matsuhira N (2010) Virtual robot experimentation platform V-REP: A versatile 3-D robot simulator. Proceedings of international conference on simulation, modeling, and programming for autonomous robots. pp. 51–62

Download references

Acknowledgements

This work is supported in part by the National Natural Science Foundation of China under Grant 62066015, in part by the Hunan Natural Science Foundation of China under Grants 2020JJ4510 and 2020JJ4511, and in part by the Research Foundation of Education Bureau of Hunan Province, China, under Grant 20A396.

Author information

Authors and Affiliations

  1. College of Information Science and Engineering, Jishou University, Jishou, 416000, Hunan, China

    Weibing Li, Luyang Han, Xiao Xiao, Bolin Liao & Chen Peng

Authors
  1. Weibing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Luyang Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiao Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bolin Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chen Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bolin Liao.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, W., Han, L., Xiao, X. et al. A gradient-based neural network accelerated for vision-based control of an RCM-constrained surgical endoscope robot. Neural Comput & Applic 34, 1329–1343 (2022). https://doi.org/10.1007/s00521-021-06465-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06465-x

Keywords

Search

Navigation