Skip to main content

Advertisement

SpringerLink
Log in

A Fuzzy Reinforcement Learning Approach for Continuum Robot Control

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Continuum robots (CRs) hold great potential for many medical and industrial applications where compliant interaction within the potentially confined environment is required. However, the navigation of CRs poses several challenges due to their limited actuation channels and the hyper-flexibility of their structure. Environmental uncertainty and characteristic hysteresis in such procedures add to the complexity of their operation. Therefore, the quality of trajectory tracking for continuum robots plays an essential role in the success of the application procedures. While there are a few different actuation configurations available for CRs, the focus of this paper will be placed on tendon-driven manipulators. In this research, a new fuzzy reinforcement learning (FRL) approach is introduced. The proposed FRL-based control parameters are tuned by the Taguchi method and evolutionary genetic algorithm (GA) to provide faster convergence to the Nash Equilibrium. The approach is verified through a comprehensive set of simulations using a Cosserat rod model. The results show a steady and accurate trajectory tracking capability for a CR.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Hasanzadeh, S., Janabi-Sharifi, F.: Model-based force estimation for intracardiac catheters. IEEE/ASME Trans. Mech. 21(1), 154–162 (2015)

    Google Scholar 

  2. Hasanzadeh, S., Janabi-Sharifi, F.: An efficient static analysis of continuum robots. J. Mech. Robot. 6(3), 1–5 (2014)

    Article  Google Scholar 

  3. Ganji, Y., Janabi-Sharifi, F., Cheema, A.N.: Robot-assisted catheter manipulation for intracardiac navigation. Int. J. Comput. Ass. Rad. 4(4), 307–315 (2009)

    Google Scholar 

  4. N. Garbin, L. Wang, J. H. Chandler, K. L. Obstein, N. Simaan, P. Valdastri, A disposable continuum endoscope using piston-driven parallel bellow actuator, in: Proceedings of International Symposium on Medical Robotics (ISMR), Atlanta, pp. 1–6 (2018)

  5. Camarillo, D.B., Carlson, C.R., Salisbury, J.K.: Configuration tracking for continuum manipulators with coupled tendon drive. IEEE Trans. Robot. 25(4), 798–808 (2009)

    Article  Google Scholar 

  6. Lotfavar, A., Hasanzadeh, S., Janabi-Sharifi, F.: Cooperative continuum robots: concept, modeling, and workspace analysis. IEEE Robot. Autom. Lett. 3(1), 426–433 (2018)

    Article  Google Scholar 

  7. Yip, M.C., Sganga, J.A., Camarillo, D.B.: Autonomous control of continuum robot manipulators for complex cardiac ablation tasks. J. Med. Robot. Res. 2(1), 1–13 (2017)

    Article  Google Scholar 

  8. M. N. Boushaki, C. Liu, P. Poignet, Task-space position control of concentric-tube robot with inaccurate kinematics using approximate Jacobian, in: Proceedings of IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, pp. 5877–5882 (2014)

  9. Braganza, D., Dawson, D.M., Walker, I.D., Nath, N.: Neural Network Grasping Controller for Continuum Robots, in: Proceedings of the 45th IEEE Conference on Decision and Control, pp. 6445–6449, San Diego (2006)

  10. Falkenhahn, V., Hildebrandt, A., Neumann, R., Sawodny, O.: Model-based feedforward position control of constant curvature continuum robots using feedback linearization, in: Proceedings of IEEE International Conference on Robotics and Automation (ICRA), pp. 762–767, Seattle (2015)

  11. Ivanescu, M., Stoian, V.: A variable structure controller for a tentacle manipulator, in: Proceedings of IEEE International Conference on Robotics and Automation, pp. 3155–3160, Nagoya (1995)

  12. Kapadia, A., Walker, I.D.: Task-space control of extensible continuum manipulators, in: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1087–1092, San Francisco (2011)

  13. Marchese, A.D., Komorowski, K., Onal, C.D., Rus, D.: Design and control of a soft and continuously deformable 2D robotic manipulation system, in: Proceedings of IEEE International Conference on Robotics and Automation (ICRA), pp. 2189–2196, Hong Kong (2014)

  14. Penning, R.S., Jung, J., Borgstadt, J.A., Ferrier, N.J., Zinn, M.R.: Towards closed loop control of a continuum robotic manipulator for medical applications, in: Proceedings of IEEE International Conference on Robotics and Automation, pp. 4822–4827, Shanghai (2011)

  15. Goldman, R.E., Bajo, A., Simaan, N.: Compliant motion control for multisegment continuum robots with actuation force sensing. IEEE Trans. Robot. 30(4), 890–902 (2014)

    Article  Google Scholar 

  16. Penning, R.S., Jung, J., Ferrier, N.J., Zinn, M.R.: An evaluation of closed-loop control options for continuum manipulators, in: Proceedings of IEEE International Conference on Robotics and Automation, pp. 5392–5397, Saint Paul (2012)

  17. Chikhaoui, M.T., Granna, J., Starke, J., Burgner-Kahrs, J.: Toward motion coordination control and design optimization for dual-arm concentric tube continuum robots. IEEE Robot. Autom. Lett. 3(3), 1793–1800 (2018)

    Article  Google Scholar 

  18. Gravagne, I.A., Rahn, C.D., Walker, I.D.: Large deflection dynamics and control for planar continuum robots. IEEE-ASME Trans. Mech. 8(2), 299–307 (2003)

    Article  Google Scholar 

  19. Zhang, Z., Dequidt, J., Back, J., Liu, H., Duriez, C.: Motion control of cable-driven continuum catheter robot through contacts. IEEE Robot. Autom. Lett. 4(2), 1852–1859 (2019)

    Article  Google Scholar 

  20. Thuruthel, T.G., Falotico, E., Renda, F., Laschi, C.: Model-based reinforcement learning for closed-loop dynamic control of soft robotic manipulators. IEEE Trans. Robot. 35(1), 124–134 (2019)

    Article  Google Scholar 

  21. Yip, M.C., Camarillo, D.B.: Model-less hybrid position/force control: a minimalist approach for continuum manipulators in unknown, constrained environments. IEEE Robot. Autom. Lett. 1(2), 844–851 (2016)

    Article  Google Scholar 

  22. Yip, M.C., Camarillo, D.B.: Model-less feedback control of continuum manipulators in constrained environments. IEEE Trans. Robot. 30(4), 880–889 (2014)

    Article  Google Scholar 

  23. You, X., Zhang, Y., Chen, X., Liu, X., Wang, Z., Jiang, H., Chen, X.: Model-free control for soft manipulators based on reinforcement learning, in: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2909–2915, Vancouver (2017)

  24. Li, M., Kang, R., Branson, D.T., Dai, J.S.: Model-Free Control for Continuum Robots Based on an Adaptive Kalman Filter. IEEE/ASME Trans. Mech. 23(1), 286–297

  25. Zhao, X.M., Jiang, M.M., Li, H.Y., Liu, H.: Adaptive fuzzy PID controller of a master-slave robotic catheter system in minimally invasive surgery. Appl. Mech. Mater. 419, 706–712 (2013)

    Article  Google Scholar 

  26. Zadeh, L.A.: Toward extended fuzzy logic—a first step. Fuzzy Sets Sys. 160(21), 3175–3181 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  27. Zadeh, L.A.: Is there a need for fuzzy logic? Inf. Sci. 178(13), 2751–2779 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  28. Qi, P., Liu, C., Ataka, A., Lam, H.K., Althoefer, K.: Kinematic control of continuum manipulators using a fuzzy-model-based approach. IEEE Trans. Ind. Electron. 63(8), 5022–5035 (2016)

    Article  Google Scholar 

  29. Du, Z., Wang, W., Yan, Z., Dong, W., Wang, W.: Variable admittance control based on fuzzy reinforcement learning for minimally invasive surgery manipulator. Sensors. 17(4), 1–15 (2017)

    Article  Google Scholar 

  30. Goharimanesh, M., Lashkaripour, A., Shariatnia, S., Akbari, A.: Diabetic control using genetic fuzzy-PI controller. Int. J. Fuzzy Sys. 16(2), 133–139 (2014)

    MathSciNet  Google Scholar 

  31. AL-Saedi, M.I., Wu, H., Handroos, H.: ANFIS and fuzzy tuning of PID controller for trajectory tracking of a flexible hydraulically driven parallel robot machine. J. Autom. Con. Eng. 1(3), 213–226 (2013)

    Google Scholar 

  32. Omisore, O.M., Han, S.P., Ren, L.X., Wang, G.S., Ou, F.L., Li, H., Wang, L.: Towards characterization and adaptive compensation of backlash in a novel robotic catheter system for cardiovascular interventions. IEEE Trans. Biomed. Circ. Sys. 12(4), 1–15 (2018)

    Article  Google Scholar 

  33. R.S. Sutton, A.G. Barto, Reinforcement Learning: An Introduction, Cambridge Univ Press, (1998)

  34. Sutton, R.S.: Generalization in reinforcement learning: Successful examples using sparse coarse coding, in: Proceedings of Advances in Neural Information Processing Systems, pp. 1038–1044 (1996)

    Google Scholar 

  35. Sutton, R.S., Barto, A.G., Williams, R.J.: Reinforcement learning is direct adaptive optimal control. IEEE Con. Sys. Mag. 12(2), 19–22 (1992)

    Article  Google Scholar 

  36. Goharimanesh, M., Akbari, A.A., Naghibi-Sistani, M.B.: Combining the principles of fuzzy logic and reinforcement learning for control of dynamic systems. J. app. comput. sci imech. 27(1), 1–14 (2015)

    Google Scholar 

  37. Akbari, A.A., Goharimanesh, M.: Yaw moment control using fuzzy reinforcemnt learning, in: Proceedings of Advanced Vehicle Control conference (AVEC14), pp. 1–6, Tokyo (2014)

  38. Tibebu, A.T., Yu, B., Kassahun, Y., Vander Poorten, E., Tran, P.T.: Towards autonomous robotic catheter navigation using reinforcement learning, in: Proceedings of the 4th Joint Workshop on New Technologies for Computer/Robot Assisted Surgery, pp. 163–166, Leuven (2014)

  39. Yu, L., Yu, X., Chen, X., Zhang, F.: Laparoscope arm automatic positioning for robot-assisted surgery based on reinforcement learning. Mech. Sci. 10(1), 119–131 (2019)

    Article  Google Scholar 

  40. Zhang, Q., Li, M., Qi, X., Hu, Y., Sun, Y., Yu, G.: 3D path planning for anterior spinal surgery based on CT images and reinforcement learning, in: Proceedings of IEEE International Conference on Cyborg and Bionic Systems (CBS), pp. 317–321, Shenzhen (2018)

  41. Baek, D., Hwang, M., Kim, H., Kwon, D.-S.: Path planning for automation of surgery robot based on probabilistic roadmap and reinforcement learning, in: Proceedings of 15th International Conference on Ubiquitous Robots (UR), pp. 342–347, Honolulu (2018)

  42. Chattopadhyay, S., Bhattacherjee, S., Bandyopadhyay, S., Sengupta, A., Bhaumik, S.: Control of single-segment continuum robots: reinforcement learning vs. neural network based PID, in: Proceedings of International Conference on Control, Power, Communication and Computing Technologies (ICCPCCT), pp. 222–226, Kannur (2018)

  43. You, H., Bae, E., Moon, Y., Kweon, J., Choi, J.: Automatic control of cardiac ablation catheter with deep reinforcement learning method. J. Mech. Sci. Tech. 33(11), 5415–5423 (2019)

    Article  Google Scholar 

  44. Satheeshbabu, S., Uppalapati, N.K., Chowdhary, G., Krishnan, G.: Open loop position control of soft continuum arm using deep reinforcement learning, in: Proceedings of International Conference on Robotics and Automation (ICRA), pp. 5133–5139, Montreal (2019)

  45. Hu, J., Wellman, M.P.: Nash Q-learning for general-sum stochastic games. J. Mach. Learn. Res. 4(Nov), 1039–1069 (2003)

    MathSciNet  MATH  Google Scholar 

  46. Akchurina, N.: Multiagent reinforcement learning: algorithm converging to Nash Equilibrium in general-sum discounted stochastic games, in: Proceedings of 8th International Conference on Autonomous Agents and Multiagent Systems-Volume 2, pp. 725–732. International Foundation for Autonomous Agents and Multiagent Systems, Budapest (2009)

    Google Scholar 

  47. Busoniu, L., Babuska, R., Schutter, B.D.: A comprehensive survey of multiagent reinforcement learning. IEEE Trans. Sys. Man Cyber., Part C (Applications and Reviews). 38(2), 156–172 (2008)

    Article  Google Scholar 

  48. A. K. Sadhu, A. Konar, An efficient computing of correlated equilibrium for cooperative Q-learning-based multi-robot planning, IEEE trans. Syst. Man Cybern.: Syst., (2018, Sep.) 1–16

  49. Goharimanesh, M., Abbasi Jannatabadi, E., Moeinkhah, H., Naghibi-Sistani, M.B., Akbari, A.A.: An intelligent controller for ionic polymer metal composites using optimized fuzzy reinforcement learning. J. Intell. Fuzzy Sys. 33(1), 125–136 (2017)

    Article  Google Scholar 

  50. Soltani, M.K., Khanmohammadi, S., Ghalichi, F., Janabi-Sharifi, F.: A soft robotics nonlinear hybrid position/force control for tendon driven catheters. Int. J Control Autom. 15(1), 54–63 (2017)

    Article  Google Scholar 

  51. Watkins, C.J., Dayan, P.: Q-learning. Mach. Learn. 8(3–4), 279–292 (1992)

    MATH  Google Scholar 

  52. C.J.C.H. Watkins, Learning from Delayed Rewards (PhD Dissertation), King’s college, Cambridge, (1989)

  53. Li, H., Liu, D., Wang, D.: Integral reinforcement learning for linear continuous-time zero-sum games with completely unknown dynamics. IEEE Trans. Autom. Sci. Eng. 11(3), 706–714 (2014)

    Article  Google Scholar 

  54. Glorennec, P.Y., Jouffe, L.: Fuzzy Q-learning, in: Proceedings of the 6th IEEE International Conference on Fuzzy Systems, pp. 659–662, Barcelona (1997)

  55. Goharimanesh, M., Akbari, A.: Optimum parameters of nonlinear integrator using design of experiments based on Taguchi method. J. App. Mech. 46(2), 233–241 (2015)

    Google Scholar 

  56. Torczon, V.: On the convergence of pattern search algorithms. SIAM J. on Opt. 7(1), 1–25 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  57. Yuan, H., Zhou, L., Xu, W.: A comprehensive static model of cable-driven multi-section continuum robots considering friction effect. Mech. Theory. 135, 130–149 (2019)

    Article  Google Scholar 

Download references

Acknowledgments

This work was sponsored by the National Sciences and Engineering Research Council of Canada (NSERC) through Discovery Grant # 2017-06930. The authors would also like to acknowledge the funding from the Dean of the Faculty (of Engineering and Architectural Science) at Ryerson University for the financial support of the project.

Author information

Authors and Affiliations

  1. Department of Mechanical Engineering, University of Torbat Heydarieh, Torbat Heydarieh, Iran

    M. Goharimanesh

  2. Department of Mechanical and Industrial Engineering, Ryerson University, 350 Victoria Street, Toronto, M5B 2K3, Canada

    A. Mehrkish & F. Janabi-Sharifi

Authors
  1. M. Goharimanesh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. Mehrkish
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. F. Janabi-Sharifi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to F. Janabi-Sharifi.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Goharimanesh, M., Mehrkish, A. & Janabi-Sharifi, F. A Fuzzy Reinforcement Learning Approach for Continuum Robot Control. J Intell Robot Syst 100, 809–826 (2020). https://doi.org/10.1007/s10846-020-01237-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-020-01237-6

Keywords

Search

Navigation