Hostname: page-component-8448b6f56d-cfpbc Total loading time: 0 Render date: 2024-04-23T13:00:19.167Z Has data issue: false hasContentIssue false
  • English
  • Français

Discrete soft actor-critic with auto-encoder on vascular robotic system

Published online by Cambridge University Press:  17 November 2022

Hao Li Open the ORCID record for Hao Li [Opens in a new window] ,
Xiao-Hu Zhou Open the ORCID record for Xiao-Hu Zhou [Opens in a new window] ,
Xiao-Liang Xie ,
Shi-Qi Liu ,
Mei-Jiang Gui ,
Tian-Yu Xiang ,
Jin-Li Wang Open the ORCID record for Jin-Li Wang [Opens in a new window]  and
Zeng-Guang Hou
Hao Li
Affiliation:
The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China The School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Xiao-Hu Zhou*
Affiliation:
The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China The School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Xiao-Liang Xie*
Affiliation:
The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China The School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Shi-Qi Liu
Affiliation:
The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China The School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Mei-Jiang Gui
Affiliation:
The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China The School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Tian-Yu Xiang
Affiliation:
The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China The School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
Jin-Li Wang
Affiliation:
The School of Mechanical Electronic and Information Engineering, China University of Mining and Technology, Beijing, China
Zeng-Guang Hou
Affiliation:
The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China The School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China The CAS Center for Excellence in Brain Science and Intelligence Technology, Beijing, China
*
*Corresponding authors. E-mail: xiaohu.zhou@ia.ac.cn; xiaoliang.xie@ia.ac.cn
*Corresponding authors. E-mail: xiaohu.zhou@ia.ac.cn; xiaoliang.xie@ia.ac.cn
Get access Rights & Permissions [Opens in a new window]

Abstract

Instrument delivery is critical part in vascular intervention surgery. Due to the soft-body structure of instruments, the relationship between manipulation commands and instrument motion is non-linear, making instrument delivery challenging and time-consuming. Reinforcement learning has the potential to learn manipulation skills and automate instrument delivery with enhanced success rates and reduced workload of physicians. However, due to the sample inefficiency when using high-dimensional images, existing reinforcement learning algorithms are limited on realistic vascular robotic systems. To alleviate this problem, this paper proposes discrete soft actor-critic with auto-encoder (DSAC-AE) that augments SAC-discrete with an auxiliary reconstruction task. The algorithm is applied with distributed sample collection and parameter update in a robot-assisted preclinical environment. Experimental results indicate that guidewire delivery can be automatically implemented after 50k sampling steps in less than 15 h, demonstrating the proposed algorithm has the great potential to learn manipulation skill for vascular robotic systems.

Keywords

surgical robotsvascular robotic systemautomationreinforcement learningdeep neural network
Type
Research Article
Information
Robotica , Volume 41 , Issue 4 , April 2023 , pp. 1115 - 1126
Copyright
© The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Mensah, G. A., Roth, G. A. and Fuster, V., “The global burden of cardiovascular diseases and risk factors: 2020 and beyond,” J. Am. Coll. Cardiol. 74(20), 2529–2532 (2019).CrossRefGoogle ScholarPubMed
Zhou, X.-H., Xie, X.-L., Liu, S.-Q., Ni, Z.-L., Zhou, Y.-J., Li, R.-Q., Gui, M.-J., Fan, C.-C., Feng, Z.-Q., Bian, G.-B. and Hou, Z.-G., “Learning skill characteristics from manipulations,” IEEE Trans. Neural Netw. Learn. Syst., 1–15 (2022). doi: 10.1109/TNNLS.2022.3160159.CrossRefGoogle Scholar
Rafii-Tari, H., Payne, C. J. and Yang, G.-Z., “Current and emerging robot-assisted endovascular catheterization technologies: A review,” Ann. Biomed. Eng. 42(4), 697–715 (2014).CrossRefGoogle ScholarPubMed
Zhou, X.-H., Xie, X.-L., Liu, S.-Q., Feng, Z.-Q., Gui, M.-J., Wang, J.-L., Li, H., Xiang, T.-Y., Bian, G.-B. and Hou, Z.-G., “Surgical skill assessment based on dynamic warping manipulations,” IEEE Trans. Med. Robot. Bionics. 4(1), 50–61 (2022).CrossRefGoogle Scholar
Roguin, A., Goldstein, J. and Bar, O., “Brain tumours among interventional cardiologists: A cause for alarm? Report of four new cases from two cities and a review of the literature,” EuroIntervention 7(9), 1081–1086 (2012).CrossRefGoogle Scholar
Karatasakis, A., Brilakis, H. S., Danek, B. A., Karacsonyi, J., Martinez-Parachini, J. R., Nguyen-Trong, P. J., Alame, A. J., Roesle, M. K., Rangan, B. V., Rosenfield, K., Mehran, R., Mahmud, E., Chambers, C. E., Banerjee, S. and Brilakis, E. S., “Radiation-associated lens changes in the cardiac catheterization laboratory: Results from the IC-CATARACT(CATaracts Attributed to RAdiation in the CaTh lab) study,” Catheter. Cardiovasc. Interv. 91(4), 647–654 (2018).CrossRefGoogle ScholarPubMed
Elmaraezy, A., Morra, M. E., Mohammed, A. T., AlHabaa, A., Elgebaly, A. S., Ghazy, A. A., Khalil, A., Huy, N. T. and Hirayama, K., “Risk of cataract among interventional cardiologists and catheterization lab staff: A systematic review and meta-analysis,” Catheter. Cardiovasc. Interv. 90(1), 1–9 (2017).CrossRefGoogle Scholar
Klein, L. W., Tra, Y., Garratt, K. N., Powell, W. A., Lopez-Cruz, G., Chambers, C. E. and Goldstein, J. A., “Occupational health hazards of interventional cardiologists in the current decade: Results of the 2014 SCAI membership survey,” Catheter. Cardiovasc. Interv. 86(5), 913–924 (2015).CrossRefGoogle ScholarPubMed
Granada, J. F., Delgado, J. A., Uribe, M. P., Fernández, A., Blanco, G., Leon, M. B. and Weisz, G., “First-in-human evaluation of a novel robotic-assisted coronary angioplasty system,” J. Am. Coll. Cardiol. Cardiovas. Interv. 4(4), 460–465 (2011).CrossRefGoogle ScholarPubMed
Woo, J.-H., Song, H.-S., Cha, H.-J. and Yi, B.-J., “Advantage of steerable catheter and haptic feedback for a 5-DOF vascular intervention robot system,” Appl. Sci. 9(20), 4305 (2019).CrossRefGoogle Scholar
Guo, S., Song, Y., Yin, X., Zhang, L., Tamiya, T., Hirata, H. and Ishihara, H., “A novel robot-assisted endovascular catheterization system with haptic force feedback,” IEEE Trans. Robot. 35, 685–696 (2019).CrossRefGoogle Scholar
Gui, M.-J., Zhou, X.-H., Xie, X.-L., Liu, S.-Q., H., L., Xia, T.-Y., Wang, J.-L. and Hou, Z.-G., “Design and experiments of a novel Halbach-cylinder-based magnetic skin: A preliminary study,” IEEE Trans. Instrum. Meas. 71, 1–11 (2022). doi: 10.1109/TIM.2022.3147904.CrossRefGoogle Scholar
Patel, T. M., Shah, S. and Pancholy, S. B., “Long distance tele-robotic-assisted percutaneous coronary intervention: A report of first-in-human experience,” EClinicalMedicine 14, 53–58 (2019).CrossRefGoogle ScholarPubMed
Yang, G.-Z., Cambias, J., Cleary, K., Daimler, E., Drake, J., Dupont, P. E., Hata, N., Kazanzides, P., Martel, S., Patel, R. V., Santos, V. J. and Taylor, R. H., “Medical robotics-regulatory, ethical, and legal considerations for increasing levels of autonomy,” Sci. Robot. 2(4), eaam8638 (2017).CrossRefGoogle ScholarPubMed
Nooryani, A. A. and Aboushokka, W., “Rotate-on-retract procedural automation for robotic-assisted percutaneous coronary intervention: First clinical experience,” Case Rep. Cardiol. 2018, 1–3 (2018).Google ScholarPubMed
Rafii-Tari, H., Liu, J., Lee, S.-L., Bicknell, C. D., and Yang, G.-Z., “Learning-Based Modeling of Endovascular Navigation for Collaborative Robotic Catheterization,” In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (Springer, Berlin, 2013) pp. 369–377.CrossRefGoogle Scholar
Rafii-Tari, H., Liu, J., Payne, C. J., Bicknell, C. D. and Yang, G.-Z., “Hierarchical HMM Based Learning of Navigation Primitives for Cooperative Robotic Endovascular Catheterization,” In: Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention (Springer, Berlin, 2014) pp. 496–503.CrossRefGoogle Scholar
Chi, W., Liu, J., Rafii-Tari, H., Riga, C. V., Bicknell, C. D. and Yang, G.- Z., “Learning-based endovascular navigation through the use of non-rigid registration for collaborative robotic catheterization,” Int. J. Comput. Assist. Radiol. Surg. 13(6), 855–864 (2018).CrossRefGoogle ScholarPubMed
Zhao, Y., Guo, S., Wang, Y., Cui, J., Ma, Y., Zeng, Y., Liu, X., Jiang, Y., Li, Y., Shi, L. and Xiao, N., “A CNN-based prototype method of unstructured surgical state perception and navigation for an endovascular surgery robot,” Med. Biol. Eng. Comput. 57(9), 1875–1887 (2019).CrossRefGoogle ScholarPubMed
Guo, J., Feng, S. and Guo, S., “Study on the Automatic Surgical Method of the Vascular Interventional Surgical Robot Based on Deep Learning,” In: Proceedings of 2021 IEEE International Conference on Mechatronics and Automation (Institute of Electrical and Electronics Engineers Inc., Piscataway, 2021) pp. 1076–1081.CrossRefGoogle Scholar
Chi, W., Dagnino, G., Kwok, T. M. Y., Nguyen, A., Kundrat, D., Abdelaziz, M. E. M. K., Riga, C., Bicknell, C. and Yang, G.-Z., “Collaborative Robot-Assisted Endovascular Catheterization with Generative Adversarial Imitation Learning,” In: Proceedings of 2020 IEEE International Conference on Robotics and Automation (Institute of Electrical and Electronics Engineers Inc., Piscataway, 2020) pp. 2414–2420.CrossRefGoogle Scholar
Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., Lillicrap, T. and Silver, D., “Mastering Atari, Go, chess and shogi by planning with a learned model,” Nature 588(7839), 604–609 (2020).CrossRefGoogle ScholarPubMed
Azimirad, V. and Sani, M. F., “Experimental study of reinforcement learning in mobile robots through spiking architecture of Thalamo-Cortico-Thalamic circuitry of mammalian brain,” Robotica 38(9), 1558–1575 (2020).CrossRefGoogle Scholar
Gómez, M., González, R. V., Martínez-Marín, T., Meziat, D. and Sánchez, S., “Optimal motion planning by reinforcement learning in autonomous mobile vehicles,” Robotica 30(2), 159–170 (2012).CrossRefGoogle Scholar
Karstensen, L., Behr, T., Pusch, T. P., Mathis-Ullrich, F. and Stallkamp, J., “Autonomous guidewire navigation in a two dimensional vascular phantom,” Curr. Dir. Biomed. Eng. 6(1), 20200007 (2020).CrossRefGoogle Scholar
Behr, T., Pusch, T. P., Siegfarth, M., Hüsener, D., Mörschel, T. and Karstensen, L., “Deep reinforcement learning for the navigation of neurovascular catheters,” Curr. Dir. Biomed. Eng. 5(1), 5–8 (2019).CrossRefGoogle Scholar
Chi, W., Liu, J., Abdelaziz, M. E. M. K., Dagnino, G., Riga, C. V., Bicknell, C. D. and Yang, G.-Z., “Trajectory Optimization of Robot-Assisted Endovascular Catheterization with Reinforcement Learning,” In: Proceedings of 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (Institute of Electrical and Electronics Engineers Inc., Piscataway, 2018) pp. 3875–3881.CrossRefGoogle Scholar
Shi, C., Luo, X., Guo, J., Najdovski, Z., Fukuda, T. and Ren, H., “Three-dimensional intravascular reconstruction techniques based on intravascular ultrasound: A technical review,” IEEE J. Biomed. Health Inform. 22(3), 806–817 (2018).CrossRefGoogle ScholarPubMed
Wang, Z., de Freitas, N. and Lanctot, M., “Dueling network architectures for deep reinforcement learning,” CoRR, abs/1511.06581 (2015).Google Scholar
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D. and Kavukcuoglu, K., “Asynchronous Methods for Deep Reinforcement Learning,” In: Proceedings of the 33rd International Conference on Machine Learning (Proceedings of Machine Learning Research, New York, 2016) pp. 1928–1937.Google Scholar
You, H., Bae, E., Moon, Y., Kweon, J. and Choi, J., “Automatic control of cardiac ablation catheter with deep reinforcement learning method,” J. Mech. Sci. Technol. 33(11), 5415–5423 (2019).CrossRefGoogle Scholar
M, F. eng, , Guo, S., Zhou, W. and Chen, Z., “Evaluation of a Reinforcement Learning Algorithm for Vascular Intervention Surgery,” In: Proceedings of 2021 IEEE International Conference on Mechatronics and Automation (Institute of Electrical and Electronics Engineers Inc., Piscataway, 2021) pp. 1033–1037.CrossRefGoogle Scholar
Yarats, D., Zhang, A., Kostrikov, I., Amos, B., Pineau, J. and Fergus, R., “Improving Sample Efficiency in Model-Free Reinforcement Learning from Images,” In: Proceedings of the 35th AAAI Conference on Artificial Intelligence (Association for the Advancement of Artificial Intelligence, Menlo Park, 2021) pp. 10674–10681.CrossRefGoogle Scholar
Sr, A. inivas, , Laskin, M. and Abbeel, P., “CURL: Contrastive Unsupervised Representations for Reinforcement Learning,” In: Proceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research, NewYork, 2020) pp. 5639–5650.Google Scholar
, H.- Zhao, L., Liu, S.-Q., Zhou, X.-H., Xie, X.-L., Hou, Z.-G., Zhou, Y.-J., Zhang, L.-S., Gui, M.-J. and Wang, J.-L., “Design and Performance Evaluation of a Novel Vascular Robotic System for Complex Percutaneous Coronary Interventions,” In: Proceedings of 2021 43rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (Institute of Electrical and Electronics Engineers Inc., Piscataway, 2021) pp. 4679–4682.CrossRefGoogle Scholar
Zie, B. D. bart, , Maas, A. L., Bagnell, J. A. and Dey, A. K., “Maximum Entropy Inverse Reinforcement Learning,” In: Proceedings of the 23rd AAAI Conference on Artificial Intelligence (Association for the Advancement of Artificial Intelligence, Menlo Park, 2008) pp. 1433–1438.Google Scholar
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P. and Levine, S., “Soft actor-critic algorithms and applications,” CoRR, abs/1812.05905 (2018).Google Scholar
Christodoulou, P., “Soft actor-critic for discrete action settings,” CoRR, abs/1910.07207 (2019).Google Scholar
Lin, T.-Y., Goyal, P., Girshick, R. B., He, K. and Dollár, P., “Focal loss for dense object detection,” IEEE Trans. Pattern Anal. Mach. Intell. 42, 318–327 (2020).CrossRefGoogle ScholarPubMed
Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Paul, W., Jordan, M. I. and Stoica, I., “Ray: A distributed framework for emerging ai applications,” CoRR, abs/1712.05889 (2017).Google Scholar
Lew, E., Ricardo Chavarriaga, S. S. and Millán, J., “Detection of self-paced reaching movement intention from EEG signals,” Front. Neuroeng. 5, 13 (2012).CrossRefGoogle ScholarPubMed
Heidbuchel, H., Wittkampf, F. H. M., Vañó, E., Ernst, S., Schilling, R. J., Picano, E., Mont, L., Jais, P., de Bono, J., Piorkowski, C., Saad, E. B. and Femenía, F. J., “Practical ways to reduce radiation dose for patients and staff during device implantations and electrophysiological procedures,” Europace 16(7), 946–964 (2014).CrossRefGoogle ScholarPubMed