Abstract
In endoscopic surgery, it is important to search for organs, tumors and blood vessels, recognize their locations, and perform the surgery safely and quickly. Robot-assisted surgery has made it possible to perform precise operations. However, it is still difficult to perform surgery in a narrow surgical field with a narrow view. Several surgical support systems using augmented reality (AR) have been studied, in which three-dimensional computer graphics (3DCG) models of organs, tumors and blood vessels are superimposed on the endoscope camera image. We have developed an AR surgical support system based on SLAM (Simultaneous Localization and Mapping) technology. The SLAM can estimate the camera position and orientation in global coordinates using only the camera image. However, it is difficult to achieve robust AR because of the errors in camera position and orientation estimation and the failure of SLAM due to camera shake, heavy movement and the reflection of surgical instruments. To solve this problem, we have attempted to perform local position and orientation estimation using ICP on the 3DCG model of the organ and the intra-abdominal point cloud.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Li-Ming, S., Vagvolgyi, B.P., Agarwal, R., Reiley, C.E., Taylor, R.H., Hager, G.D.: Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. Urology 73(4), 896–900 (2009)
Edgcumbe, P., Singla, R., Pratt, P., Schneider, C., Nguan, C., Rohling, R.: Augmented reality imaging for robot-assisted partial nephrectomy surgery. In: Zheng, G., Liao, H., Jannin, P., Cattin, P., Lee, S.-L. (eds.) MIAR 2016. LNCS, vol. 9805, pp. 139–150. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-43775-0_13
Wang, J., Shen, Yu., Yang, S.: A practical marker-less image registration method for augmented reality oral and maxillofacial surgery. Int. J. Comput. Assist. Radiol. Surg. 14(5), 763–773 (2019). https://doi.org/10.1007/s11548-019-01921-5
Bertrand, L.R., et al.: A case series study of augmented reality in laparoscopic liver resection with a deformable preoperative model. Surg. Endosc. 34(12), 5642–5648 (2020). https://doi.org/10.1007/s00464-020-07815-x
Koeda, M., et al.: Image overlay support with 3DCG organ model for robot-assisted laparoscopic partial nephrectomy. In: Stephanidis, C. (ed.) HCI 2016. CCIS, vol. 617, pp. 508–513. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40548-3_84
Sawada, A., Hamada, A., Sengiku, A., Koeda, M., Onishi, K., Ogawa., O.: The development of a 3D navigation system for robot-assisted partial nephrectomy using augmented reality technology. In: Proceedings of The 2019 Annual EAU Congress, vol. 18, no. 1 (2019)
Sengiku, A., et al.: Augmented reality navigation system for robot-assisted laparoscopic partial nephrectomy. In: Marcus, A., Wang, W. (eds.) DUXU 2017. LNCS, vol. 10289, pp. 575–584. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-58637-3_45
Hamada, A., et al.: The current status and challenges in augmented-reality navigation system for robot-assisted laparoscopic partial nephrectomy. In: Kurosu, M. (ed.) HCII 2020. LNCS, vol. 12182, pp. 620–629. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49062-1_42
Koeda, M., Hamada, A., Sawada, A., Onishi, K., Noborio, H., Ogawa, O.: VR-based surgery navigation system with 3D user interface for robot-assisted laparoscopic partial nephrectomy. In: Kurosu, M. (ed.) HCII 2021. LNCS, vol. 12763, pp. 538–550. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-78465-2_39
Open3D. http://www.open3d.org/
OpenCV. https://opencv.org/
ICP registration - Open3D 0.11.0 documentation. http://www.open3d.org/docs/0.11.0/tutorial/pipelines/icp_registration.html
3D Slicer. https://www.slicer.org/
Voxel downsampling - Open3D 0.11.0 documentation. http://www.open3d.org/docs/0.11.0/tutorial/geometry/pointcloud.html#Voxel-downsampling
Acknowledgement
This study was supported by the 2021 Grants-in-Aid for Scientific Research (No.21K03967) from the Ministry of Education, Culture, Sports, Science and Technology, Japan.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Koeda, M. et al. (2022). Position and Orientation Registration of Intra-abdominal Point Cloud Generated from Stereo Endoscopic Images and Organ 3D Model Using Open3D. In: Kurosu, M. (eds) Human-Computer Interaction. Technological Innovation. HCII 2022. Lecture Notes in Computer Science, vol 13303. Springer, Cham. https://doi.org/10.1007/978-3-031-05409-9_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-05409-9_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-05408-2
Online ISBN: 978-3-031-05409-9
eBook Packages: Computer ScienceComputer Science (R0)