Abstract
One of the promising fields for active-stereo sensors is medical applications such as 3D endoscope systems. For such systems, robust correspondence estimation between the detected patterns and the projected pattern is the most crucial. In this paper, we propose an auto-calibrating 3D endoscopic system using a 2D grid-graph pattern, where codes are embedded into each grid point. Since the pattern is a grid graph, we use a graph convolutional network (GCN) to calculate node-wise embedding accumulating code information of nearby grid points in the graph. The correspondence estimation using the GCN-calculated feature embedding is shown to be stable, even without using epipolar constraints. Using the correspondence estimation, we show that the auto-calibrating 3D measurement system can be realized. In the experiment, we confirmed that the proposed system achieved high accuracy and robust estimation comparing to the previous methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Defferrard, M., Bresson, X., Vandergheynst, P.: Convolutional neural networks on graphs with fast localized spectral filtering. In: Advances in Neural Information Processing Systems, pp. 3844–3852 (2016)
Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient belief propagation for early vision. Int. J. Comput. Vision 70(1), 41–54 (2006)
Furukawa, R., et al.: Simultaneous shape and camera-projector parameter estimation for 3D endoscopic system using CNN-based grid-oneshot scan. Healthcare Technol. Lett. 6(6), 249–254 (2019)
Furukawa, R., et al.: 2-DOF auto-calibration for a 3d endoscope system based on active stereo. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 7937–7941. IEEE (2015)
Furukawa, R., Mizomori, M., Hiura, S., Oka, S., Tanaka, S., Kawasaki, H.: Wide-area shape reconstruction by 3D endoscopic system based on CNN decoding, shape registration and fusion. In: Stoyanov, D., et al. (eds.) CARE/CLIP/OR 2.0/ISIC -2018. LNCS, vol. 11041, pp. 139–150. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01201-4_16
Furukawa, R., Morinaga, H., Sanomura, Y., Tanaka, S., Yoshida, S., Kawasaki, H.: Shape acquisition and registration for 3d endoscope based on grid pattern projection. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 399–415. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_24
Furukawa, R., et al.: Fully auto-calibrated active-stereo-based 3d endoscopic system using correspondence estimation with graph convolutional network, vol. 2020, pp. 4357–4360 (2020). https://doi.org/10.1109/EMBC44109.2020.9176417
Geurten, J., Xia, W., Jayarathne, U., Peters, T.M., Chen, E.C.S.: Endoscopic laser surface scanner for minimally invasive abdominal surgeries. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 143–150. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_17
Kawasaki, H., Furukawa, R., Sagawa, R., Yagi, Y.: Dynamic scene shape reconstruction using a single structured light pattern. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2008)
Liu, X., et al.: Self-supervised learning for dense depth estimation in monocular endoscopy. In: Stoyanov, D., et al. (eds.) CARE/CLIP/OR 2.0/ISIC -2018. LNCS, vol. 11041, pp. 128–138. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01201-4_15
Ma, R., Wang, R., Pizer, S., Rosenman, J., McGill, S.K., Frahm, J.-M.: Real-time 3D reconstruction of Colonoscopic surfaces for determining missing regions. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11768, pp. 573–582. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32254-0_64
Mahmood, F., Chen, R., Durr, N.J.: Unsupervised reverse domain adaptation for synthetic medical images via adversarial training. IEEE Trans. Med. Imaging 37(12), 2572–2581 (2018)
Mahmood, F., Durr, N.J.: Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy. Med. Image Anal. 48, 230–243 (2018)
Maurice, X., Albitar, C., Doignon, C., de Mathelin, M.: A structured light-based laparoscope with real-time organs’ surface reconstruction for minimally invasive surgery. In: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 5769–5772. IEEE (2012)
Nagakura, T., Michida, T., Hirao, M., Kawahara, K., Yamada, K.: The study of three-dimensional measurement from an endoscopic images with stereo matching method. In: 2006 World Automation Congress, pp. 1–4. IEEE (2006)
Rau, A., et al.: Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy. Int. J. Comput. Assist. Radiol. Surg. 14(7), 1167–1176 (2019). https://doi.org/10.1007/s11548-019-01962-w
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Salvi, J., Pages, J., Batlle, J.: Pattern codification strategies in structured light systems. Pattern Recogn. 37(4), 827–849 (2004)
Schmalz, C., Forster, F., Schick, A., Angelopoulou, E.: An endoscopic 3D scanner based on structured light. Med. Image Anal. 16(5), 1063–1072 (2012)
Song, L., Tang, S., Song, Z.: A robust structured light pattern decoding method for single-shot 3D reconstruction. In: 2017 IEEE International Conference on Real-time Computing and Robotics (RCAR), pp. 668–672. IEEE (2017)
Stoyanov, D., Scarzanella, M.V., Pratt, P., Yang, G.-Z.: Real-time stereo reconstruction in robotically assisted minimally invasive surgery. In: Jiang, T., Navab, N., Pluim, J.P.W., Viergever, M.A. (eds.) MICCAI 2010. LNCS, vol. 6361, pp. 275–282. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15705-9_34
Visentini-Scarzanella, M., Sugiura, T., Kaneko, T., Koto, S.: Deep monocular 3D reconstruction for assisted navigation in bronchoscopy. Int. J. Comput. Assist. Radiol. Surg. 12(7), 1089–1099 (2017)
Zagoruyko, S., Komodakis, N.: Learning to compare image patches via convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4353–4361 (2015)
Žbontar, J., LeCun, Y.: Stereo matching by training a convolutional neural network to compare image patches. J. Mach. Learn. Res. 17(1), 2287–2318 (2016)
Zhang, L., Curless, B., Seitz, S.M.: Rapid shape acquisition using color structured light and multi-pass dynamic programming. In: Proceedings. First International Symposium on 3D Data Processing Visualization and Transmission, pp. 24–36. IEEE (2002)
Zhang, Z.: Microsoft Kinect sensor and its effect. IEEE MultiMedia 19, 4–12 (2012). https://www.microsoft.com/en-us/research/publication/microsoft-kinect-sensor-and-its-effect/
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Mikamo, M., Kawasaki, H., Sagawa, R., Furukawa, R. (2021). GCN-Calculated Graph-Feature Embedding for 3D Endoscopic System Based on Active Stereo. In: Jeong, H., Sumi, K. (eds) Frontiers of Computer Vision. IW-FCV 2021. Communications in Computer and Information Science, vol 1405. Springer, Cham. https://doi.org/10.1007/978-3-030-81638-4_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-81638-4_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-81637-7
Online ISBN: 978-3-030-81638-4
eBook Packages: Computer ScienceComputer Science (R0)