Abstract
Dense depth prediction for 3-D reconstruction of monocular endoscopic images is an essential way to expand the surgical field and augment the perception of surgeons in robotic endoscopic surgery. However, it is generally challenging to precisely estimate the monocular dense depth and reconstruct such a field due to complex surgical fields with a limited field of view, illumination variations, and weak texture information. This work proposes a new framework of self-supervised learning with a two-stage cascade training strategy for dense depth recovery of monocular endoscopic images. While the first stage is to train an initial deep-learning model through sparse depth consistency supervision, the second stage introduces photometric consistency supervision to further train and refine the initial model for improving its capability. Our framework was evaluated on patient data of monocular endoscopic images acquired from colonoscopic procedures, with the experimental results demonstrating that our self-supervised learning model with cascade training provides a promising strategy outperforming other models. On the one hand, both visual quality and quantitative assessment of our method are better than current monocular dense depth estimation approaches. On the other hand, our method relies less on sparse depth data for supervision than other self-supervised methods.
W. Jiang and W. Fan—Equally contributed.
X. Luo and H. Shi—Corresponding authors.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bae, G., Budvytis, I., Yeung, C.-K., Cipolla, R.: Deep multi-view stereo for dense 3D reconstruction from monocular endoscopic video. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12263, pp. 774–783. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59716-0_74
Crismale, J.F., Mahmoud, D., Moon, J., Fiel, M.I., Iyer, K., Schiano, T.D.: The role of endoscopy in the small intestinal transplant recipient: a review. Am. J. Transplant. 21(5), 1705–1712 (2021)
Engel, J., Koltun, V., Cremers, D.: Direct sparse odometry. IEEE Transactions on Pattern Analysis & Machine Intelligence, p. 1 (2016)
Farhan, E., Meir, E., Hagege, R.: Local region expansion: a method for analyzing and refining image matches. SIAM J. Imag. Sci. 8(4), 2771–2813 (2017)
Gottlieb, K., et al.: Endoscopy and central reading in inflammatory bowel disease clinical trials: achievements, challenges and future developments. Gut 70(2), 418–426 (2021)
Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: 2010 20th International Conference on Pattern Recognition, pp. 2366–2369. IEEE (2010)
Jégou, S., Drozdzal, M., Vazquez, D., Romero, A., Bengio, Y.: The one hundred layers tiramisu: Fully convolutional DenseNets for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 11–19 (2017)
Klingner, M., Termöhlen, J.A., Mikolajczyk, J., Fingscheidt, T.: Self-supervised monocular depth estimation: solving the dynamic object problem by semantic guidance. ArXiv abs/2007.06936 (2020)
Leonard, S., et al.: Evaluation and stability analysis of video-based navigation system for functional endoscopic sinus surgery on in vivo clinical data. IEEE Trans. Med. Imaging 37(10), 2185–2195 (2018)
Liu, X., et al.: Dense depth estimation in monocular endoscopy with self-supervised learning methods. IEEE Trans. Med. Imag. PP(99), 1 (2019)
Liu, X., et al.: Reconstructing sinus anatomy from endoscopic video – towards a radiation-free approach for quantitative longitudinal assessment. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12263, pp. 3–13. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59716-0_1
Ma, R., et al.: RNNSLAM: reconstructing the 3d colon to visualize missing regions during a colonoscopy. Med. Image Anal. 72, 102100 (2021)
Mur-Artal, R., Tardós, J.D.: ORB-SLAM2: an open-source slam system for monocular, stereo, and RGB-D cameras. IEEE Trans. Rob. 33, 1255–1262 (2017)
Ozyoruk, K.B., et al.: EndoSLAM dataset and an unsupervised monocular visual odometry and depth estimation approach for endoscopic videos. Med. Image Anal. 71, 102058 (2021)
Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4104–4113 (2016)
Smith, L.N.: Cyclical learning rates for training neural networks. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464–472. IEEE (2017)
Sung, H., et al.: Global cancer statistics 2020: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: Cancer J. Clin. 71(3), 209–249 (2021)
Wang, R., Pizer, S.M., Frahm, J.M.: Recurrent neural network for (un-)supervised learning of monocular videovisual odometry and depth (2019)
Yin, Z., Shi, J.: Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Zhou, T., Brown, M., Snavely, N., Lowe, D.G.: Unsupervised learning of depth and ego-motion from video. Arxiv (2017)
Zhu, S., et al.: Very large-scale global SFM by distributed motion averaging. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4568–4577 (2018)
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grants 61971367, 82272133, and 62001403, in part by the Natural Science Foundation of Fujian Province of China under Grants 2020J01004 and 2020J05003, and in part by the Fujian Provincial Technology Innovation Joint Funds under Grant 2019Y9091.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Jiang, W., Fan, W., Chen, J., Shi, H., Luo, X. (2024). Self-supervised Cascade Training for Monocular Endoscopic Dense Depth Recovery. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14429. Springer, Singapore. https://doi.org/10.1007/978-981-99-8469-5_38
Download citation
DOI: https://doi.org/10.1007/978-981-99-8469-5_38
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8468-8
Online ISBN: 978-981-99-8469-5
eBook Packages: Computer ScienceComputer Science (R0)