Abstract
Purpose
A three-dimensional (3D) structure extraction technique viewed from a two-dimensional image is essential for the development of a computer-aided diagnosis (CAD) system for colonoscopy. However, a straightforward application of existing depth-estimation methods to colonoscopic images is impossible or inappropriate due to several limitations of colonoscopes. In particular, the absence of ground-truth depth for colonoscopic images hinders the application of supervised machine learning methods. To circumvent these difficulties, we developed an unsupervised and accurate depth-estimation method.
Method
We propose a novel unsupervised depth-estimation method by introducing a Lambertian-reflection model as an auxiliary task to domain translation between real and virtual colonoscopic images. This auxiliary task contributes to accurate depth estimation by maintaining the Lambertian-reflection assumption. In our experiments, we qualitatively evaluate the proposed method by comparing it with state-of-the-art unsupervised methods. Furthermore, we present two quantitative evaluations of the proposed method using a measuring device, as well as a new 3D reconstruction technique and measured polyp sizes.
Results
Our proposed method achieved accurate depth estimation with an average estimation error of less than 1 mm for regions close to the colonoscope in both of two types of quantitative evaluations. Qualitative evaluation showed that the introduced auxiliary task reduces the effects of specular reflections and colon wall textures on depth estimation and our proposed method achieved smooth depth estimation without noise, thus validating the proposed method.
Conclusions
We developed an accurate depth-estimation method with a new type of unsupervised domain translation with the auxiliary task. This method is useful for analysis of colonoscopic images and for the development of a CAD system since it can extract accurate 3D information.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Nadeem S, Kaufman A (2016) Computer-aided detection of polyps in optical colonoscopy images. In: Proceedings of SPIE 9785, medical imaging 2016: computer-aided diagnosis, pp 549–560
Itoh H, Roth HR, Lu L, Oda M, Misawa M, Mori Y, Kudo S-E, Mori K (2018) Towards automated colonoscopy diagnosis: binary polyp size estimation via unsupervised depth learning. In: Proceedings of medical image computing and computer assisted intervention, pp 611–619
Ma R, Wang R, Pizer S, Rosenman J, McGill SK, Frahm J-H (2019) Real-time 3D reconstruction of colonoscopic surfaces for determining missing regions. In: Proceedings of medical image computing and computer assisted intervention, pp 573–582
Chen, RJ, Bobrow TL, Athey T, Mahmood F, Durr NJ (2019) slam endoscopy enhanced by adversarial depth prediction. In: Proceedings of KDD’19 workshop on applied data science for healthcare
Saxena A, Sung HC, Andrew YN (2006) Learning depth from single monocular images. Adv Neural Inf Process Syst 18:1161–1168
Eigen D, Fergus R (2015) Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: Proceedings of IEEE international conference on computer vision, pp 2650–2658
Ma F, Karaman S (2018) Sparse-to-dense: depth prediction from sparse depth samples and a single image. In: Proceedings of IEEE international conference on robotics and automation, pp 4796–4803
Prados E, Faugeras O (2006) Shape from shading. Handbook of mathematical models in computer vision. Springer, Berlin, pp 375–388
Hartley R, Zisserman A (2003) Multiple view geometry in computer vision. Cambridge University Press, Cambridge
Garg R, Vijay Kumar BG, Carneiro G, Reid I (2016) Unsupervised CNN for single view depth estimation: geometry to the rescue. In: Proceedings of European conference on computer vision, pp 740–756
Zhou T, Brown M, Snavely N, Lowe DG (2017) Unsupervised learning of depth and ego-motion from video. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 6612–6619
Wang C, Buenaposada JM, Zhu R, Lucey S (2018) Learning depth from monocular videos using direct methods. In: Proceedings of IEEE conference on computer vision and pattern, pp 2022–2030
Godard C, Aodha OM, Firman M, Brostow G (2019) Digging into self-supervised monocular depth estimation. In: Proceedings of IEEE international conference on computer vision, pp 3827–3837
Mori K, Suenaga Y, Toriwaki J (2003) Fast software-based volume rendering using multimedia instructions on PC platforms and its application to virtual endoscopy. In: Proceedings of SPIE Med Imaging 5031:111–122
Belhumeur PN, Kriegman DJ, Yuille AL (1999) The bas-relief ambiguity. Int J Comput Vis 35(1):33–44
Faisal M, Nicholas JD (2018) Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy. Med Image Anal 48:230–243
Rau A, Edwards PJE, Ahmad OF, Riordan P, Janatka M, Lovat LB, Stoyanov D (2019) Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy. Int J Comput Assist Radiol Surg 14:1167–1176
Oda M, Tanaka K, Takabatake H, Mori M, Natori H, Mori K (2019) Realistic endoscopic image generation method using virtual-to-real image-domain translation. IET Healthcare Technol Lett 6(6):214–219
Mathew S, Nadeem S, Kumari S, Kaufman A (2020) Augmenting colonoscopy using extended and directional CycleGAN for lossy image translation. In: Proceedings of IEEE international conference on computer vision, pp 4695–4704
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, pp 2672–2680
Zhu J-Y, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of IEEE international conference on computer vision, pp 2242–2251
Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334
Cignoni P, Callieri M, Corsini M, Dellepiane M, Ganovelli F, Ranzuglia G (2008) MeshLab: an open-source mesh processing tool. In: Proceedings of eurographics Italian chapter conference
Chu C, Zhmoginov A, Sandler M (2017) CycleGAN, a master of steganography. In: Proceedings of NIPS 2017 workshop “machine deception”
Acknowledgements
This study was funded by Grants from AMED (19hs0110006h0003), JSPS MEXT KAKENHI (26108006, 17H00867, 17K20099), and the JSPS Bilateral Joint Research Project.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Kudo SE and Misawa M received lecture fees from Olympus. Mori Y received consultant fees and lecture fees from Olympus. Mori K is supported by Cybernet Systems and Olympus (research grant) in this work, and by NTT outside of the submitted work. The other authors have no conflicts of interest.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical committee of Nagoya University (No. 357), and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Informed consent was obtained via an opt-out procedure from all individual participants included in the study.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Itoh, H., Oda, M., Mori, Y. et al. Unsupervised colonoscopic depth estimation by domain translations with a Lambertian-reflection keeping auxiliary task. Int J CARS 16, 989–1001 (2021). https://doi.org/10.1007/s11548-021-02398-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-021-02398-x