Hostname: page-component-8448b6f56d-wq2xx Total loading time: 0 Render date: 2024-04-24T02:19:30.069Z Has data issue: false hasContentIssue false
  • English
  • Français

Depth Estimation for Local Colon Structure in Monocular Capsule Endoscopy Based on Brightness and Camera Motion

Published online by Cambridge University Press:  27 May 2020

Lei Xu ,
Jing Li Open the ORCID record for Jing Li [Opens in a new window] ,
Yang Hao ,
Peisen Zhang ,
Gastone Ciuti ,
Paolo Dario  and
Qiang Huang
Lei Xu
Affiliation:
Advanced Innovation Center for Intelligent Robots and Systems, Beijing Institute of Technology, Beijing, China. E-mail: 7420180002@bit.edu.cn
Jing Li*
Affiliation:
Advanced Innovation Center for Intelligent Robots and Systems, Beijing Institute of Technology, Beijing, China. E-mail: 7420180002@bit.edu.cn
Yang Hao
Affiliation:
School of Mechatronical Engineering, Beijing Institute of Technology, Beijing, China. E-mails: 3120150088@bit.edu.cn, zpsbit@foxmail.com, qhuang@bit.edu.cn
Peisen Zhang
Affiliation:
School of Mechatronical Engineering, Beijing Institute of Technology, Beijing, China. E-mails: 3120150088@bit.edu.cn, zpsbit@foxmail.com, qhuang@bit.edu.cn
Gastone Ciuti
Affiliation:
Advanced Innovation Center for Intelligent Robots and Systems, Beijing Institute of Technology, Beijing, China. E-mail: 7420180002@bit.edu.cn The BioRobotics Institute, Scuola Superiore Sant'Anna, Pisa, Italy. E-mails: gastone.ciuti@santannapisa.it, paolo.dario@santannapisa.it
Paolo Dario
Affiliation:
Advanced Innovation Center for Intelligent Robots and Systems, Beijing Institute of Technology, Beijing, China. E-mail: 7420180002@bit.edu.cn The BioRobotics Institute, Scuola Superiore Sant'Anna, Pisa, Italy. E-mails: gastone.ciuti@santannapisa.it, paolo.dario@santannapisa.it
Qiang Huang
Affiliation:
Advanced Innovation Center for Intelligent Robots and Systems, Beijing Institute of Technology, Beijing, China. E-mail: 7420180002@bit.edu.cn School of Mechatronical Engineering, Beijing Institute of Technology, Beijing, China. E-mails: 3120150088@bit.edu.cn, zpsbit@foxmail.com, qhuang@bit.edu.cn
*
*Corresponding author. E-mail: 10902016@bit.edu.cn
Get access Rights & Permissions [Opens in a new window]

Summary

We present a 3D reconstruction method using brightness and camera motion estimation for registering local colon structure in colonoscopy. The proposed method is based on reverse projection from 2D fold contours to 3D space, motion estimation from 3D reconstructed points between neighboring frames, and model registration to reconstruct the fold structure. On the synthetic colon, the average percentages of the reconstructed depth error and circumference error are about 14.2% and 15.2%, respectively. The accuracy is enough for the navigation and control in capsule robot. This work demonstrates that the proposed method is superior to the methods using single-frame-based brightness intensity.

Keywords

3D reconstructionMotion estimationColon fold contoursModel registration
Type
Articles
Information
Robotica , Volume 39 , Issue 2 , February 2021 , pp. 334 - 345
Copyright
Copyright © The Author(s), 2020. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Smith, L., “Screening for colorectal cancer: Surveillance after resection of a colorectal cancer, and the removal of large adenomas,” Endoscopy 17, 98–102 (1987).Google Scholar
Harewood, G. C., “Relationship of colonoscopy completion rates and endoscopist features,” Digestive Dis. Sci. 50(1), 47–51 (2005).10.1007/s10620-005-1276-yCrossRefGoogle ScholarPubMed
Ciuti, G., Valdastri, P., Menciassi, A. and Dario, P., “Robotic magnetic steering and locomotion of capsule endoscope for diagnostic and surgical endoluminal procedures,” Robotica 28(2), 199–207 (2010).10.1017/S0263574709990361CrossRefGoogle Scholar
Furukawa, Y. and Ponce, J., “Accurate, dense, and robust multi-view stereopsis,” IEEE Trans. Pattern Anal. Mach. Intell. 32(8), 1362–1376 (2010).10.1109/TPAMI.2009.161CrossRefGoogle Scholar
Cremers, D. and Kolev, K., “Multi-view stereo and silhouette consistency via convex functionals over convex domains,” IEEE Trans. Pattern Anal. Mach. Intell. 33(6), 1161–1174 (2011).10.1109/TPAMI.2010.174CrossRefGoogle Scholar
Kazó, C. and Hajder, L., “Rapid Weak-Perspective Structure from Motion with Missing Data,” Proceedings of the IEEE International Conference on Computer Vision Workshops (2011) pp. 491–498.Google Scholar
Chandraker, M., “What Camera Motion Reveals about Shape with Unknown BRDF,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR) (2014) pp. 2179–2186.Google Scholar
Hajder, L. and Chetverikov, D., “Weak-perspective structure from motion for strongly contaminated data,” Pattern Recognit. Lett. 27(14), 1581–1589 (2006).10.1016/j.patrec.2006.03.007CrossRefGoogle Scholar
Ahmed, A. H. and Farag, A. A., “Shape from Shading for Hybrid Surfaces,” Proceedings of the IEEE International Conference on Image Processing (2007) pp. 525–528.Google Scholar
Ikeda, O., “Shape-from-Shading Algorithm for Oblique Light Source,” Proceedings of the International Symposium on Advances in Visual Computing (2007) pp. 357–366.10.1007/978-3-540-76856-2_35CrossRefGoogle Scholar
Ming, X., Zhao, R. C. and Maria, P., “Solving Self-Shadow Problem of Shape from Shading in Light Source Projected System,” Proceedings of the International Symposium on Intelligent Multimedia, Video and Speech Processing (2004) pp. 334–337.10.1109/ISIMP.2004.1434068CrossRefGoogle Scholar
VisentiniScarzanella, M., Stoyanov, D. and G. Yang, Z., “Metric Depth Recovery from Monocular Images Using Shape-from-Shading and Specularities,” Proceedings of the IEEE Conference on Image Processing (ICIP) (2013) pp. 25–28.Google Scholar
Ciuti, G., VisentiniScarzanella, M., Dore, A., Menciassi, A., Dario, P. and Yang, G. Z., “Intra-Operative Monocular 3D Reconstruction for Image-Guided Navigation in Active Locomotion Capsule Endoscopy,” Proceedings of the IEEE RAS and EMBS International Conference on Biomedical Robotics and Biomechatronics (2012) pp. 768–774.Google Scholar
Armin, M. A., Barnes, N., Alvarez, J., Li, H. D., Grimpen, F. and Salvado, O., “Learning Camera Pose from Optical Colonoscopy Frames Through Deep Convolutional Neural Network (CNN),” In: Computer Assisted and Robotic Endoscopy and Clinical Image-Based Procedures. 4th International Workshop (CARE, 2017) pp. 50–59.10.1007/978-3-319-67543-5_5CrossRefGoogle Scholar
Mahmood, F. and Durr, N. J., “Deep learning and conditional random fields-based depth estimation and topographical reconstruction from conventional endoscopy,” Med. Image Anal. 48, 230–243 (2018).10.1016/j.media.2018.06.005CrossRefGoogle ScholarPubMed
Rau, A., Edwards, P., Ahmad, O., Riordan, P., Janatka, M., Lovat, L. and Stoyanov, D., “Implicit domain adaptation with conditional generative adversarial networks for depth prediction in endoscopy,” Int. J. Comput. Assist. Radiol. Surg. 14(7), 1167–1176 (2019).10.1007/s11548-019-01962-wCrossRefGoogle ScholarPubMed
Hong, D., Tavanapong, W., Wong, J., Oh, J. and Groen, P. D., “3D Reconstruction of virtual colon structures from colonoscopy images,” Comput. Med. Imaging Graphics 38(1), 22–33 (2014).10.1016/j.compmedimag.2013.10.005CrossRefGoogle ScholarPubMed
Canny, J., “A computational approach to edge detection,” IEEE Trans. Pattern Anal. Mach. Intell. 8(6), 679–698 (1986).10.1109/TPAMI.1986.4767851CrossRefGoogle ScholarPubMed
Carsten, R. and Steger, A., “A comprehensive and versatile camera model for cameras with tilt lenses,” Int. J. Comput. Vis. 123(2), 121–159 (2017).Google Scholar
Bakstein, H. and Pajdla, T., “Panoramic Mosaicing with a 180 Degree Field of View Lens,” Proceedings of the IEEE Workshop on Omnidirectional Vision (2002) pp. 60–68.Google Scholar
Kaufman, A. and Wang, J., “3D surface reconstruction from endoscopic videos,” In: Visualization in Medicine and Life Sciences (Encarnação, J., eds.) (Springer Berlin Heidelberg, 2008) pp. 61–74.10.1007/978-3-540-72630-2_4CrossRefGoogle Scholar
Kazhdan, M., olitho, M. and Hoppe, H., “Poisson Surface Reconstruction,” Proceedings of the Symposium on Geometry Processing (2006) pp. 61–70.Google Scholar
Bruhn, A., Weickert, J., Feddern, C., Kohlberger, T. and Schnorr, C., “Variational optical flow computation in real-time,” IEEE Trans. Image Process. 14(5), 608–615 (2005).10.1109/TIP.2005.846018CrossRefGoogle Scholar
Nagel, H. H. and Enkelmann, W., “An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences,” IEEE Trans. Pattern Anal. Mach. Intell. 8(5), 565–593 (1986).10.1109/TPAMI.1986.4767833CrossRefGoogle ScholarPubMed
Brox, T., Bruhn, A., Papenberg, N. and Weickert, J., “High Accuracy Optical Flow Estimation Based on a Theory for Warping,” Proceedings of the European Conference on Computer Vision(ECCV) (2004) pp. 25–36.Google Scholar