Skip to main content

Advertisement

SpringerLink
Log in

Disparity-constrained stereo endoscopic image super-resolution

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

With the increasing usage of stereo cameras in computer-assisted surgery techniques, surgeons can benefit from better 3D context of the surgical site in minimally invasive operations. However, since stereo cameras are placed together at the confined endoscope tip, the size of lens and sensors is limited, resulting in low resolution of stereo endoscopic images. How to effectively exploit and utilize stereo information in stereo endoscopic super-resolution (SR) becomes a challenging problem.

Methods

In this work, we propose a disparity-constrained stereo super-resolution network (DCSSRnet) to reconstruct images using a stereo image pair. In particular, a disparity constraint mechanism is incorporated into the generation of SR images in the deep neural network framework with effective feature extractors and atrous parallax attention modules.

Results

Extensive experiments were conducted to evaluate the performance of proposed DCSSRnet on the da Vinci dataset and Medtronic dataset. The results on endoscopic image datasets demonstrate that the proposed approach produces a more effective improvement over current SR methods on both quantitative measurements. The ablation studies further verify the effectiveness of the components of the proposed framework.

Conclusion

The proposed DCSSRnet provides a promising solution on enhancing the spatial resolution of stereo endoscopic image pairs. Specifically, the disparity consistency of the stereo image pair provides informative supervision for image reconstruction. The proposed model can serve as a tool for improving the quality of stereo endoscopic images of endoscopic surgery systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Shenbagaraj L, Thomas-Gibson S, Stebbing J, Broughton R, Dron M, Johnston D, Shaw T, Haboubi HN, Green JT (2019) Endoscopy in 2017: a national survey of practice in the UK. Front Gastroenterol 10(1):7–15. https://doi.org/10.1136/flgastro-2018-100970

    Article  Google Scholar 

  2. Kim M, Lee C, Hong N, Kim YJ, Kim S (2017) Development of stereo endoscope system with its innovative master interface for continuous surgical operation. Biomed Eng Online 16(1):1–16. https://doi.org/10.1186/s12938-017-0376-1

    Article  Google Scholar 

  3. Dong C, Loy CC, He K, Tang X (2015) Image super-resolution using deep convolutional networks. IEEE Trans on Pattern Anal Machine Intell 38(2):295–307. https://doi.org/10.1109/TPAMI.2015.2439281

    Article  Google Scholar 

  4. Kim J, Lee JK, Lee KM (2016) Accurate image super-resolution using very deep convolutional networks. . In Proceedings of IEEE conference on Computer Vision and Pattern Recognition (CVPR) 1646–1654. https://doi.org/10.1109/CVPR.2016.182

  5. Ahn N, Kang B, Sohn KA (2018) Fast, accurate, and lightweight super-resolution with cascading residual network. In Proceedings of European Conference on Computer Vision (ECCV): 252–268. https://doi.org/10.1007/978-3-030-01249-6_16

  6. Jeon DS, Baek SH, Choi I, Min HK (2018) Enhancing the Spatial Resolution of Stereo Images Using a Parallax Prior. In Proceedings of IEEE conference on Computer Vision and Pattern Recognition (CVPR): 1721–1730. https://doi.org/10.1109/CVPR.2018.00185

  7. Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873. https://doi.org/10.1109/TIP.2010.2050625

    Article  PubMed  Google Scholar 

  8. Glasner D, Bagon S, Irani M (2009) Super-resolution from a single image. In Proceedings of IEEE international conference on computer vision (ICCV): 349–356. https://doi.org/10.1109/ICCV.2009.5459271

  9. Freeman WT, Jones TR, Pasztor EC (2002) Example-based super-resolution. IEEE Comput Graph Appl 22(2):56–65. https://doi.org/10.1109/38.988747

    Article  Google Scholar 

  10. Tai Y, Yang J, Liu X (2017) Image super-resolution via deep recursive residual network. In Proceedings of IEEE conference on Computer Vision and Pattern Recognition (CVPR): 2790–2798. https://doi.org/10.1109/CVPR.2017.298

  11. Wang L, Guo Y, Wang Y, Liang Z, Lin Z, Yang J, An W (2020) Parallax attention for unsupervised stereo correspondence learning. IEEE transactions on pattern analysis and machine intelligence. https://doi.org/10.1109/TPAMI.2020.3026899

  12. Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y (2018) Image super-resolution using very deep residual channel attention networks. In Proceedings of European conference on computer vision (ECCV): 286–301. https://doi.org/10.1007/978-3-030-01234-2_18

  13. Chen Y, Shi F, Christodoulou A G, Xie Y, Zhou Z, Li D (2018) Efficient and accurate MRI super-resolution using a generative adversarial network and 3D multi-level densely connected network. In Proceedings of International conference on medical image computing and computer-assisted intervention (MICCAI): 91–99. https://doi.org/10.1007/978-3-030-00928-1_11

  14. Zhao C, Carass A, Dewey B E, Woo J, Oh J, Calabresi P A, Reich D S, Sati P, Pham D L, Prince J L (2018). A deep learning based anti-aliasing self super-resolution algorithm for MRI. In Proceedings of International conference on medical image computing and computer-assisted intervention (MICCAI): 100–108. https://doi.org/10.1007/978-3-030-00928-1_12

  15. Szczotka AB, Shakir DI, Ravì D, Clarkson MJ, Pereira SP, Vercauteren T (2020) Learning from irregularly sampled data for endomicroscopy super-resolution: a comparative study of sparse and dense approaches. Int J Comput Assis Radiol Surg 15(7):1167–1175. https://doi.org/10.1007/s11548-020-02170-7

    Article  Google Scholar 

  16. Bhavsar AV, Rajagopalan AN (2010) Resolution enhancement in multi-image stereo. IEEE Trans Pattern Anal Machine Intell 32(9):1721–1728. https://doi.org/10.1109/TPAMI.2010.90

    Article  Google Scholar 

  17. Yang M, Yu K, Zhang C, Li Z, Yang K (2018) DenseASPP for semantic segmentation in street scenes. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR): 3684–3692. https://doi.org/10.1109/CVPR.2018.00388

  18. Lerotic M, Yang G-Z (2006) The Use of Super Resolution in Robotic Assisted Minimally Invasive Surgery. In Proceedings of International conference on medical image computing and computer-assisted intervention (MICCAI): 462–469. https://doi.org/10.3109/10929080701727777

  19. Wang R, Zhang D, Li Q, Zhou XY, Lo B (2021) Real-time Surgical Environment Enhancement for Robot-Assisted Minimally Invasive Surgery Based on Super-Resolution. In Proceedings of international conference on robotics and automation (ICRA): 3434–3440

Download references

Acknowledgements

The authors would like to acknowledge Medtronic Inc. for providing part of stereo image data.

Funding

This research was partly supported by National Key R&D Program of China (No. 2019YFB1311503), National Natural Science Foundation of China (No. 62003208), Committee of Science and Technology, Shanghai, China (No. 19510711200), and Shanghai Sailing Program (No. 20YF1420800).

Author information

Authors and Affiliations

  1. Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, Shanghai, China

    Tianyi Zhang, Yun Gu, Xiaolin Huang & Jie Yang

  2. Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China

    Tianyi Zhang, Yun Gu, Xiaolin Huang, Jie Yang & Guang-Zhong Yang

  3. School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China

    Guang-Zhong Yang

Authors
  1. Tianyi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yun Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaolin Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jie Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guang-Zhong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jie Yang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

The study conducted experiments based on open-source available datasets. For this type of study, formal consent is not required.

Code availability

The code we used is available at https://github.com/hgfe/DCSSR.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, T., Gu, Y., Huang, X. et al. Disparity-constrained stereo endoscopic image super-resolution. Int J CARS 17, 867–875 (2022). https://doi.org/10.1007/s11548-022-02611-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-022-02611-5

Keywords

Search

Navigation