Skip to main content

Advertisement

SpringerLink
Log in

Development and validation of a deep learning-based laparoscopic system for improving video quality

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

This article has been updated

Abstract

Purpose

A clear surgical field of view is a prerequisite for successful laparoscopic surgery. Surgical smoke, image blur, and lens fogging can affect the clarity of laparoscopic imaging. We aimed to develop a real-time assistance system (namely LVQIS) for removing these interfering factors during laparoscopic surgery, thereby improving laparoscopic video quality.

Methods

LVQIS was developed with generative adversarial networks (GAN) and transfer learning, which included two classification models (ResNet-50), a motion blur removal model (MPRNet), and a smoke/fog removal model (GAN). 136 laparoscopic surgery videos were retrospectively collected in a tripartite dataset for training and validation. A synthetic dataset was simulated using the image enhancement library Albumentations and the 3D rendering software Blender. The objective evaluation results were through PSNR, SSIM and FID, and the subjective evaluation includes the operation pause time and the degree of anxiety of surgeons.

Results

The synthesized dataset contained 19,245 clear images, 19,245 motion blur images, and 19,245 smoke/fog images. The ResNet-50 CNN model identified whether a single laparoscopic image had motion blur and smoke/fog with an accuracy of over 0.99. The PSNR, SSIM and FID of the de-smoke model were 29.67, 0.9551 and 74.72, respectively, and the PSNR, SSIM and FID of the de-blurring model were 26.78, 0.9020 and 80.10, respectively, which were better than other advanced de-blurring and de-smoke/fog models. In a comparative study of 100 laparoscopic surgeries, the use of LVQIS significantly reduced the operation pause time (P < 0.001) and the anxiety of surgeons (P = 0.004).

Conclusions

In this study, LVQIS is an efficient and robust system that can improve the quality of laparoscopic video, reduce surgical pause time and the anxiety of surgeons, and has the potential for real-time application in real clinical settings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability

Code and data to be released with this paper.

Change history

  • 05 November 2022

    Small error in the Discussion section, penultimate paragraph: the word has been from changed “Interfering” to “interfering”

Abbreviations

DL:

Deep learning

GAN:

Generative adversarial networks

CNN:

Convolutional neural network

SSIM:

Structural similarity index measure

PSNR:

Peak signal-to-noise ratio

FID:

Frechet inception distance

VRT:

Video restoration transformer

AOD-NET:

All-in-one dehazing network

FFA-NET:

Feature fusion attention network

EDN-GTM:

Encoder-decoder network for transfer graphs

DW-GAN:

Discrete wavelet transform GAN

References

  1. Johnson A (1997) Laparoscopic surgery. Lancet 349:631–635

    Article  CAS  Google Scholar 

  2. Trilling B, Mancini A, Fiard G, Barraud PA, Decrouez M, Vijayan S, Tummers M, Faucheron JL, Silvent S, Schwartz C, Voros S (2021) Improving vision for surgeons during laparoscopy: the enhanced laparoscopic vision system (ELViS). SURG ENDOSC 35:2403–2415

    Article  Google Scholar 

  3. Lawrentschuk N, Fleshner NE, Bolton DM (2010) Laparoscopic lens fogging: a review of etiology and methods to maintain a clear visual field. J Endourol 24:905–913

    Article  Google Scholar 

  4. Stoyanov D (2012) Surgical vision. Ann Biomed Eng 40:332–345

    Article  Google Scholar 

  5. Ha HI, Choi MC, Jung SG, Joo WD, Lee C, Song SH, Park H (2019) Chemicals in surgical smoke and the efficiency of built-in-filter ports. JSLS 23:e201900037

    Article  Google Scholar 

  6. Choi SH, Kwon TG, Chung SK, Kim TH (2014) Surgical smoke may be a biohazard to surgeons performing laparoscopic surgery. Surg Endosc 28:2374–2380

    Article  Google Scholar 

  7. Beckmeier L, Klapdor R, Soergel P, Kundu S, Hillemanns P, Hertel H (2014) Evaluation of active camera control systems in gynecological surgery: construction, handling, comfort, surgeries and results. Arch Gynecol Obstet 289:341–348

    Article  Google Scholar 

  8. Song T, Lee DH (2020) A randomized Comparison of laparoscopic LEns defogging using Anti-fog solution, waRm saline, and chlorhexidine solution (CLEAR). Surg Endosc 34:940–945

    Article  Google Scholar 

  9. Baid A, Kotwal A, Bhalodia R, Merchant SN, Awate SP (2017), IEEE: joint desmoking, specularity removal, and denoising of laparoscopy images via graphical models and bayesian inference. In: 2017 IEEE 14th international symposium on biomedical imaging (ISBI 2017), IEEE 14th international symposium on biomedical imaging (ISBI) - From Nano to Macro, pp 732–736

  10. Kotwal A, Bhalodia R, Awate SP (2016) IEEE: joint desmoking and denoising of laparoscopy images. In: 2016 IEEE 13th international symposium on biomedical imaging (ISBI), 13th IEEE International Symposium on Biomedical Imaging (ISBI), pp 1050–1054

  11. Luo XB, McLeod AJ, Pautler SE, Schlachta CM, Peters TM (2017) Vision-based surgical field defogging. IEEE T Med Imaging 36:2021–2030

    Article  Google Scholar 

  12. Shin J, Kim M, Paik J, Lee S (2020) Radiance-reflectance combined optimization and structure-guided $\ell _0$-norm for single image dehazing. IEEE T Multimedia 22:30–44

    Article  Google Scholar 

  13. Wang C, Cheikh FA, Kaaniche M, Elle OJ (2018) A smoke removal method for laparoscopic images. arXiv preprint arXiv:1803.08410

  14. Wang C, Alaya CF, Kaaniche M, Beghdadi A, Elle OJ (2018) Variational based smoke removal in laparoscopic images. Biomed Eng Online 17:139

    Article  Google Scholar 

  15. Gu L, Liu PL, Jiang CH, Luo M, Xu Q (2015) Virtual digital defogging technology improves laparoscopic imaging quality. Surg Innov 22:171–176

    Article  Google Scholar 

  16. Zhu QS, Mai JM, Shao L (2015) A fast single image haze removal algorithm using color attenuation prior. IEEE T Image Process 24:3522–3533

    Article  Google Scholar 

  17. Tchaka K, Pawar VM, Stoyanov D (2017) Chromaticity Based Smoke Removal in Endoscopic Images. In: Medical Imaging 2017: Image Processing. Styner MA and Angelini ED (eds.), Conference on Medical Imaging - Image Processing

  18. Bolkar S, Wang CC, Cheikh FA, Yildirim S (2018) IEEE: deep smoke removal from minimally invasive surgery videos. In: 2018 25th IEEE international conference on image processing (ICIP), 25th IEEE International Conference on Image Processing (ICIP), pp 3403–3407

  19. Vishal V, Sharma N, Singh M (2019) Guided Unsupervised Desmoking of Laparoscopic Images Using Cycle-Desmoke. In: OR 2.0 context-aware operating theaters and machine learning in clinical neuroimaginG. Zhou L, Sarikaya D and Kia SM, et al. (eds.), 2nd International Workshop on Context-Aware Surgical Theaters (OR)/2nd International Workshop on Machine Learning in Clinical Neuroimaging (MLCN), pp 21–28

  20. Vishal V, Lochan K, Venkatesh V, Singh M, Sharma N (2020) Unsupervised Desmoking of Laparoscopy Images Using Multi-scale DesmokeNet. Springer, Cham

    Book  Google Scholar 

  21. Wang CC, Mohammed AK, Cheikh FA, Beghdadi A, Elle OJ (2019) Multiscale deep desmoking for laparoscopic surgery. In: Medical Imaging 2019: Image Processing. Angelini ED and Landman BA (eds.), Conference on Medical Imaging: Image Processing

  22. Pan Y, Bano S, Vasconcelos F, Park H, Jeong TT, Stoyanov D (2022) DeSmoke-LAP: improved unpaired image-to-image translation for desmoking in laparoscopic surgery. Int J Comput Assist Radiol Surg 17:885–893

    Article  Google Scholar 

  23. Mascagni P, Vardazaryan A, Alapatt D, Urade T, Emre T, Fiorillo C, Pessaux P, Mutter D, Marescaux J, Costamagna G, Dallemagne B, Padoy N (2022) Artificial intelligence for surgical safety: automatic assessment of the critical view of safety in laparoscopic cholecystectomy using deep learning. Ann Surg 275:955–961

    Article  Google Scholar 

  24. Zheng Q, Yang R, Yang S, Ni X, Li Y, Jiang Z, Wang X, Wang L, Chen Z, Liu X (2022) Development and validation of a deep-learning based assistance system for enhancing laparoscopic control level. Int J Med Robot Comput Assist Surg : MRCAS. https://doi.org/10.1002/rcs.2449

    Article  Google Scholar 

  25. Zhang J, Gao X (2020) Object extraction via deep learning-based marker-free tracking framework of surgical instruments for laparoscope-holder robots. Int J Comput Assist Radiol Surg 15:1335–1345

    Article  Google Scholar 

  26. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Advances in neural information processing systems, vol 27. Curran, Red Hook, NY

    Google Scholar 

  27. Lu T, Chen T, Gao F, Sun B, Ntziachristos V, Li J (2021) LV-GAN: a deep learning approach for limited-view optoacoustic imaging based on hybrid datasets. J BIOPHOTONICS 14:e202000325

    Article  Google Scholar 

  28. Ma Y, Liu J, Liu Y, Fu H, Hu Y, Cheng J, Qi H, Wu Y, Zhang J, Zhao Y (2021) Structure and illumination constrained gan for medical image enhancement. IEEE Trans Med Imaging 40:3955–3967

    Article  Google Scholar 

  29. Liu T, de Haan K, Rivenson Y, Wei Z, Zeng X, Zhang Y, Ozcan A (2019) Deep learning-based super-resolution in coherent imaging systems. Sci Rep 9:3926

    Article  Google Scholar 

  30. He KM, Zhang XY, Ren SQ, Sun J (2016) IEEE: Deep Residual Learning for Image Recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 770–778, 2016

  31. Zamir SW, Arora A, Khan S, Hayat M, Khan FS, Yang MH, Shao L (2021) IEEE CS: Multi-Stage Progressive Image Restoration. In: 2021 IEEE/CVF conference on computer vision and pattern recognition, CVPR 2021, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 14816–14826, 2021

  32. Salazar-Colores S, Jiménez HM, Ortiz-Echeverri CJ, Flores G (2020) Desmoking laparoscopy surgery images using an image-to-image translation guided by an embedded dark channel. IEEE ACCESS 8:208898–208909

    Article  Google Scholar 

  33. Twinanda AP, Shehata S, Mutter D, Marescaux J, Padoy N (2016) EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos. IEEE T MED IMAGING 36:86–97

    Article  Google Scholar 

  34. Stauder R, Ostler D, Kranzfelder M, Koller S, Feußner H, Navab N (2016) The TUM LapChole dataset for the M2CAI 2016 workflow challenge. arXiv preprint arXiv:1610.09278.

  35. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13:600–612

    Article  Google Scholar 

  36. Hensel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S 2017 GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In: advances in neural information processing systems 30 (NIPS 2017). Guyon I, Luxburg UV and Bengio S, et al. (eds.), 31st Annual Conference on Neural Information Processing Systems (NIPS)

  37. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

  38. Kupyn O, Budzan V, Mykhailych M, Mishkin D, Matas J (2018) IEEE: DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks. In: 2018 IEEE/CVF conference on computer vision and pattern recognition (CVPR), 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 8183–8192

  39. Kupyn O, Martyniuk T, Wu JR, Wang ZY (2019) IEEE: DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better. In: 2019 IEEE/CVF international conference on computer vision (ICCV 2019), IEEE/CVF International Conference on Computer Vision (ICCV), pp 8877–8886

  40. Liang J, Cao J, Fan Y, Zhang K, Ranjan R, Li Y, Timofte R, Van Gool L (2022) Vrt: A video restoration transformer. arXiv preprint arXiv:2201.12288.

  41. Li BY, Peng XL, Wang ZY, Xu JZ, Feng D (2017) IEEE: AOD-Net: All-in-One Dehazing Network. In: 2017 IEEE international conference on computer vision (ICCV), 16th IEEE International Conference on Computer Vision (ICCV), pp 4780–4788

  42. Qin X, Wang ZL, Bai YC, Xie XD, Jia HZ (2020) Assoc AAI: FFA-Net: Feature Fusion Attention Network for Single Image Dehazing. In: thirty-fourth AAAI conference on artificial intelligence, the thirty-second innovative applications of artificial intelligence conference and the tenth AAAI symposium on educational advances in artificial intelligence, 34th AAAI Conference on Artificial Intelligence/32nd Innovative Applications of Artificial Intelligence Conference/10th AAAI Symposium on Educational Advances in Artificial Intelligence, pp 11908–11915

  43. Tran L, Moon S, Park D (2022) A novel encoder-decoder network with guided transmission map for single image dehazing. arXiv preprint arXiv:2202.04757.

  44. Fu MH, Liu H, Yu YK, Chen J, Wang KY (2021) IEEE CS: DW-GAN: A Discrete Wavelet Transform GAN for NonHomogeneous Dehazing. In: 2021 IEEE/CVF conference on computer vision and pattern recogition workshops (CVPRW 2021), IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 203–212

  45. Chen L, Tang W, John NW, Wan TR, Zhang JJ (2020) De-smokeGCN: generative cooperative networks for joint surgical smoke detection and removal. IEEE Trans Med Imaging 39:1615–1625

    Article  Google Scholar 

  46. Gu L, Liu PL, Zhou H, Xu Q (2018) A pilot study for a better visibility in the 3D laparoscopic right colectomy surgery. World J Surg 42:1872–1876

    Article  Google Scholar 

  47. Chen L, Tang W, John NW, Wan TR, Zhang JJ (2018) SLAM-based dense surface reconstruction in monocular minimally invasive surgery and its application to augmented reality. Comput Meth Prog Bio 158:135–146

    Article  Google Scholar 

Download references

Acknowledgements

We acknowledge the grant from the Project of Hubei Province Key Research and Development Project of China (Grant No. 2020BCB051), the Ministry of Education Industry-University Cooperation and Collaborative Education Project of China (Grant No. 202102012021), and the National Medical Education Development Center Medical Simulation Education Research Project of China (Grant No. 2021MNYB11) for supporting the work.

Author information

Author notes

  1. Qingyuan Zheng and Rui Yang have contributed equally to this work.

Authors and Affiliations

  1. Department of Urology, Renmin Hospital of Wuhan University, 99 Zhang Zhi-dong Road, Wuhan, Hubei, 430060, People’s Republic of China

    Qingyuan Zheng, Rui Yang, Xinmiao Ni, Song Yang, Zhengyu Jiang, Lei Wang, Zhiyuan Chen & Xiuheng Liu

  2. Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan, 430060, Hubei, China

    Qingyuan Zheng, Rui Yang, Xinmiao Ni, Song Yang, Zhengyu Jiang, Lei Wang, Zhiyuan Chen & Xiuheng Liu

Authors
  1. Qingyuan Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rui Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinmiao Ni
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Song Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhengyu Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Zhiyuan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xiuheng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

XHL reports being Director and the sponsor of this study. QYZ and RY performed the research, analysed the data, and wrote the paper. XMN, SY, ZYJ, LW and ZYC contributed essential tools.

Corresponding authors

Correspondence to Zhiyuan Chen or Xiuheng Liu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflicts of interest with regard to the work presented in this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Visual comparison of laparoscopic video clip with and without LVQIS: https://youtu.be/-0pR_3JmaPw.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 22 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zheng, Q., Yang, R., Ni, X. et al. Development and validation of a deep learning-based laparoscopic system for improving video quality. Int J CARS 18, 257–268 (2023). https://doi.org/10.1007/s11548-022-02777-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-022-02777-y

Keywords

Search

Navigation