Skip to main content

Advertisement

Log in

Multi-stages de-smoking model based on CycleGAN for surgical de-smoking

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Smoke generated during laparoscopic surgery blocks the doctor’s sight and degrades the quality of the images severely; thus, surgical de-smoking is a crucial task during laparoscopic surgery. Previous deep learning methods extract the features of smoke images to restore clear images using convolutional neural networks. However, these methods training on simulated images result in performance degradation when generalized to real smoke images. In this paper, we introduce cycle generative adversarial networks to bridge the gap between simulated and real surgical images. Therefore, we propose a multi-stages surgical de-smoking model based on cycle generative adversarial networks(MS-CycleGAN). By leveraging the convolutional neural networks-based de-smoking module in the first stage, we additionally utilize the simulated-to-real module in the second stage to pull simulated smoke-free images to the real surgical domain, generating real-like smoke-free images that even the discriminator cannot distinguish from real smoke-free images. Furthermore, to make real images and de-smoking images more consistent in image feature space instead of pixel space, the perceptual loss function is employed to calculate the loss in feature space. MS-CycleGAN outperforms state-of-the-art de-smoking methods on the evaluation metrics of both Peak Signal to Noise Ratio and Structural Similarity Index Measure. Most importantly, our MS-CycleGAN achieves qualitatively superior results on de-smoking for real surgical smoke images.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The data that support the findings of this study are available in cholec80 at 10.1109/TMI.2016.2593957, EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos. These data were derived from the following resources available in the public domain: https://paperswithcode.com/dataset/cholec80.

Notes

  1. https://github.com/SquidDev/Python-Clouds.

References

  1. Chen L, Tang W, John NW, Wan TR, Zhang JJ (2020) De-smokegcn: generative cooperative networks for joint surgical smoke detection and removal. IEEE Trans Med Imaging 39(5):1615–1625

    Article  Google Scholar 

  2. Tchaka K, Pawar V M, Stoyanov D (2017). Chromaticity based smoke removal in endoscopic images. In: Med. Imaging 2017: Image Processing, pp 463–470.

  3. Bolkar S, Wang C, Cheikh FA, Yildirim S (2018) Deep smoke removal from minimally invasive surgery videos. In: Proc. IEEE int. conf. image process, pp 3403–3407

  4. Chen D, He M, Fan Q, Liao J, Zhang L, Hou D, Yuan L, Hua G (2019) Gated context aggregation network for image dehazing and deraining. In: Proc. IEEE winter conf. appl. comput. vis., pp 1375–1383

  5. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Proc. Adv. Neural Inf. Process. Syst. 2:2672–2680

    Google Scholar 

  6. Bousmalis K, Silberman N, Dohan D, Erhan D, Krishnan D (2017) Unsupervised pixel-level domain adaptation with generative adversarial networks. In: Proc. IEEE conf. comput. vis. pattern recog., pp 3722–3731

  7. Chang H, Lu J, Yu F, Finkelstein A (2018) Pairedcyclegan: asymmetric style transfer for applying and removing makeup. In: Proc. IEEE conf. comput. vis. pattern recog., pp 40–48

  8. Zhu J-Y, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proc. IEEE conf. comput. vis. pattern recog., pp 2223–2232

  9. Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: Proc. Eur. conf. comput. vis., pp 694–711

  10. Hide R (1977) Optics of the atmosphere: scattering by molecules and particles. Phys Bull 28(11):521

    Article  Google Scholar 

  11. He K, Sun J, Tang X (2010) Single image haze removal using dark channel prior. IEEE Trans Pattern Anal Mach Intell 33(12):2341–2353

    Google Scholar 

  12. Zhu Q, Mai J, Shao L (2015) A fast single image haze removal algorithm using color attenuation prior. IEEE Trans Image Process 24(11):3522–3533

    Article  MathSciNet  MATH  Google Scholar 

  13. Berman D, treibitz T, Avidan S (2016) Non-local image dehazing. In: Proc. IEEE conf. comput. vis. pattern recog., pp 1674–1682

  14. Wang C, Cheikh FA, Kaaniche M, Beghdadi A, Elle OJ (2018) Variational based smoke removal in laparoscopic images. Biomed Eng Online 17(1):1–18

    Article  Google Scholar 

  15. Kotwal A, Bhalodia R, Awate SP (2016) Joint desmoking and denoising of laparoscopy images. In: Proc. IEEE comput. soc. conf. comput. vis. pattern recogn., pp 1050–1054

  16. Baid A, Kotwal A, Bhalodia R, Merchant S, Awate SP (2017) Joint desmoking, specularity removal, and denoising of laparoscopy images via graphical models and Bayesian inference. In: Proc. IEEE comput. soc. conf. comput. vis. pattern recogn., pp 732–736

  17. Luo X, McLeod AJ, Pautler SE, Schlachta CM, Peters TM (2017) Vision-based surgical field defogging. IEEE Trans Med Imaging 36(10):2021–2030

    Article  Google Scholar 

  18. Cai B, Xu X, Jia K, Qing C, Tao D (2016) Dehazenet: an end-to-end system for single image haze removal. IEEE Trans Image Process 25(11):5187–5198

    Article  MathSciNet  MATH  Google Scholar 

  19. Li B, Peng X, Wang Z, Xu J, Feng D (2017) Aod-net: all-in-one dehazing network. In: Proc. IEEE conf. comput. vis. pattern recog., pp 4770–4778

  20. Kanakatte A, Seemakurthy K, Gubbi J, Saha J, Ghose A, Purushothaman B (2021) Surgical smoke dehazing and color reconstruction. In: Proc. IEEE comput. soc. conf. comput. vis. pattern recogn. IEEE, pp 280–284

  21. Ren W, Ma L, Zhang J, Pan J, Cao X, Liu W, Yang M-H (2018) Gated fusion network for single image dehazing. In: Proc. IEEE conf. comput. vis. pattern recogn., pp 3253–3261

  22. Wang C, Mohammed AK, Cheikh FA, Beghdadi A, Elle OJ (2019) Multiscale deep desmoking for laparoscopic surgery. In: Med. imaging 2019: image process, vol 10949, pp 109491Y–1

  23. Sengar V, Seemakurthy K, Gubbi J (2021) Multi-task learning based approach for surgical video desmoking. In: Proceedings of the twelfth Indian conference on computer vision, graphics and image processing, pp 1–9

  24. Azam MA, Khan KB, Rehman E, Khan SU (2022) Smoke removal and image enhancement of laparoscopic images by an artificial multi-exposure image fusion method. Soft Comput 26:8003–8015

    Article  Google Scholar 

  25. Bai H, Pan J, Xiang X, Tang J (2022) Self-guided image dehazing using progressive feature fusion. IEEE Trans Image Process 31:1217–1229

    Article  Google Scholar 

  26. Salazar-Colores S, Jiménez HM, Ortiz-Echeverri CJ, Flores G (2020) Desmoking laparoscopy surgery images using an image-to-image translation guided by an embedded dark channel. IEEE Access 8:208898–208909

    Article  Google Scholar 

  27. Vishal V, Sharma N, Singh M (2019) Guided unsupervised desmoking of laparoscopic images using cycle-desmoke. OR 2.0 context-aware operating theaters and machine learning in clinical neuroimaging. Springer, New York, pp 21–28

    Chapter  Google Scholar 

  28. Venkatesh V, Sharma N, Srivastava V, Singh M (2020) Unsupervised smoke to desmoked laparoscopic surgery images using contrast driven cyclic-desmokegan. Comput Biol Med 123:103873

    Article  Google Scholar 

  29. Huang Y, Chen X, Xu L, Li K (2021) Single image desmoking via attentive generative adversarial network for smoke detection process. Fire Technol 57(6):3021–3040

    Article  Google Scholar 

  30. Wu H, Qu Y, Lin S, Zhou J, Qiao R, Zhang Z, Xie Y, Ma L (2021) Contrastive learning for compact single image dehazing. In: Proc. IEEE conf. comput. vis. pattern recogn., pp 10551–10560

  31. Chen X, Fan Z, Li P, Dai L, Kong C, Zheng Z, Huang Y, Li Y (2022) Unpaired deep image dehazing using contrastive disentanglement learning. In: European conference on computer vision. Springer, pp 632–648

  32. Lin T-Y, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: Proc. IEEE conf. comput. vis. pattern recogn., pp 2117–2125

  33. Kirillov A, Girshick R, He K, Dollar P (2019) Panoptic feature pyramid networks. In: Proc. IEEE conf. comput. vis. pattern recogn., pp 6399–6408

  34. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  35. Twinanda AP, Shehata S, Mutter D, Marescaux J, De Mathelin M, Padoy N (2016) Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36(1):86–97

    Article  Google Scholar 

  36. Leibetseder A, Primus MJ, Petscharnig S, Schoeffmann K (2017) Real-time image-based smoke detection in endoscopic videos. In: Proc. themat. workshops ACM multimed., pp 296–304

  37. Hore A, Ziou D (2010) Image quality metrics: Psnr vs. ssim. In: Int. conf. pattern recognit., pp 2366–2369

  38. Shao Y, Li L, Ren W, Gao C, Sang N (2020) Domain adaptation for image dehazing. In: Proc. IEEE conf. comput. vis. pattern recog., pp 2808–2817

  39. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Proc. int. conf. med. image comp. comput.-assisted intervention, pp 234–241

  40. Chaurasia A, Culurciello E (2017) Linknet: exploiting encoder representations for efficient semantic segmentation. In: Proc. IEEE vis. commun. image process., pp 1–4

  41. Cheng K, You J, Wu S, Chen Z, Zhou Z, Guan J, Peng B, Wang X (2021) Artificial intelligence-based automated laparoscopic cholecystectomy surgical phase recognition and analysis. Surg Endosc 36(5):3160–3168

    Article  Google Scholar 

  42. Jin Y, Long Y, Chen C, Zhao Z, Dou Q, Heng P-A (2021) Temporal memory relation network for workflow recognition from surgical video. IEEE Trans Med Imaging 40(7):1911–1923

    Article  Google Scholar 

  43. Kondo S (2021) Lapformer: surgical tool detection in laparoscopic surgical video using transformer architecture. Computer Methods Biomech Biomed Eng Imaging Vis 9(3):302–307

    Article  Google Scholar 

  44. Yi F, Jiang T (2021) Not end-to-end: Explore multi-stage architecture for online surgical phase recognition. arXiv preprint. arXiv:2107.04810

  45. Loukas C (2018) Surgical phase recognition of short video shots based on temporal modeling of deep features. arXiv preprint. arXiv:1807.07853

  46. Yang Y, Zhao Z, Shi P, Hu S (2021) An efficient one-stage detector for real-time surgical tools detection in robot-assisted surgery. Annual conference on medical image understanding and analysis. Springer, Berlin, pp 18–29

    Google Scholar 

  47. Gao X, Jin Y, Long Y, Dou Q, Heng P-A (2021) Trans-svnet: accurate phase recognition from surgical videos via hybrid embedding aggregation transformer. International conference on medical image computing and computer-assisted intervention. Springer, New York, pp 593–603

    Google Scholar 

Download references

Acknowledgements

This work presented in this paper is partially supported by grants from National Natural Science Foundation of China (No. 61772225), Guangdong Basic and Applied Basic Research Foundation (No. 2020A1515010558 and No. 2021A1515011972).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Qiuxia Wu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Su, X., Wu, Q. Multi-stages de-smoking model based on CycleGAN for surgical de-smoking. Int. J. Mach. Learn. & Cyber. 14, 3965–3978 (2023). https://doi.org/10.1007/s13042-023-01875-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-023-01875-w

Keywords

Navigation