Abstract
Smoke generated during laparoscopic surgery blocks the doctor’s sight and degrades the quality of the images severely; thus, surgical de-smoking is a crucial task during laparoscopic surgery. Previous deep learning methods extract the features of smoke images to restore clear images using convolutional neural networks. However, these methods training on simulated images result in performance degradation when generalized to real smoke images. In this paper, we introduce cycle generative adversarial networks to bridge the gap between simulated and real surgical images. Therefore, we propose a multi-stages surgical de-smoking model based on cycle generative adversarial networks(MS-CycleGAN). By leveraging the convolutional neural networks-based de-smoking module in the first stage, we additionally utilize the simulated-to-real module in the second stage to pull simulated smoke-free images to the real surgical domain, generating real-like smoke-free images that even the discriminator cannot distinguish from real smoke-free images. Furthermore, to make real images and de-smoking images more consistent in image feature space instead of pixel space, the perceptual loss function is employed to calculate the loss in feature space. MS-CycleGAN outperforms state-of-the-art de-smoking methods on the evaluation metrics of both Peak Signal to Noise Ratio and Structural Similarity Index Measure. Most importantly, our MS-CycleGAN achieves qualitatively superior results on de-smoking for real surgical smoke images.





Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availability
The data that support the findings of this study are available in cholec80 at 10.1109/TMI.2016.2593957, EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos. These data were derived from the following resources available in the public domain: https://paperswithcode.com/dataset/cholec80.
References
Chen L, Tang W, John NW, Wan TR, Zhang JJ (2020) De-smokegcn: generative cooperative networks for joint surgical smoke detection and removal. IEEE Trans Med Imaging 39(5):1615–1625
Tchaka K, Pawar V M, Stoyanov D (2017). Chromaticity based smoke removal in endoscopic images. In: Med. Imaging 2017: Image Processing, pp 463–470.
Bolkar S, Wang C, Cheikh FA, Yildirim S (2018) Deep smoke removal from minimally invasive surgery videos. In: Proc. IEEE int. conf. image process, pp 3403–3407
Chen D, He M, Fan Q, Liao J, Zhang L, Hou D, Yuan L, Hua G (2019) Gated context aggregation network for image dehazing and deraining. In: Proc. IEEE winter conf. appl. comput. vis., pp 1375–1383
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Proc. Adv. Neural Inf. Process. Syst. 2:2672–2680
Bousmalis K, Silberman N, Dohan D, Erhan D, Krishnan D (2017) Unsupervised pixel-level domain adaptation with generative adversarial networks. In: Proc. IEEE conf. comput. vis. pattern recog., pp 3722–3731
Chang H, Lu J, Yu F, Finkelstein A (2018) Pairedcyclegan: asymmetric style transfer for applying and removing makeup. In: Proc. IEEE conf. comput. vis. pattern recog., pp 40–48
Zhu J-Y, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proc. IEEE conf. comput. vis. pattern recog., pp 2223–2232
Johnson J, Alahi A, Fei-Fei L (2016) Perceptual losses for real-time style transfer and super-resolution. In: Proc. Eur. conf. comput. vis., pp 694–711
Hide R (1977) Optics of the atmosphere: scattering by molecules and particles. Phys Bull 28(11):521
He K, Sun J, Tang X (2010) Single image haze removal using dark channel prior. IEEE Trans Pattern Anal Mach Intell 33(12):2341–2353
Zhu Q, Mai J, Shao L (2015) A fast single image haze removal algorithm using color attenuation prior. IEEE Trans Image Process 24(11):3522–3533
Berman D, treibitz T, Avidan S (2016) Non-local image dehazing. In: Proc. IEEE conf. comput. vis. pattern recog., pp 1674–1682
Wang C, Cheikh FA, Kaaniche M, Beghdadi A, Elle OJ (2018) Variational based smoke removal in laparoscopic images. Biomed Eng Online 17(1):1–18
Kotwal A, Bhalodia R, Awate SP (2016) Joint desmoking and denoising of laparoscopy images. In: Proc. IEEE comput. soc. conf. comput. vis. pattern recogn., pp 1050–1054
Baid A, Kotwal A, Bhalodia R, Merchant S, Awate SP (2017) Joint desmoking, specularity removal, and denoising of laparoscopy images via graphical models and Bayesian inference. In: Proc. IEEE comput. soc. conf. comput. vis. pattern recogn., pp 732–736
Luo X, McLeod AJ, Pautler SE, Schlachta CM, Peters TM (2017) Vision-based surgical field defogging. IEEE Trans Med Imaging 36(10):2021–2030
Cai B, Xu X, Jia K, Qing C, Tao D (2016) Dehazenet: an end-to-end system for single image haze removal. IEEE Trans Image Process 25(11):5187–5198
Li B, Peng X, Wang Z, Xu J, Feng D (2017) Aod-net: all-in-one dehazing network. In: Proc. IEEE conf. comput. vis. pattern recog., pp 4770–4778
Kanakatte A, Seemakurthy K, Gubbi J, Saha J, Ghose A, Purushothaman B (2021) Surgical smoke dehazing and color reconstruction. In: Proc. IEEE comput. soc. conf. comput. vis. pattern recogn. IEEE, pp 280–284
Ren W, Ma L, Zhang J, Pan J, Cao X, Liu W, Yang M-H (2018) Gated fusion network for single image dehazing. In: Proc. IEEE conf. comput. vis. pattern recogn., pp 3253–3261
Wang C, Mohammed AK, Cheikh FA, Beghdadi A, Elle OJ (2019) Multiscale deep desmoking for laparoscopic surgery. In: Med. imaging 2019: image process, vol 10949, pp 109491Y–1
Sengar V, Seemakurthy K, Gubbi J (2021) Multi-task learning based approach for surgical video desmoking. In: Proceedings of the twelfth Indian conference on computer vision, graphics and image processing, pp 1–9
Azam MA, Khan KB, Rehman E, Khan SU (2022) Smoke removal and image enhancement of laparoscopic images by an artificial multi-exposure image fusion method. Soft Comput 26:8003–8015
Bai H, Pan J, Xiang X, Tang J (2022) Self-guided image dehazing using progressive feature fusion. IEEE Trans Image Process 31:1217–1229
Salazar-Colores S, Jiménez HM, Ortiz-Echeverri CJ, Flores G (2020) Desmoking laparoscopy surgery images using an image-to-image translation guided by an embedded dark channel. IEEE Access 8:208898–208909
Vishal V, Sharma N, Singh M (2019) Guided unsupervised desmoking of laparoscopic images using cycle-desmoke. OR 2.0 context-aware operating theaters and machine learning in clinical neuroimaging. Springer, New York, pp 21–28
Venkatesh V, Sharma N, Srivastava V, Singh M (2020) Unsupervised smoke to desmoked laparoscopic surgery images using contrast driven cyclic-desmokegan. Comput Biol Med 123:103873
Huang Y, Chen X, Xu L, Li K (2021) Single image desmoking via attentive generative adversarial network for smoke detection process. Fire Technol 57(6):3021–3040
Wu H, Qu Y, Lin S, Zhou J, Qiao R, Zhang Z, Xie Y, Ma L (2021) Contrastive learning for compact single image dehazing. In: Proc. IEEE conf. comput. vis. pattern recogn., pp 10551–10560
Chen X, Fan Z, Li P, Dai L, Kong C, Zheng Z, Huang Y, Li Y (2022) Unpaired deep image dehazing using contrastive disentanglement learning. In: European conference on computer vision. Springer, pp 632–648
Lin T-Y, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: Proc. IEEE conf. comput. vis. pattern recogn., pp 2117–2125
Kirillov A, Girshick R, He K, Dollar P (2019) Panoptic feature pyramid networks. In: Proc. IEEE conf. comput. vis. pattern recogn., pp 6399–6408
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
Twinanda AP, Shehata S, Mutter D, Marescaux J, De Mathelin M, Padoy N (2016) Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans Med Imaging 36(1):86–97
Leibetseder A, Primus MJ, Petscharnig S, Schoeffmann K (2017) Real-time image-based smoke detection in endoscopic videos. In: Proc. themat. workshops ACM multimed., pp 296–304
Hore A, Ziou D (2010) Image quality metrics: Psnr vs. ssim. In: Int. conf. pattern recognit., pp 2366–2369
Shao Y, Li L, Ren W, Gao C, Sang N (2020) Domain adaptation for image dehazing. In: Proc. IEEE conf. comput. vis. pattern recog., pp 2808–2817
Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Proc. int. conf. med. image comp. comput.-assisted intervention, pp 234–241
Chaurasia A, Culurciello E (2017) Linknet: exploiting encoder representations for efficient semantic segmentation. In: Proc. IEEE vis. commun. image process., pp 1–4
Cheng K, You J, Wu S, Chen Z, Zhou Z, Guan J, Peng B, Wang X (2021) Artificial intelligence-based automated laparoscopic cholecystectomy surgical phase recognition and analysis. Surg Endosc 36(5):3160–3168
Jin Y, Long Y, Chen C, Zhao Z, Dou Q, Heng P-A (2021) Temporal memory relation network for workflow recognition from surgical video. IEEE Trans Med Imaging 40(7):1911–1923
Kondo S (2021) Lapformer: surgical tool detection in laparoscopic surgical video using transformer architecture. Computer Methods Biomech Biomed Eng Imaging Vis 9(3):302–307
Yi F, Jiang T (2021) Not end-to-end: Explore multi-stage architecture for online surgical phase recognition. arXiv preprint. arXiv:2107.04810
Loukas C (2018) Surgical phase recognition of short video shots based on temporal modeling of deep features. arXiv preprint. arXiv:1807.07853
Yang Y, Zhao Z, Shi P, Hu S (2021) An efficient one-stage detector for real-time surgical tools detection in robot-assisted surgery. Annual conference on medical image understanding and analysis. Springer, Berlin, pp 18–29
Gao X, Jin Y, Long Y, Dou Q, Heng P-A (2021) Trans-svnet: accurate phase recognition from surgical videos via hybrid embedding aggregation transformer. International conference on medical image computing and computer-assisted intervention. Springer, New York, pp 593–603
Acknowledgements
This work presented in this paper is partially supported by grants from National Natural Science Foundation of China (No. 61772225), Guangdong Basic and Applied Basic Research Foundation (No. 2020A1515010558 and No. 2021A1515011972).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Su, X., Wu, Q. Multi-stages de-smoking model based on CycleGAN for surgical de-smoking. Int. J. Mach. Learn. & Cyber. 14, 3965–3978 (2023). https://doi.org/10.1007/s13042-023-01875-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13042-023-01875-w