Abstract
The presence of surgical smoke in laparoscopic surgery reduces the visibility of the operative field. In order to ensure better visualization, the present paper proposes an unsupervised deep learning approach for the task of desmoking of the laparoscopic images. This network builds upon generative adversarial networks (GANs) and converts laparoscopic images from smoke domain to smoke-free domain. The network comprises a new generator architecture that has an encoder-decoder structure composed of multi-scale feature extraction (MSFE) blocks at each encoder block. The MSFE blocks of the generator capture features at multiple scales to obtain a robust deep representation map and help to reduce the smoke component in the image. Further, a structure-consistency loss has been introduced to preserve the structure in the desmoked images. The proposed network is called Multi-scale DesmokeNet, which has been evaluated on the laparoscopic images obtain from Cholec80dataset. The quantitative and qualitative results shows the efficacy of the proposed Multi-scale DesmokeNet in comparison with other state-of-the-art desmoking methods.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Baid, A., Kotwal, A., Bhalodia, R., Merchant, S., Awate, S.P.: Joint desmoking, specularity removal, and denoising of laparoscopy images via graphical models and Bayesian inference. In: 2017 IEEE 14th International Symposium on Biomedical Imaging, ISBI 2017, pp. 732–736. IEEE (2017)
Barrett, W.L., Garber, S.M.: Surgical smoke: a review of the literature. Surg. Endosc. 17(6), 979–987 (2003)
Berman, D., Avidan, S., et al.: Non-local image dehazing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1674–1682 (2016)
Bolkar, S., Wang, C., Cheikh, F.A., Yildirim, S.: Deep smoke removal from minimally invasive surgery videos. In: 2018 25th IEEE International Conference on Image Processing (ICIP), pp. 3403–3407. IEEE (2018)
Cai, B., Xu, X., Jia, K., Qing, C., Tao, D.: DehazeNet: an end-to-end system for single image haze removal. IEEE Trans. Image Process. 25(11), 5187–5198 (2016)
Cai, Z., Fan, Q., Feris, R.S., Vasconcelos, N.: A unified multi-scale deep convolutional neural network for fast object detection. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 354–370. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_22
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 833–851. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_49
Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing systems, pp. 2672–2680 (2014)
He, K., Sun, J., Tang, X.: Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2341–2353 (2010)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)
Karacan, L., Akata, Z., Erdem, A., Erdem, E.: Learning to generate images of outdoor scenes from attributes and semantic layouts. arXiv preprint arXiv:1612.00215 (2016)
Kim, T., Cha, M., Kim, H., Lee, J.K., Kim, J.: Learning to discover cross-domain relations with generative adversarial networks. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1857–1865. JMLR. org (2017)
Kotwal, A., Bhalodia, R., Awate, S.P.: Joint desmoking and denoising of laparoscopy images. In: 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp. 1050–1054. IEEE (2016)
Lawrentschuk, N., Fleshner, N.E., Bolton, D.M.: Laparoscopic lens fogging: a review of etiology and methods to maintain a clear visual field. J. Endourol. 24(6), 905–913 (2010)
Leibetseder, A., Primus, M.J., Petscharnig, S., Schoeffmann, K.: Real-time image-based smoke detection in endoscopic videos. In: Proceedings of the on Thematic Workshops of ACM Multimedia 2017, pp. 296–304. ACM (2017)
Lim, B., Son, S., Kim, H., Nah, S., Mu Lee, K.: Enhanced deep residual networks for single image super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 136–144 (2017)
Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: Advances in Neural Information Processing Systems, pp. 700–708 (2017)
Luo, X., McLeod, A.J., Pautler, S.E., Schlachta, C.M., Peters, T.M.: Vision-based surgical field defogging. IEEE Trans. Med. Imaging 36(10), 2021–2030 (2017)
Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2794–2802 (2017)
Mittal, A., Moorthy, A.K., Bovik, A.C.: No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process. 21(12), 4695–4708 (2012)
Sangkloy, P., Lu, J., Fang, C., Yu, F., Hays, J.: Scribbler: controlling deep image synthesis with sketch and color. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5400–5409 (2017)
Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1874–1883 (2016)
Takahashi, H., et al.: Automatic smoke evacuation in laparoscopic surgery: a simplified method for objective evaluation. Surg. Endosc. 27(8), 2980–2987 (2013)
Tchaka, K., Pawar, V.M., Stoyanov, D.: Chromaticity based smoke removal in endoscopic images. In: Medical Imaging 2017: Image Processing, vol. 10133, p. 101331M. International Society for Optics and Photonics (2017)
Twinanda, A.P., Shehata, S., Mutter, D., Marescaux, J., De Mathelin, M., Padoy, N.: EndoNet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans. Med. Imaging 36(1), 86–97 (2016)
Vishal, V., Sharma, N., Singh, M.: Guided unsupervised desmoking of laparoscopic images using cycle-desmoke. In: Zhou, L., et al. (eds.) OR 2.0/MLCN -2019. LNCS, vol. 11796, pp. 21–28. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32695-1_3
Wang, C., Cheikh, F.A., Kaaniche, M., Beghdadi, A., Elle, O.J.: Variational based smoke removal in laparoscopic images. Biomed. Eng. Online 17(1), 139 (2018)
Wang, C., Mohammed, A.K., Cheikh, F.A., Beghdadi, A., Elle, O.J.: Multiscale deep desmoking for laparoscopic surgery. In: Medical Imaging 2019: Image Processing, vol. 10949, p. 109491Y. International Society for Optics and Photonics (2019)
Yan, J., Li, J., Fu, X.: No-reference quality assessment of contrast-distorted images using contrast enhancement. arXiv preprint arXiv:1904.08879 (2019)
Yi, Z., Zhang, H., Tan, P., Gong, M.: DualGAN: unsupervised dual learning for image-to-image translation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2849–2857 (2017)
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Vishal, V., Venkatesh, V., Lochan, K., Sharma, N., Singh, M. (2020). Unsupervised Desmoking of Laparoscopy Images Using Multi-scale DesmokeNet. In: Blanc-Talon, J., Delmas, P., Philips, W., Popescu, D., Scheunders, P. (eds) Advanced Concepts for Intelligent Vision Systems. ACIVS 2020. Lecture Notes in Computer Science(), vol 12002. Springer, Cham. https://doi.org/10.1007/978-3-030-40605-9_36
Download citation
DOI: https://doi.org/10.1007/978-3-030-40605-9_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-40604-2
Online ISBN: 978-3-030-40605-9
eBook Packages: Computer ScienceComputer Science (R0)