Abstract
Semantic segmentation of laparoscopic images is an important issue for intraoperative guidance in laparoscopic surgery. However, acquiring and annotating laparoscopic datasets is labor-intensive, which limits the research on this topic. In this paper, we tackle the Domain-Adaptive Semantic Segmentation (DASS) task, which aims to train a segmentation network using only computer-generated simulated images and unlabeled real images. To bridge the large domain gap between generated and real images, we propose a Masked Frequency Consistency (MFC) module that encourages the network to learn frequency-related information of the target domain as additional cues for robust recognition. Specifically, MFC randomly masks some high-frequency information of the image to improve the consistency of the network’s predictions for low-frequency images and real images. We conduct extensive experiments on existing DASS frameworks with our MFC module and show performance improvements. Our approach achieves comparable results to fully supervised learning method on the CholecSeg8K dataset without using any manual annotation. The code is available at github.com/MoriLabNU/MFC.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Aklilu, J., Yeung, S.: ALGES: active learning with gradient embeddings for semantic segmentation of laparoscopic surgical images. In: Proceedings of Machine Learning for Healthcare, pp. 892–911. PMLR (2022)
Araslanov, N., Roth, S.: Self-supervised augmentation consistency for adapting semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15384–15394. IEEE (2021)
Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. 40(04), 834–848 (2018)
Contributors, M.: MMSegmentation: openmmlab semantic segmentation toolbox and benchmark (2020). https://github.com/open-mmlab/mmsegmentation
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000–16009. IEEE (2022)
Hong, W.Y., Kao, C.L., Kuo, Y.H., Wang, J.R., Chang, W.L., Shih, C.S.: CholecSeg8k: a semantic segmentation dataset for laparoscopic cholecystectomy based on Cholec80. arXiv preprint arXiv:2012.12453 (2020)
Hoyer, L., Dai, D., Van Gool, L.: DaFormer: improving network architectures and training strategies for domain-adaptive semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9924–9935. IEEE (2022)
Hoyer, L., Dai, D., Van Gool, L.: HRDA: context-aware high-resolution domain-adaptive semantic segmentation. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13690, pp. 372–391. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-20056-4_22
Hoyer, L., Dai, D., Wang, H., Van Gool, L.: MIC: masked image consistency for context-enhanced domain adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11721–11732. IEEE (2023)
Hu, S., Liao, Z., Xia, Y.: Domain specific convolution and high frequency reconstruction based unsupervised domain adaptation for medical image segmentation. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) MICCAI 2022. LNCS, vol. 13437, pp. 650–659. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16449-1_62
Liu, J., Guo, X., Yuan, Y.: Prototypical interaction graph for unsupervised domain adaptation in surgical instrument segmentation. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12903, pp. 272–281. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87199-4_26
Liu, Q., Chen, C., Qin, J., Dou, Q., Heng, P.A.: FedDG: federated domain generalization on medical image segmentation via episodic learning in continuous frequency space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1013–1023. IEEE (2021)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
Lyu, J., Zhang, Y., Huang, Y., Lin, L., Cheng, P., Tang, X.: AADG: automatic augmentation for domain generalization on retinal image segmentation. IEEE Trans. Med. Imaging 41(12), 3699–3711 (2022)
Madani, A., et al.: Artificial intelligence for intraoperative guidance: using semantic segmentation to identify surgical anatomy during laparoscopic cholecystectomy. Ann. Surg. 276(2), 363–369 (2022)
Nussbaumer, H.J., Nussbaumer, H.J.: The fast fourier transform. Fast Fourier Transform and Convolution Algorithms, pp. 80–111 (1982)
Pfeiffer, M., et al.: Generating large labeled data sets for laparoscopic image processing tasks using unpaired image-to-image translation. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11768, pp. 119–127. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32254-0_14
Qiu, J., Hayashi, Y., Oda, M., Kitasaka, T., Mori, K.: Class-wise confidence-aware active learning for laparoscopic images segmentation. Inter. J. Comput. Assisted Radiol. Surgery, 1–10 (2022)
Sahu, M., Mukhopadhyay, A., Zachow, S.: Simulation-to-real domain adaptation with teacher-student learning for endoscopic instrument segmentation. Int. J. Comput. Assist. Radiol. Surg. 16(5), 849–859 (2021)
Sahu, M., Strömsdörfer, R., Mukhopadhyay, A., Zachow, S.: Endo-Sim2Real: consistency learning-based domain adaptation for instrument segmentation. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12263, pp. 784–794. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59716-0_75
Silva, B., et al.: Analysis of current deep learning networks for semantic segmentation of anatomical structures in laparoscopic surgery. In: 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 3502–3505. IEEE (2022)
Tarvainen, A., Valpola, H.: Mean teachers are better role models: weight-averaged consistency targets improve semi-supervised deep learning results. In: Advances in Neural Information Processing Systems 30 (2017)
Tranheden, W., Olsson, V., Pinto, J., Svensson, L.: DACS: domain adaptation via cross-domain mixed sampling. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1379–1389 (2021)
Twinanda, A.P., Shehata, S., Mutter, D., Marescaux, J., De Mathelin, M., Padoy, N.: Endonet: a deep architecture for recognition tasks on laparoscopic videos. IEEE Trans. Med. Imaging 36(1), 86–97 (2016)
Way, L.W., et al.: Causes and prevention of laparoscopic bile duct injuries: analysis of 252 cases from a human factors and cognitive psychology perspective. Ann. Surg. 237(4), 460–469 (2003)
Xie, E., Wang, W., Yu, Z., Anandkumar, A., Alvarez, J.M., Luo, P.: SegFormer: simple and efficient design for semantic segmentation with transformers. Adv. Neural. Inf. Process. Syst. 34, 12077–12090 (2021)
Yang, Y., Soatto, S.: FDA: fourier domain adaptation for semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4085–4095. IEEE (2020)
Zakazov, I., Shaposhnikov, V., Bespalov, I., Dylov, D.V.: Feather-light fourier domain adaptation in magnetic resonance imaging. In: Kamnitsas, K., et al. (eds.) DART 2022. LNCS, vol. 13542, pp. 88–97. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16852-9_9
Zhou, Q., et al.: Context-aware mixup for domain adaptive semantic segmentation. IEEE Trans. Circuits Syst. Video Technol. 33, 804–817 (2021)
Acknowledgments
This work was supported in part by the JSPS KAKENHI Grant Numbers 17H00867, 21K19898, 26108006; in part by the JST CREST Grant Number JPMJCR20D5; and in part by the fellowship of the Nagoya University TMI WISE program from MEXT.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhao, X., Hayashi, Y., Oda, M., Kitasaka, T., Mori, K. (2023). Masked Frequency Consistency for Domain-Adaptive Semantic Segmentation of Laparoscopic Images. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14220. Springer, Cham. https://doi.org/10.1007/978-3-031-43907-0_63
Download citation
DOI: https://doi.org/10.1007/978-3-031-43907-0_63
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43906-3
Online ISBN: 978-3-031-43907-0
eBook Packages: Computer ScienceComputer Science (R0)
