Abstract
Existing thyroid nodule segmentation methods are primarily developed based on ultrasound images, which generally neglects the clinical reports that include rich semantic information for nodules. However, current text guided segmentation methods for natural images are not applicable to the image-report thyroid nodule dataset, due to the many-to-one correspondence between images and reports in current data. To this end, we propose a clinical report guided thyroid nodule segmentation framework with Adversarial Keyword Extraction (AKE) module to extract keywords from reports and Semantic-Spatial Feature Aggregation (SSFA) module to integrate reports into the segmentation model. To alleviate the many-to-one correspondence issue, we devise the AKE module to highlight the keywords about current ultrasound images from clinical reports with a keywords mask, which adopts adversarial learning to encourage the mask generator to mask out the useful descriptions to boost segmentation performance. We further propose the SSFA module to effectively and efficiently map semantic information from reports to each pixel of spatial features, so as to emphasize the target regions. Moreover, we manually collect a clinical Reports Assisted Thyroid Nodule segmentation dataset (RATN), which includes the ultrasound images, the pixel-wise nodule segmentation annotation, and the clinical reports. Extensive experiments have been conducted on the RATN dataset, and the results prove the effectiveness and computational efficiency of the proposed method over the existing methods. Code and data are available at https://github.com/cvi-szu.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Acuña-Ruiz, A., Carrasco-López, C., Santisteban, P.: Genomic and epigenomic profile of thyroid cancer. Best Pract. Res. Clin. Endocrinol. Metab. 37(1), 101656 (2023)
Chen, J., et al.: TransUNet: transformers make strong encoders for medical image segmentation. ArXiv (2021)
Fan, T., Wang, G., Li, Y., Wang, H.: MA-Net: a multi-scale attention network for liver and tumor segmentation. IEEE Access 8, 179656–179665 (2020)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)
Jang, E., Gu, S., Poole, B.: Categorical reparameterization with Gumbel-Softmax. ArXiv (2016)
Jin, Z., Li, X., Zhang, Y., Shen, L., Lai, Z., Kong, H.: Boundary regression-based reep neural network for thyroid nodule segmentation in ultrasound images. Neural. Comput. Appl. 34, 1–10 (2022)
Jing, Y., Kong, T., Wang, W., Wang, L., Li, L., Tan, T.: Locate then segment: a strong pipeline for referring image segmentation. In: CVPR, pp. 9858–9867 (2021)
Kaur, J., Jindal, A.: Comparison of thyroid segmentation algorithms in ultrasound and scintigraphy images. Int. J. Comput. Appl. 50(23), 24–27 (2012)
Kollorz, E.N., Hahn, D.A., Linke, R., Goecke, T.W., Hornegger, J., Kuwert, T.: Quantification of thyroid volume using 3-D ultrasound imaging. IEEE Trans. Med. Imaging 27(4), 457–466 (2008)
Li, Z., Zhou, S., Chang, C., Wang, Y., Guo, Y.: A weakly supervised deep active contour model for nodule segmentation in thyroid ultrasound images. Pattern Recognit. Lett. 165, 128–137 (2023)
Li, Z., et al.: LViT: language meets vision transformer in medical image segmentation. ArXiv (2022)
Ma, J., Wu, F., Jiang, T., Zhao, Q., Kong, D.: Ultrasound image-based thyroid nodule automatic segmentation using convolutional neural networks. Int. J. Comput. Assist. Radiol. Surg. 12, 1895–1910 (2017)
Monajatipoor, M., Rouhsedaghat, M., Li, L.H., Jay Kuo, C.C., Chien, A., Chang, K.W.: BERTHop: an effective vision-and-language model for chest X-ray disease diagnosis. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) MICCAI 2022. LNCS, vol. 13435, pp. 725–734. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16443-9_69
Mylona, E.A., Savelonas, M.A., Maroulis, D.: Automated adjustment of region-based active contour parameters using local image geometry. IEEE Trans. Cybern. 44(12), 2757–2770 (2014)
Pan, H., Zhou, Q., Latecki, L.J.: SGUNet: semantic guided UNet for thyroid nodule segmentation. In: ISBI, pp. 630–634 (2021)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Savelonas, M.A., Iakovidis, D.K., Legakis, I., Maroulis, D.: Active contours guided by echogenicity and texture for delineation of thyroid nodules in ultrasound images. IEEE Trans. Inf. Technol. Biomed. 13(4), 519–527 (2008)
Tang, Z., Ma, J.: Coarse to fine ensemble network for thyroid nodule segmentation. In: Shusharina, N., Heinrich, M.P., Huang, R. (eds.) MICCAI 2020. LNCS, vol. 12587, pp. 122–128. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-71827-5_16
Tomar, N.K., Jha, D., Bagci, U., Ali, S.: TGANet: text-guided attention for improved polyp segmentation. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) MICCAI 2022. LNCS, vol. 13433, pp. 151–160. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16437-8_15
Turc, I., Chang, M.W., Lee, K., Toutanova, K.: Well-read students learn better: the impact of student initialization on knowledge distillation. ArXiv 13 (2019)
Valanarasu, J.M.J., Patel, V.M.: UNeXt: MLP-based rapid medical image segmentation network. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) MICCAI 2022. LNCS, vol. 13435, pp. 23–33. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16443-9_3
Vaswani, A., et al.: Attention is all you need. NIPS 30, 5998–6008 (2017)
Wang, J., et al.: Deep high-resolution representation learning for visual recognition. IEEE TPAMI 43(10), 3349–3364 (2020)
Yang, Z., Wang, J., Tang, Y., Chen, K., Zhao, H., Torr, P.H.: LAVT: language-aware vision transformer for referring image segmentation. In: CVPR, pp. 18155–18165 (2022)
Zhang, Y., Lai, H., Yang, W.: Cascade UNet and CH-UNet for thyroid nodule segmentation and benign and malignant classification. In: Shusharina, N., Heinrich, M.P., Huang, R. (eds.) MICCAI 2020. LNCS, vol. 12587, pp. 129–134. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-71827-5_17
Acknowledgement
This work was supported in part by the Natural Science Foundation of China under Grant 82261138629 and 62272319; and Guangdong Basic and Applied Basic Research Foundation under Grant 2023A1515010688, 2021A1515220072 and 2023A1515010677; and Shenzhen Municipal Science and Technology Innovation Council under Grant JCYJ20220531101412030, JCYJ20220530155811025, and JCYJ20220818095803007.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zhang, Y., Chen, W., Li, X., Shen, L., Lai, Z., Kong, H. (2024). Adversarial Keyword Extraction and Semantic-Spatial Feature Aggregation for Clinical Report Guided Thyroid Nodule Segmentation. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14437. Springer, Singapore. https://doi.org/10.1007/978-981-99-8558-6_20
Download citation
DOI: https://doi.org/10.1007/978-981-99-8558-6_20
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8557-9
Online ISBN: 978-981-99-8558-6
eBook Packages: Computer ScienceComputer Science (R0)