Abstract
Purpose
Most recently transformer models became the state of the art in various medical image segmentation tasks and challenges, outperforming most of the conventional deep learning approaches. Picking up on that trend, this study aims at applying various transformer models to the highly challenging task of colorectal cancer (CRC) segmentation in CT imaging and assessing how they hold up to the current state-of-the-art convolutional neural network (CNN), the nnUnet. Furthermore, we wanted to investigate the impact of the network size on the resulting accuracies, since transformer models tend to be significantly larger than conventional network architectures.
Methods
For this purpose, six different transformer models, with specific architectural advancements and network sizes were implemented alongside the aforementioned nnUnet and were applied to the CRC segmentation task of the medical segmentation decathlon.
Results
The best results were achieved with the Swin-UNETR, D-Former, and VT-Unet, each transformer models, with a Dice similarity coefficient (DSC) of 0.60, 0.59 and 0.59, respectively. Therefore, the current state-of-the-art CNN, the nnUnet could be outperformed by transformer architectures regarding this task. Furthermore, a comparison with the inter-observer variability (IOV) of approx. 0.64 DSC indicates almost expert-level accuracy. The comparatively low IOV emphasizes the complexity and challenge of CRC segmentation, as well as indicating limitations regarding the achievable segmentation accuracy.
Conclusion
As a result of this study, transformer models underline their current upward trend in producing state-of-the-art results also for the challenging task of CRC segmentation. However, with ever smaller advances in total accuracies, as demonstrated in this study by the on par performances of multiple network variants, other advantages like efficiency, low computation demands, or ease of adaption to new tasks become more and more relevant.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, Bray F (2021) Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: Cancer J Clin 71(3):209–249
Yao L, Xia Y, Zhang H, Yao J, Jin D, Qiu B, Zhang Y, Li S, Liang Y, Hua XS, Lu L, Chen X, Liu Z, Zhang L (2022) DeepCRC: colorectum and colorectal cancer segmentation in CT scans via deep colorectal coordinate transform. In: International conference on medical image computing and computer-assisted intervention. Springer Nature, Switzerland, pp 564–573
Mainenti PP, Stanzione A, Guarino S, Romeo V, Ugga L, Romano F, Storto G, Maurea S, Brunetti A (2019) Colorectal cancer: Parametric evaluation of morphological, functional and molecular tomographic imaging. World J Gastroenterol 25(35):5233
Ahmad HM, Khan MJ, Yousaf A, Ghuffar S, Khurshid K (2020) Deep learning: a breakthrough in medical imaging. Curr Med Imaging 16(8):946–956
Antonelli M, Reinke A, Bakas S, Farahani K, Kopp-Schneider A, Landman BA, Litjens G, Menze B, Ronneberger O, Summers RM, van Ginneken B, Bilello M, Bilic P, Christ PF, Do RKG, Gollub MJ, Heckers SH, Huisman H, Jarnagin WR, McHugo MK et al (2022) The medical segmentation decathlon. Nat Commun 13(1):4128
Jia X, Mai X, Cui Y, Yuan Y, Xing X, Seo H, Xing L, Meng MQH (2020) Automatic polyp recognition in colonoscopy images using deep learning and two-stage pyramidal feature prediction. IEEE Trans Autom Sci Eng 17(3):1570–1584
Duc NT, Oanh NT, Thuy NT, Triet TM, Dinh VS (2022) Colonformer: an efficient transformer-based method for colon polyp segmentation. IEEE Access 10:80575–80586
Sui D, Zhang K, Liu W, Chen J, Ma X, Tian Z (2021) CST: A multitask learning framework for colorectal cancer region mining based on transformer. BioMed Res Int 2021:6207964
Soomro MH, De Cola G, Conforto S, Schmid M, Giunta G, Guidi E, Neri E, Caruso D, Ciolina M, Laghi A (2018) Automatic segmentation of colorectal cancer in 3D MRI by combining deep learning and 3D level-set algorithm—a preliminary study. In: IEEE middle east conference on biomedical engineering (MECBME), pp 198–203
Pei Y, Mu L, Fu Y, He K, Li H, Guo S, Liu X, Li M, Zhang H, Li X (2020) Colorectal tumor segmentation of CT scans based on a convolutional neural network with an attention mechanism. IEEE Access 8:64131–64138
Tang Y, Yang D, Li W, Roth HR, Landman B, Xu D, Nath V, Hatamizadeh A (2022) Self-supervised pre-training of swin transformers for 3d medical image analysis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 20730–20740
Zhang R, Bai Z, Yu R, Pang W, Wang L, Zhu L, Zhang X, Zhang H, Hu W (2023) AG-CRC: anatomy-guided colorectal cancer segmentation in CT with imperfect anatomical knowledge. arXiv preprint arXiv:2310.04677
Liu X, Guo S, Zhang H, He K, Mu S, Guo Y, Li X (2019) Accurate colorectal tumor segmentation for CT scans based on the label assignment generative adversarial network. Med Phys 46(8):3532–3542
Isensee F, Jaeger PF, Kohl SA, Petersen J, Maier-Hein KH (2021) nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods 18(2):203–211
Hatamizadeh A, Tang Y, Nath V, Yang D, Myronenko A, Landman B, Roth HR, Xu D (2022) UNETR: Transformers for 3D medical image segmentation. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 574–584
Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, 2015, proceedings, Part III 18. Springer International Publishing, pp 234–241
Hatamizadeh A, Nath V, Tang Y, Yang D, Roth HR, Xu D (2021) Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images. International MICCAI Brainlesion Workshop. Springer International Publishing, Cham, pp 272–284
Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 10012–10022
Peiris H, Hayat M, Chen Z, Egan G, Harandi M (2022) A robust volumetric transformer for accurate 3D tumor segmentation. In: international conference on medical image computing and computer-assisted intervention. Springer Nature, Switzerland, pp 162–172
Zhou HY, Guo J, Zhang Y, Han X, Yu L, Wang L, Yu Y (2023) nnformer: Volumetric medical image segmentation via a 3D transformer. In: IEEE transactions on image processing
Chen J, Lu Y, Yu Q, Luo X, Adeli E, Wang Y, Zhou Y (2021) Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306
Chen J, Mei J, Li X, Lu Y, Yu Q, Wei Q, Zhou Y (2023) 3D transunet: advancing medical image segmentation through vision transformers. arXiv preprint arXiv:2310.07781
Wu Y, Liao K, Chen J, Wang J, Chen DZ, Gao H, Wu J (2023) D-former: a U-shaped dilated transformer for 3d medical image segmentation. Neural Comput Appl 35(2):1931–1944
Tummala P, Hille G, Saalfeld S (2023) Automatic lung nodule segmentation in CT imaging using an improved 3D-Res2Unet. BVM Workshop. Springer Fachmedien Wiesbaden, Wiesbaden, pp 165–170
Hille G, Agrawal S, Tummala P, Wybranski C, Pech M, Surov A, Saalfeld S (2023) Joint liver and hepatic lesion segmentation in MRI using a hybrid CNN with transformer layers. Comput Methods Programs Biomed 240:107647
Funding
This work is funded by the Federal Ministry of Education and Research within the research campus STIMULATE under the grant number ‘13GW0473A’.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. For this type of study, formal consent is not required.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Hille, G., Tummala, P., Spitz, L. et al. Transformers for colorectal cancer segmentation in CT imaging. Int J CARS 19, 2079–2087 (2024). https://doi.org/10.1007/s11548-024-03217-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-024-03217-9