Skip to main content
SpringerLink
Log in

Hybrid transformer-CNN with boundary-awareness network for 3D medical image segmentation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

3D volumetric medical image segmentation is a crucial task in computer-aided diagnosis applications, but it remains challenging due to low contrast and boundary ambiguity between organs and surrounding tissues. Considering that accurate boundary voxels are of importance for organ segmentation, which relies on rich detailed features information. The most recent convolutional neural networks and transformer networks have attempted to enhance 2D boundaries during feature extraction. Few approaches focus on boundary voxels preservation for 3D scenarios. To address these issues, we propose the Hybrid Transformer-CNN with Boundary-awareness(HTCB-Net) network, which follows an encoder-decoder segmentation paradigm with learnable boundary modules. The 3D swin-transformer encoder is embedded with auxiliary object-related boundary map by designing a learnable boundary extracting module(BEM), which assists model in obtaining rich and discriminative feature representations. Our boundary map obtained from BEM supervises explicitly feature extraction process. Subsequently, boundary preserving module(BPM) adopts a novel fusion strategy, which integrates extracted boundary map and the corresponding encoder features. Through this module, the boundary position awareness is combined for feature enhancement with spatial complement and channel attention. We evaluate the performance of the proposed method with quantitative experiments on three public available datasets in both CT and MRI modalities: OAI-ZIB, Spleen, Pancreas. The comparative experimental results demonstrate that our HTCB-Net preserves more precise 3D boundaries and obtains significant improvements, particularly in terms of Average Symmetric Surface Distance(ASSD).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Zhou S, Nie D, Adeli E, Yin J, Lian J, Shen D (2019) High-resolution encoder-decoder networks for low-contrast medical image segmentation. IEEE Trans Image Process. 29:461–475

    Article  MathSciNet  MATH  Google Scholar 

  2. Hu H, Shen L, Guan Q, Li X, Zhou Q, Ruan S (2022) Deep co-supervision and attention fusion strategy for automatic covid-19 lung infection segmentation on ct images. Pattern Recog. 124:108452

    Article  Google Scholar 

  3. Hsu W-Y, Lu C-C, Hsu Y-Y (2020) Improving segmentation accuracy of ct kidney cancer images using adaptive active contour model. Medicine 99

  4. Sedghi Gamechi Z, Bons LR, Giordano M, Bos D, Budde RP, Kofoed KF, Pedersen JH, Roos-Hesselink JW, Bruijne M (2019) Automated 3d segmentation and diameter measurement of the thoracic aorta on non-contrast enhanced ct. Eur Radiol. 29:4613–4623

    Article  Google Scholar 

  5. Myller KAH, Honkanen JTJ, Jurvelin JS, Saarakkala S, Töyräs J, Väänänen SP (2018) Method for segmentation of knee articular cartilages based on contrast-enhanced ct images. Ann Biomed Eng 46:1756–1767

    Article  Google Scholar 

  6. Schlemper J, Oktay O, Schaap M, Heinrich MP, Kainz B, Glocker B, Rueckert D (2018) Attention gated networks: Learning to leverage salient regions in medical images. Med Image Anal 53:197–207

    Article  Google Scholar 

  7. Li X, Chen H, Qi X, Dou Q, Fu C-W, Heng P-A (2018) H-denseunet: Hybrid densely connected unet for liver and tumor segmentation from ct volumes. IEEE Trans Med Imaging. 37:2663–2674

    Article  Google Scholar 

  8. Zhou C, Ding C, Wang X, Lu Z, Tao D (2019) One-pass multi-task networks with cross-task guided attention for brain tumor segmentation. IEEE Trans Image Process. 29:4516–4529

    Article  MATH  Google Scholar 

  9. Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J (2019) Unet++: Redesigning skip connections to exploit multiscale features in image segmentation. IEEE Trans Med Imaging. 39:1856–1867

    Article  Google Scholar 

  10. Xia H, Ma M, Li H, Song S (2021) Mc-net: multi-scale context-attention network for medical ct image segmentation. Appl Intell 52:1508–1519

    Article  Google Scholar 

  11. Hu Q, Wei Y, Li XL, Wang C, Wang H, Wang S (2022) Svf-net: spatial and visual feature enhancement network for brain structure segmentation. Appl Intell 53:4180–4200

    Article  Google Scholar 

  12. Hu H, Zhang Z, Xie Z, Lin S (2019) Local relation networks for image recognition. IEEE/CVF International Conference on Computer Vision (ICCV) 2019:3463–3472

    Article  Google Scholar 

  13. Xue C, Zhu L, Fu H, Hu X, Li X, Zhang H, Heng P-A (2021) Global guidance network for breast lesion segmentation in ultrasound images. Med Image Anal 70:101989

    Article  Google Scholar 

  14. Lin X, Yu L, Cheng K-T, Yan Z (2022) The lighter the better: Rethinking transformers in medical image segmentation through adaptive pruning. IEEE transactions on medical imaging, PP

    Google Scholar 

  15. Shi C, Xu C, He J, Chen Y, Cheng Y, Yang Q, Qi H (2022) Graph-based convolution feature aggregation for retinal vessel segmentation. Simul Model Pract Theory 121:102653

    Article  Google Scholar 

  16. Wu Y, Liao K-Y, Chen J, Wang J, Chen DZ, Gao H, Wu J (2022) D-former: a u-shaped dilated transformer for 3d medical image segmentation. Neural Comput Appl 35:1931–1944

    Article  Google Scholar 

  17. Hatamizadeh A, Yang D, Roth HR, Xu D (2021) Unetr: Transformers for 3d medical image segmentation. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2022:1748–1758

    Google Scholar 

  18. Lin A-J, Chen B, Xu J, Zhang Z, Lu G, Zhang D (2021) Ds-transunet: Dual swin transformer u-net for medical image segmentation. IEEE Trans Instrum Meas 71:1–15

    Google Scholar 

  19. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: Hierarchical vision transformer using shifted windows. IEEE/CVF International Conference on Computer Vision (ICCV) 2021:9992–10002

    Article  Google Scholar 

  20. Yuan F, Zhang Z, Fang Z (2022) An effective cnn and transformer complementary network for medical image segmentation. Pattern Recognit 136:109228

    Article  Google Scholar 

  21. Hatamizadeh A, Nath V, Tang Y, Yang D, Roth HR, Xu D (2022) Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In: BrainLes@MICCAI

  22. Heidari M, Kazerouni A, Kadarvish MS, Azad R, Aghdam EK, Cohen-Adad J, Merhof D (2022) Hiformer: Hierarchical multi-scale representations using transformers for medical image segmentation. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2023:6191–6201

    Google Scholar 

  23. Lin J, Lin J, Lu C, Chen H, Lin H, Zhao B, Shi Z, Qiu B, Pan X, Xu Z, Huang B, Liang C, Han G, Liu Z, Han C (2022) Ckd-transbts: Clinical knowledge-driven hybrid transformer with modality-correlated cross-attention for brain tumor segmentation. IEEE transactions on medical imaging, PP

    Google Scholar 

  24. Zhu Q, Du B, Yan P (2019) Boundary-weighted domain adaptive neural network for prostate mr image segmentation. IEEE Trans Med Imaging 39:753–763

  25. Wang R, Chen S, Ji C, Fan J, Li Y (2022) Boundary-aware context neural network for medical image segmentation. Medical image analysis. 78:102395

  26. Sun Y, Wang S, Chen C, Xiang T (2022) Boundary-guided camouflaged object detection. In: IJCAI

  27. Yang J, Jiao L, Shang R, Liu X, Li R, Xu L (2023) Ept-net: Edge perception transformer for 3d medical image segmentation. IEEE transactions on medical imaging, PP

    Google Scholar 

  28. Lee HJ, Kim JU, Lee S, Kim HG, Ro YM (2020) Structure boundary preserving segmentation for medical image with ambiguous boundary. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2020:4816–4825

    Google Scholar 

  29. Gu R, Wang G, Song T, Huang R, Aertsen M, Deprest JA, Ourselin S, Vercauteren T, Zhang S (2020) Ca-net: Comprehensive attention convolutional neural networks for explainable medical image segmentation. IEEE Trans Med Imaging 40:699–711

    Article  Google Scholar 

  30. Isensee F, Jaeger PF, Kohl SA, Petersen J, Maier-Hein KH (2021) nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods 18(2):203–211

    Article  Google Scholar 

  31. Gu Z, Cheng J, Fu H, Zhou K, Hao H, Zhao Y, Zhang T, Gao S, Liu J (2019) Ce-net: Context encoder network for 2d medical image segmentation. IEEE Trans Med Imaging 38:2281–2292

    Article  Google Scholar 

  32. Feng S, Zhao H, Shi F, Cheng X, Wang M, Ma Y, Xiang D, Zhu W, Chen X (2020) Cpfnet: Context pyramid fusion network for medical image segmentation. IEEE Trans Med Imaging 39(10):3008–3018

    Article  Google Scholar 

  33. He Q, Sun X, Diao W, Yan Z, Yin D, Fu K (2022) Transformer-induced graph reasoning for multimodal semantic segmentation in remote sensing. ISPRS J Photogramm Remote Sens

  34. Tang Y, Yang D, Li W, Roth HR, Landman BA, Xu D, Nath V, Hatamizadeh A (2022) Self-supervised pre-training of swin transformers for 3d medical image analysis. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2022:20698–20708

    Google Scholar 

  35. Kervadec H, Bouchtiba J, Desrosiers C, Granger É, Dolz J, Ayed IB (2019) Boundary loss for highly unbalanced segmentation. Med Image Anal 67:101851

    Article  Google Scholar 

  36. Karimi D, Salcudean SE (2020) Reducing the hausdorff distance in medical image segmentation with convolutional neural networks. IEEE Trans Med Imaging 39:499–513

    Article  Google Scholar 

  37. Ma C, Xu Q, Wang X, Jin B, Zhang X, Wang Y, Zhang Y (2020) Boundary-aware supervoxel-level iteratively refined interactive 3d image segmentation with multi-agent reinforcement learning. IEEE Trans Med Imaging 40:2563–2574

    Article  Google Scholar 

  38. Hu J, Shen L, Albanie S, Sun G, Wu E (2020) Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell 42:2011–2023

    Article  Google Scholar 

  39. Ambellan F, Tack A, Ehlke M, Zachow S (2019) Automated segmentation of knee bone and cartilage combining statistical shape knowledge and convolutional neural networks: Data from the osteoarthritis initiative. Med Image Anal 52:109–118

    Article  Google Scholar 

  40. Antonelli M, Reinke A, Bakas S, Farahani K, Kopp-Schneider Annette, Landman BA, Litjens GJS, Menze BH, Ronneberger O, Summers RM, Ginneken B, Bilello M, Bilic P, Christ PF, Do RKG, Gollub MJ, Heckers S, Huisman HJ, Jarnagin WR, McHugo M, Napel S, Pernicka JSG, Rhode KS, Tobon-Gomez C, Vorontsov E, Meakin JA, Ourselin S, Wiesenfarth M, Arbeláez P, Bae B, Chen S, Daza LA, Feng J-J, He B, Isensee F, Ji Y, Jia F, Kim N, Kim I, Merhof D, Pai A, Park B, Perslev M, Rezaiifar R, Rippel O, Sarasua I, Shen W, Son J, Wachinger C, Wang L, Wang Y, Xia Y, Xu D, Xu Z, Zheng Y, Simpson AL, Maier-Hein L, Cardoso MJ (2022) The medical segmentation decathlon. Nat Commun 13:4128

  41. Dawant BM, Li R, Lennon B, Li S (2007) Semi-automatic segmentation of the liver and its evaluation on the miccai 2007 grand challenge data set

  42. Huang Q, Sun J, Ding H, Wang X, Wang G (2018) Robust liver vessel extraction using 3d u-net with variant dice loss function. Comput Biol Med 101:153–162

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant No. 61806107 and 61702135, Shandong Key Laboratory of Wisdom Mine Information Technology, and the Opening Project of State Key Laboratory of Digital Publishing Technology.

Author information

Authors and Affiliations

  1. School of Information Science and Technology, Qingdao University of Science and Technology, 266000, ShanDong, Qingdao, China

    Jianfei He & Canhui Xu

Authors
  1. Jianfei He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Canhui Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Canhui Xu.

Ethics declarations

Competing Interests

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

He, J., Xu, C. Hybrid transformer-CNN with boundary-awareness network for 3D medical image segmentation. Appl Intell 53, 28542–28554 (2023). https://doi.org/10.1007/s10489-023-05032-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-05032-2

Keywords

Search

Navigation