Skip to main content
SpringerLink
Log in

Boundary-guided feature integration network with hierarchical transformer for medical image segmentation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

A variety of convolutional neural network (CNN) based methods for medical image segmentation have achieved outstanding performance, however, inherently suffered from a limited ability to capture long-range dependencies. In addition, some of the features integrated by complex strategy may be used repeatedly, resulting in more redundant information. To solve this problem, we propose a novel feature integration network by conducting hierarchical transformer under both explicit label and boundary guidance for accurate medical image segmentation. First, the encoder hierarchically extracts multiscale features by each hybrid attention module consisting of residual and Transformer blocks. Second, the parallel features from adjacent layers are integrated via cross fusion block to complement semantics to low-level features. Then, both deep boundary and label supervision are deployed for layer-wise decoders. Finally, the output features are fused by each layer instead of a top-to-down way to integrate multiscale semantic representation for prediction. The proposed method was evaluated on Synapse dataset with Dice of 81.63% and Promise12 dataset with Dice of 92.47%, 95HD of 2.06mm, and aRVD of 4.09%. Extensive experiments demonstrate that our proposed method achieves promising performance on multi-organ CT and prostate MR image segmentation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3.
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

The raw data can be shared if the researchers need to do research on relevant topic and cite our paper.

References

  1. Cao H, Wang Y, Chen J, Jiang D, Zhang X, Tian Q, Wang M (2022) Swin-unet: unet-like pure transformer for medical image segmentation. In: European conference on computer vision. pp 205–218

  2. Carion N, Massa F, Synnaeve G, Usunier N, Kirillov A, Zagoruyko S (2020) End-to-end object detection with transformers. In: European conference on computer vision. pp 213–229

  3. Chen J, Wang X, Guo Z, Zhang X, Sun J (2021) Dynamic region-aware convolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 8060–8069

  4. Chen J, Lu Y, Yu Q, Luo X, Adeli E, Wang Y, Lu L, Yuille A, Zhou Y (2021) Transunet: transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306. pp 1–13

  5. Chen L, Bentley P, Mori K, Misawa K, Fujiwara M, Rueckert D (2018) DRINet for medical image segmentation. IEEE Trans Med Imaging 37(11):2453–2462

    Article  Google Scholar 

  6. Çiçek Ö, Abdulkadir A, Lienkamp S S, Brox T, Ronneberger O (2016) 3D U-net: learning dense volumetric segmentation from sparse annotation. In: Medical image computing and computer assisted intervention–MICCAI 2016. pp 424–432

  7. Dosovitskiy A, Beyer L, Kolesnikov A et al (2020) An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. pp 1–21

  8. Fang X, Yan P (2020) Multi-organ segmentation over partially labeled datasets with multi-scale feature abstraction. IEEE Trans Med Imaging 39(11):3619–3629

    Article  Google Scholar 

  9. Gao Y, Zhou M, Metaxas DN (2021) UTNet: a hybrid transformer architecture for medical image segmentation. In: Medical image computing and computer assisted intervention–MICCAI 2021. pp 61–71

  10. Nooshin G et al (2019) Automatic segmentation of prostate MRI using convolutional neural networks: investigating the impact of network architecture on the accuracy of volume measurement and MRI-ultrasound registration. Med Image Anal 58:101558

    Article  Google Scholar 

  11. Gridach M, Voiculescu I (2021) Dopnet: densely oriented pooling network for medical image segmentation. In: 2021 IEEE 18th international symposium on biomedical imaging (ISBI). pp 1714–1717

  12. Gu Z, Cheng J, Fu H et al (2019) CE-net: context encoder network for 2d medical image segmentation. IEEE Trans Med Imaging 38(10):2281–2292

    Article  Google Scholar 

  13. Ran G et al (2020) CA-net: comprehensive attention convolutional neural networks for explainable medical image segmentation. IEEE Trans Med Imaging 40(2):699–711

    Google Scholar 

  14. Guo X, Liu J, Yuan Y (2022) Semantic-oriented labeled-to-unlabeled distribution translation for image segmentation. IEEE Trans Med Imaging 2(2022):41

    Google Scholar 

  15. Hatamizadeh A, Tang Y, Nath V et al (2022) UNETR: transformers for 3d medical image segmentation. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp 1748–1758

  16. Huang H, Lin L, Tong R et al. (2020) UNet 3+: a full-scale connected unet for medical image segmentation. In: ICASSP 2020-2020 IEEE international conference on acoustics, speech and signal processing. pp 1055–1059

  17. Fabian I et al (2021) nnU-net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods 18(2):203–211

    Article  Google Scholar 

  18. Jia H, Xia Y, Cai W, Fulham M, Feng DD (2017) Prostate segmentation in MR images using ensemble deep convolutional neural networks. In: 2017 IEEE 14th international symposium on biomedical imaging (ISBI 2017). pp 762–765

  19. Landman B, Xu Z, Igelsias J, Styner M, Langerak T, Klein A (2015) MICCAI multi-atlas labeling beyond the cranial vault–workshop and challenge. In: MICCAI multi-atlas labeling beyond cranial vault—workshop challenge, vol 5. p 12

  20. Lin A, Chen B, Xu J, Zhang Z, Lu G, Zhang D (2022) Ds-transunet: dual swin transformer u-net for medical image segmentation. IEEE Trans Instrum Meas 71:1–15

    Google Scholar 

  21. Geert L et al (2014) Evaluation of prostate segmentation algorithms for MRI: the PROMISE12 challenge. Med Image Anal 18(2):359–373

    Article  Google Scholar 

  22. Litjens G et al (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88

    Article  Google Scholar 

  23. Liu Z, Lin Y, Cao Y et al (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision. pp 9992–10002

  24. Meyer A, Chlebus G, Rak M et al (2021) Anisotropic 3D multi-stream CNN for accurate prostate segmentation from multi-planar MRI. Comput Methods Prog Biomed 200:105821

    Article  Google Scholar 

  25. Milletari F, Navab N, Ahmadi SA (2016) V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 fourth international conference on 3D vision (3DV). pp 565–571

  26. Oktay O, Schlemper J, Folgoc L L et al (2018) Attention u-net: learning where to look for the pancreas. arXiv preprint arXiv:1804.03999. pp 1–10

  27. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Medical image computing and computer-assisted intervention–MICCAI 2015. pp 234–241

  28. Tong T et al (2015) Discriminative dictionary learning for abdominal multi-organ segmentation. Med Image Anal 23(1):92–104

    Article  Google Scholar 

  29. Valanarasu JMJ, Oza P, Hacihaliloglu I, Patel VM (2021) Medical transformer: gated axial-attention for medical image segmentation. In: Medical image computing and computer assisted intervention–MICCAI 2021. pp 36–46

  30. Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. Adv Neural Inf Proces Syst 30:6000–6010

    Google Scholar 

  31. Wang W, Chen C, Ding M, Yu H, Zha S, Li J (2021) Transbts: multimodal brain tumor segmentation using transformer. In: Medical image computing and computer assisted intervention–MICCAI 2021. pp 109–119

  32. Wang X, Girshick R, Gupta A, He K (2018) Non-local neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). pp 7794–7803

  33. Wang H, Cao P, Wang J, Zaiane O R (2022) Uctransnet: rethinking the skip connections in u-net from a channel-wise perspective with transformer. In: Proceedings of the AAAI conference on artificial intelligence, vol 36, no 3. pp 2441–2449

  34. Bo W et al (2019) Deeply supervised 3D fully convolutional networks with group dilated convolution for automatic MRI prostate segmentation. Med Phys 46(4):1707–1718

    Article  Google Scholar 

  35. Wu W, Qian C, Yang S, Wang Q, Cai Y, Zhou Q (2018) Look at boundary: a boundary-aware face alignment algorithm. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 2129–2138

  36. Xie Y, Zhang J, Shen C, Xia Y (2021) Cotr: efficiently bridging cnn and transformer for 3d medical image segmentation. In: Medical image computing and computer assisted intervention–MICCAI 2021. pp 171–180

  37. Biting Y et al (2019) Ea-GANs: edge-aware generative adversarial networks for cross-modality MR image synthesis. IEEE Trans Med Imaging 38(7):1750–1762

    Article  Google Scholar 

  38. Zhang Q, Yang YB (2021) Rest: an efficient transformer for visual recognition. Adv Neural Inf Proces Syst 34:15475–15485

    Google Scholar 

  39. Zhang Y, Liu H, Hu Q (2021) Transfuse: fusing transformers and cnns for medical image segmentation. In: Medical image computing and computer assisted intervention–MICCAI 2021. pp 14–24

  40. Zhang S, Fu H, Yan Y et al (2019) Attention guided network for retinal image segmentation. In: Medical image computing and computer assisted intervention–MICCAI 2019. pp 797–805

  41. Zhao JX, Liu JJ, Fan DP, Cao Y, Yang J, Cheng MM (2019) EGNet: edge guidance network for salient object detection. In: Proceedings of the IEEE/CVF international conference on computer vision. pp 8778–8787

  42. Zhao Y, Li H, Wan S, Sekuboyina A, Hu X, Tetteh G, Piraud M, Menze B (2019) Knowledge-aided convolutional neural network for small organ segmentation. IEEE J Biomed Health Inform 23(4):1363–1373

    Article  Google Scholar 

  43. Zhou Z, Rahman Siddiquee MM, Tajbakhsh N, Liang J (2018) Unet++: a nested u-net architecture for medical image segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support, ML-CDS 2018. pp 3–11

  44. Zhu Q, Du B, Yan P (2020) Boundary-weighted domain adaptive neural network for prostate MR image segmentation. IEEE Trans Med Imaging 39(3):753–763

    Article  Google Scholar 

Download references

Funding

This work was supported by the National Natural Science Foundation of China (No. 62041108); the Natural Science Foundation of Ningxia (No. 2020AAC03029); Innovation and Entrepreneurship Project for Returnees in Ningxia 2020.

Author information

Author notes

  1. Fan Wang and Bo Wang contributed equally to this work.

Authors and Affiliations

  1. School of Physics and Electronic-Electrical Engineering, Ningxia University, Yinchuan, 750021, China

    Fan Wang & Bo Wang

Authors
  1. Fan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bo Wang.

Ethics declarations

Conflicts of interest

No conflict of interest exits in the submission of this manuscript and the manuscript is approved by all authors for publication.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, F., Wang, B. Boundary-guided feature integration network with hierarchical transformer for medical image segmentation. Multimed Tools Appl 83, 8955–8969 (2024). https://doi.org/10.1007/s11042-023-15948-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15948-z

Keywords

Search

Navigation