Skip to main content
SpringerLink
Log in

SVF-Net: spatial and visual feature enhancement network for brain structure segmentation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Brain structure segmentation in Magnetic Resonance Images (MRI) is essential to the assessment and treatment of medical disorders, especially neuropsychiatric diseases. The key to semantic segmentation is to understand the low-level visual semantics and the high-level spatial semantics of the image. Due to the complex anatomical structures, the current approaches lack the ability to effectively extract rich semantic information, resulting in the inevitable loss of details in prediction results. To address this problem, we propose a novel spatial and visual feature enhancement network (SVF-Net) to accurately segment the brain structures. The SVF-Net is designed as a multi-task learning framework, in which an auxiliary coarse segmentation task is used for spatial information acquisition, an auxiliary image reconstruction task is used for visual information preservation, and a major refined segmentation task is used for brain structure segmentation. In this algorithm, multitasking optimization is mainly based on two strategies: firstly, a spatial feature enhancement (SFE) module is introduced to extract the location and spatial relationships of objects from the coarse prediction, which are then sent to the refined segmentation model for spatial information enhancement. Secondly, a visual feature preservation (VFP) model is introduced for image reconstruction, which shares the feature extractor with the refined segmentation model, so as to retain more useful low-level visual features for the model. Extensive experiments are performed on three public brain MRI T1 scan datasets (the IBSR dataset, the MALC dataset and the LPBA dataset) to evaluate the effectiveness of the proposed algorithm. The experimental results show that the SVF-Net achieves the best performance compared with the state-of-the-art methods. In addition, the ablation experiments and the noise interference experiments demonstrate that proposed SFE and VFP module have obvious advantages in improving segmentation accuracy and resisting noise interference.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Bernal J, Kushibar K, Cabezas M, Valverde S, Oliver A, Lladó X (2019) Quantitative analysis of patch-based fully convolutional neural networks for tissue segmentation on brain magnetic resonance imaging. IEEE Access 7:89986–90002. https://doi.org/10.1109/ACCESS.2019.2926697

    Article  Google Scholar 

  2. Petrella JR, Edward Coleman R, Murali Doraiswamy P (2003) Neuroimaging and early diagnosis of alzheimer disease: a look to the future. Radiology 226(2):315–336. https://doi.org/10.1148/radiol.2262011600

    Article  Google Scholar 

  3. Patenaude B, Smith SM, Kennedy DN, Jenkinson M (2011) A bayesian model of shape and appearance for subcortical brain segmentation. Neuroimage 56(3):907–922. https://doi.org/10.1016/j.neuroimage.2011.02.046

    Article  Google Scholar 

  4. Wu G, Kim M, Sanroma G, Wang Q, Munsell BC, Shen D, Alzheimer’s Disease Neuroimaging Initiative et al (2015) Hierarchical multi-atlas label fusion with multi-scale feature representation and label-specific patch partition. NeuroImage 106:34–46. https://doi.org/10.1016/j.neuroimage.2014.11.025

    Article  Google Scholar 

  5. van Opbroek A, van der Lijn F, de Bruijne M (2013) Automated brain-tissue segmentation by multi-feature svm classification. The MIDAS Journal. http://hdl.handle.net/10380/3443

  6. Moeskops P, Benders Manon JNL, Chiţǎ SM, Kersbergen KJ, Groenendaal F, de Vries LS, Viergever MA, Išgum I (2015) Automatic segmentation of mr brain images of preterm infants using supervised classification. NeuroImage 118:628–641. https://doi.org/10.1016/j.neuroimage.2015.06.007

    Article  Google Scholar 

  7. Prakash RM, Selva Kumari RS (2018) Modified expectation maximization method for automatic segmentation of mr brain images. http://hdl.handle.net/10380/3445

  8. Bengio Y (2009) Learning deep architectures for AI Now Publishers Inc. https://doi.org/10.1561/2200000006

  9. Rajchl M, Baxter JS, Jonathan McLeod A, Yuan J, Qiu W u, Peters TM, White JA, Khan AR (2013) Asets: Map-based brain tissue segmentation using manifold learning and hierarchical max-flow regularization. In: Proceedings of the MICCAI Grand Challenge on MR Brain Image Segmentation (MRBrainS’13). https://doi.org/10.1016/j.patrec.2017.11.016, p 375

  10. Pereira S, Pinto A, Oliveira J, M Mendrik A, Correia JH, Silva CA (2016) Automatic brain tissue segmentation in mr images using random forests and conditional random fields. Journal of Neuroscience Methods 270:111–123. https://doi.org/10.1016/j.jneumeth.2016.06.017

    Article  Google Scholar 

  11. Li W, Gao Y, Shi F, Li G, Gilmore JH, Lin W, Shen D (2015) Links: Learning-based multi-source integration framework for segmentation of infant brain images. NeuroImage 108:160–172. https://doi.org/10.1016/j.neuroimage.2014.12.042

    Article  Google Scholar 

  12. Antipov G, Berrani S-A, Ruchaud N, Dugelay J-L (2015) Learned vs. hand-crafted features for pedestrian gender recognition. In: Proceedings of the 23rd ACM international conference on multimedia, pp 1263–1266

  13. Li X, Wei Y, Wang L, Fu S, Wang C (2021) Msgse-net: Multi-scale guided squeeze-and-excitation network for subcortical brain structure segmentation. Neurocomputing 461:228–243. https://doi.org/10.1016/j.neucom.2021.07.018

    Article  Google Scholar 

  14. Chen H, Qi D, Yu L, Qin J, Heng P-A (2018) Voxresnet: Deep voxelwise residual networks for brain segmentation from 3d mr images. NeuroImage 170:446–455. https://doi.org/10.1016/j.neuroimage.2017.04.041

    Article  Google Scholar 

  15. Roy AG, Conjeti S, Sheet D, Katouzian A, Navab N, Wachinger C (2017) Error corrective boosting for learning fully convolutional networks with limited data. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 231–239. https://doi.org/10.1007/978-3-319-66179-7-27

  16. Dolz J, Gopinath K, Yuan J, Lombaert H, Desrosiers C, Ayed IB (2018) Hyperdense-net: a hyper-densely connected cnn for multi-modal image segmentation. IEEE Transactions on Medical Imaging 38(5):1116–1126. https://doi.org/10.1109/TMI.2018.2878669

    Article  Google Scholar 

  17. Kushibar K, Valverde S, González-Villà S, Bernal J, Cabezas M, Oliver A, Lladó X (2018) Automated sub-cortical brain structure segmentation combining spatial and deep convolutional features. Medical Image Analysis 48:177–186. https://doi.org/10.1016/j.media.2018.06.006

    Article  Google Scholar 

  18. Roy AG, Conjeti S, Navab N, Wachinger C, Alzheimer’s Disease Neuroimaging Initiative et al (2019) Quicknat: a fully convolutional network for quick and accurate segmentation of neuroanatomy. NeuroImage 186:713–727. https://doi.org/10.1016/j.neuroimage.2018.11.042

    Article  Google Scholar 

  19. Wachinger C, Reuter M, Klein T (2018) Deepnat: Deep convolutional neural network for segmenting neuroanatomy. NeuroImage 170:434–445. https://doi.org/10.1016/j.neuroimage.2017.02.035

    Article  Google Scholar 

  20. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241. https://doi.org/10.1007/978-3-319-24574-4-28

  21. Xia H, Ma M, Li H, Song S (2021) Mc-net: multi-scale context-attention network for medical ct image segmentation. Appl Intell, 1–12. https://doi.org/10.1007/s10489-021-02506-z

  22. Wang Z, Peng Y, Li D, Guo Y, Zhang B (2021) Mmnet: A multi-scale deep learning network for the left ventricular segmentation of cardiac mri images. Applied Intelligence. https://doi.org/10.1007/s10489-021-02720-9

  23. Shin H-C, Roth HR, Gao M, Lu L, Xu Z, Nogues I, Yao J, Mollura D, Summers RM (2016) Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning. IEEE Transactions on Medical Imaging 35(5):1285–1298

    Article  Google Scholar 

  24. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 580–587

  25. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  26. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25:1097–1105

    Google Scholar 

  27. Shakeri M, Tsogkas S, Ferrante E, Lippe S, Kadoury S, Paragios N, Kokkinos I (2016) Sub-cortical brain structure segmentation using f-cnn’s. In: 2016 IEEE 13Th international symposium on biomedical imaging (ISBI). IEEE, pp 269–272. https://doi.org/10.1109/ISBI.2016.7493261

  28. Roy AG, Nav Ab N, Wachinger C (2018) Concurrent spatial and channel squeeze & excitation in fully convolutional networks. Springer, Cham. https://doi.org/10.1007/978-3-030-00928-1_48

    Book  Google Scholar 

  29. Milletari F, Ahmadi S-A, Kroll C, Plate A, Rozanski V, Maiostre J, Levin J, Dietrich O, Ertl-Wagner B, Bötzel K. et al (2017) Hough-cnn: deep learning for segmentation of deep brain regions in mri and ultrasound. Comput Vis Image Underst 164:92–102. https://doi.org/10.1016/j.cviu.2017.04.002

    Article  Google Scholar 

  30. Roy AG, Conjeti S, Navab N, Wachinger C, Alzheimer’s Disease Neuroimaging Initiative et al (2019) Bayesian quicknat: model uncertainty in deep whole-brain segmentation for structure-wise quality control. NeuroImage 195:11–22. https://doi.org/10.1016/j.neuroimage.2019.03.042

    Article  Google Scholar 

  31. Zhou Z, Siddiquee MRM, Tajbakhsh N, Liang J (2018) Unet++: A nested u-net architecture for medical image segmentation. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, pp 3–11. https://doi.org/10.1007/978-3-030-00889-5-1

  32. Huang H, Lin L, Tong R, Hu H, Wu J (2020) Unet 3+: A full-scale connected unet for medical image segmentation. https://doi.org/10.1109/ICASSP40776.2020.9053405

  33. Oktay O, Schlemper J, Le Folgoc L, Lee M, Heinrich M, Misawa K, Mori K, McDonagh S, Hammerla NY, Kainz B et al (2018) Attention u-net: Learning where to look for the pancreas. arXiv:1804.03999

  34. Gu Z, Cheng J, Fu H, Zhou K, Hao H, Zhao Y, Zhang T, Gao S, Liu J (2019) Ce-net: Context encoder network for 2d medical image segmentation. IEEE Trans Med Imaging 38(10):2281–2292. https://doi.org/10.1109/TMI.2019.2903562

    Article  Google Scholar 

  35. Wang J, Sun K, Cheng T, Jiang B, Xiao B (2021) Deep high-resolution representation learning for visual recognition. IEEE Trans Pattern Anal Mach Intell 43(10):3349–3364. https://doi.org/10.1109/TPAMI.2020.2983686

    Article  Google Scholar 

  36. Liu X, He P, Chen W, Gao J (2019) Multi-task deep neural networks for natural language understanding. arXiv:1901.11504

  37. Kendall A, Gal Y, Cipolla R (2018) Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7482–7491. https://doi.org/10.1109/CVPR.2018.00781

  38. Kingma DP, Ba J Adam:, A method for stochastic optimization. arXiv:1412.6980

  39. Worth AJ (1996) The internet brain segmentation repository (ibsr). http://www.cma.mgh.harvard.edu/ibsr

  40. Landman BA, Warfield S (2012) Miccai: 2012 Grand challenge and workshop on multi-atlas labeling. In: Proc. international conference on medical image computing and computer assisted intervention, MICCAI, vol 2012. http://www.incf.org/community/events/miccai

  41. Shattuck DW, Mirza M, Adisetiyo V, Hojatkashani C, Salamon G, Narr KL, Poldrack RA, Bilder RM, Toga AW (2008) Construction of a 3d probabilistic atlas of human cortical structures. NeuroImage 39(3):1064–1080. https://doi.org/10.1016/j.neuroimage.2007.09.031

    Article  Google Scholar 

  42. Marcus DS, Wang TH, Parker J, Csernansky JG, Morris JC, Buckner RL (2007) Open access series of imaging studies (oasis): cross-sectional mri data in young, middle aged, nondemented, and demented older adults. Journal of Cognitive Neuroscience 19(9):1498–1507. https://doi.org/10.1162/jocn.2007.19.9.1498

    Article  Google Scholar 

  43. Yang H, Sun J, Li H, Wang L, Xu Z (2018) Neural multi-atlas label fusion: Application to cardiac mr images. Medical Image Analysis 49:60–75. https://doi.org/10.1016/j.media.2018.07.009

    Article  Google Scholar 

  44. Zhang C, Lin G, Liu F, Yao R, Shen C (2019) Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). arXiv:1903.02351v1

Download references

Acknowledgements

This work is supported by National Nature Science Foundation of China (grant No.61871106 and No.61370152), Key R & D projects of Liaoning Province, China (grant No. 2020JH2/10100029), and the Open Project Program Foundation of the Key Laboratory of Opto-Electronics Information Processing, Chinese Academy of Sciences (OEIP-O-202002).

Author information

Authors and Affiliations

  1. College of Information Science, Northeastern University, Shenyang, 110179, China

    Qian Hu, Ying Wei, Xiang Li, Chuyuan Wang & Shanze Wang

  2. Information Technology R, D Innovation Center of Peking University, Shaoxing, China

    Ying Wei

  3. College of Robot Science, Northeastern University, Shenyang, 110004, China

    Huan Wang

Authors
  1. Qian Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ying Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chuyuan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Huan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shanze Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ying Wei.

Ethics declarations

Conflict of Interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, Q., Wei, Y., Li, X. et al. SVF-Net: spatial and visual feature enhancement network for brain structure segmentation. Appl Intell 53, 4180–4200 (2023). https://doi.org/10.1007/s10489-022-03706-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03706-x

Keywords

Search

Navigation