C-Net: Cascaded convolutional neural network with global guidance and refinement residuals for breast ultrasound images segmentation

doi:10.1016/j.cmpb.2022.107086

Computer Methods and Programs in Biomedicine

Volume 225, October 2022, 107086

https://doi.org/10.1016/j.cmpb.2022.107086 Get rights and content

Highlights

•
First, we developed a novel cascaded convolutional neural network to segment the lesion from breast ultrasound images.
•
Second, a bidirectional attention guidance network was designed to capture the context between global (low-level) and local (high-level) features.
•
Third, we introduced a refinement residual network to obtain the more complete lesion mask.
•
Moreover, the experimental results demonstrate that our method achieves the best overall performance on breast and renal ultrasound images segmentation.

Abstract

Background and objective

Breast lesions segmentation is an important step of computer-aided diagnosis system. However, speckle noise, heterogeneous structure, and similar intensity distributions bring challenges for breast lesion segmentation.

Methods

In this paper, we presented a novel cascaded convolutional neural network integrating U-net, bidirectional attention guidance network (BAGNet) and refinement residual network (RFNet) for the lesion segmentation in breast ultrasound images. Specifically, we first use U-net to generate a set of saliency maps containing low-level and high-level image structures. Then, the bidirectional attention guidance network is used to capture the context between global (low-level) and local (high-level) features from the saliency map. The introduction of the global feature map can reduce the interference of surrounding tissue on the lesion regions. Furthermore, we developed a refinement residual network based on the core architecture of U-net to learn the difference between rough saliency feature maps and ground-truth masks. The learning of residuals can assist us to obtain a more complete lesion mask.

Results

To evaluate the segmentation performance of the network, we compared with several state-of-the-art segmentation methods on the public breast ultrasound dataset (BUSIS) using six commonly used evaluation metrics. Our method achieves the highest scores on six metrics. Furthermore, p-values indicate significant differences between our method and the comparative methods.

Conclusions

Experimental results show that our method achieves the most competitive segmentation results. In addition, we apply the network on renal ultrasound images segmentation. In general, our method has good adaptability and robustness on ultrasound image segmentation.

Introduction

Breast cancer is a terrible disease that seriously threatens the health of women around the world. Due to the characteristics of asymptomatic, concealed, and multiple pathogenic factors in the early stage of breast cancer, regular screening is essential for the prevention and diagnosis of breast cancer. Breast ultrasound (BUS) imaging is a widely used screening method due to its advantages of painless, high sensitivity, noninvasive, and low cost [1]. BUS image segmentation can help characterize tissues and improve diagnosis, and it is an important part of BUS computer-aided diagnosis (CAD) systems [2,3]. However, the disturbance of various factors (such as heterostructure, blurred boundaries and variable shapes) makes accurate BUS image segmentation a challenging task, see Fig. 1 for more details.

Many segmentation methods have been proposed to segment the breast lesion from ultrasound images [4]. These segmentation algorithms can be classified into three types: manual, semi-automatic and automatic according to the degree of human intervention in the segmentation process [5]. Previously, the contours of breast lesions were mainly manually annotated by radiologists. Manually segmenting lesions from ultrasound images is time-consuming and error-prone [6]. To alleviate the challenge of manual segmentation, many semi-automatic methods have been developed to segment BUS images [5]. These semi-automatic segmentation methods not only reduce the variance of manual segmentation of breast lesions, but also further improve work efficiency. However, Yin et al. [7] pointed out that many semi-automatic methods rely on hand-crafted features. Therefore, it is very meaningful to segment the lesion automatically and reliably from BUS images. Recently, convolutional neural networks (CNNs) have been widely used in medical image segmentation [8], [9], [10], [11]. Among them, FCNN [12] and U-net [8] are typical CNN models, which have received extensive attention in BUS images segmentation based on their core architectures [13], [14], [15]. However, the surrounding tissue similar to the lesion affected the segmentation performance of the network, as shown in Fig. 1. To better utilize the contextual information of images, many methods introduce dilated convolution and pooling operations to obtain larger receptive fields [16], [17], [18]. However, these operations cannot capture contextual information from the global view and only consider dependencies on the spatial domain. Recently, capturing long-range and global dependency information has been shown to improve segmentation accuracy [19,20]. Although these methods improve the segmentation accuracy of lesion regions, learning limited long-range and global features in deeper convolutional layers will affect the performance of the segmentation network [21]. How to fully exploit the global and long-range dependencies to improve the segmentation accuracy of medical images remains a challenging task.

To address the above challenges, a novel cascaded convolutional neural network integrating U-net, bidirectional attention guidance network (BAGNet) and refinement residual network (RFNet) is developed to segment breast lesions, as shown in Fig. 2. The designed BAGNet is used to capture the context between global (low-level) and local (high-level) features. The developed RFNet is used to learn the difference between rough saliency feature maps and ground-truth masks. Residual learning of RFNet can assist us to obtain a more complete lesion mask. Our main contributions are as follows:

First, we developed a novel bidirectional attention-guided network (BAGNet) and used it to construct a cascaded convolutional neural network (denote as: C-Net) to segment breast lesions from the ultrasound image.

Second, we evaluated the segmentation network on the public BUS dataset using six evaluation metrics, and the experimental result demonstrate that our method outperforms state-of-the-art segmentation methods on BUS images segmentation.

Furthermore, our method is applied to renal ultrasound images segmentation and achieves great competitiveness compared with state-of-the-art segmentation methods.

Section snippets

Traditional methods for BUS image segmentation

Conventional methods for BUS image segmentation mainly include region-based methods, graph-based approaches and deformable models [2,5]. Xian et al. [22] proposed a fully automatic BUS images segmentation method for performing accurate lesion segmentation. Xiao et al. [23] developed a method to segment ultrasound image tissue by combining maximum a posteriori (MAP) and Markov random field (MRF) and applied it on BUS images. The segmentation results of this convex optimization method on breast

Materials and method

In this section, we first describe the segmentation network architecture of breast lesions. Then, the design details of the three key components of the network are presented. Finally, the loss function and experimental parameters for network training is introduced.

Ablation study

In this section, we show the effectiveness of the principal components of our network, i.e., bidirectional attention guidance network (BAGNet) and refinement residual network (RFNet). The ablation experiments are mainly conducted on the public BUSIS dataset.

The baseline network is U-net (i.e., the first row of Table 1), which includes four down-sampling operations and four up-sampling operations. Table 1 illustrates the results of the architecture ablation study. As we can see, our method

Discussion

The generalization ability of the network is further validated on our collected renal ultrasound dataset. The dataset contains 300 clinical renal ultrasound images from 300 patients in the Fourth Medical Center of the PLA General Hospital and the Civil Aviation General Hospital. These images are from two ultrasound devices (Philips and Esaote), with 150 images for each type of device. The pixel size of all images ranging from 0.12 to 0.32 mm. The renal labels used for network training and

Conclusion

To improve the segmentation accuracy of breast lesions, this paper presented a novel cascaded convolutional neural network (C-Net) to segment lesions from BUS images. The network mainly composed of U-net, bidirectional attention guidance network (BAGNet) and refinement residual network (RFNet). The design of the bidirectional attention guidance network captures the context between global (low-level) and local (high-level) features from the saliency map, which can reduce the interference of

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (Grant Nos.: U1913207, 51875394).

Reference (40)

Y. Luo et al.
Segmentation information with attention integration for classification of breast tumor in ultrasound image
Pattern Recognit.
(2022)
J. Bai et al.
Applying deep learning in digital breast tomosynthesis for automatic breast cancer detection: a review
Med. Image Anal.
(2021)
M. Xian et al.
Automatic breast ultrasound image segmentation: a survey
Pattern Recognit.
(2018)
S. Liu et al.
Deep learning in medical ultrasound analysis: a review
Engineering
(2019)
S. Yin et al.
Automatic kidney segmentation in ultrasound images using subsequent boundary distance regression and pixelwise classification networks
Med. Image Anal.
(2020)
C. Xue et al.
Global guidance network for breast lesion segmentation in ultrasound images
Med. Image Anal.
(2021)
M. Xian et al.
Fully automatic segmentation of breast ultrasound images based on breast characteristics in space and frequency domains
Pattern Recognit.
(2015)
J. Shan et al.
Completely automated segmentation approach for breast ultrasound images using multiple-domain features
Ultrasound Med. Biol.
(2012)
B. Liu et al.
Fully automatic and segmentation-robust classification of breast tumors based on local texture analysis of ultrasound images
Pattern Recognit.
(2010)
L. Abdelrahman et al.
Convolutional neural networks for breast cancer detection in mammography: a survey
Comput. Biol. Med.
(2021)

E.H. Houssein et al.

Deep and machine learning techniques for medical imaging-based breast cancer: a comprehensive review

Expert Syst. Appl.

(2021)

B. Lei et al.

Segmentation of breast anatomy for automated whole breast ultrasound images with boundary regularized convolutional encoder–decoder network

Neurocomputing

(2018)

W.K. Moon et al.

Computer-aided diagnosis of breast ultrasound images using ensemble learning from convolutional neural networks

Comput. Methods Progr. Biomed.

(2020)

W. Al-Dhabyani et al.

Dataset of breast ultrasound images

Data Brief

(2020)

G. Chen et al.

A novel convolutional neural network for kidney ultrasound image segmentation

Comput. Methods Progr. Biomed.

(2022)

J.A. Noble et al.

Ultrasound image segmentation: a survey

IEEE Trans. Med. Imaging

(2006)

Q. Huang et al.

Breast ultrasound image segmentation: a survey

Int. J. Comput. Assist. Radiol. Surg.

(2017)

O. Ronneberger et al.

U-net: convolutional networks for biomedical image segmentation

Z. Zhou et al.

UNet++: redesigning skip connections to exploit multiscale features in image segmentation

IEEE Trans. Med. Imaging

(2020)

H. Huang, L. Lin, R. Tong, H. Hu, J.B.T.-I. 2020-2020 I.I.C. on A. Wu Speech and Signal Processing (ICASSP), UNet 3+: a...

Cited by (25)

Active learning for left ventricle segmentation in echocardiography
2024, Computer Methods and Programs in Biomedicine
Training deep learning models for medical image segmentation require large annotated datasets, which can be expensive and time-consuming to create. Active learning is a promising approach to reduce this burden by strategically selecting the most informative samples for segmentation. This study investigates the use of active learning for efficient left ventricle segmentation in echocardiography with sparse expert annotations.
We adapt and evaluate various sampling techniques, demonstrating their effectiveness in judiciously selecting samples for segmentation. Additionally, we introduce a novel strategy, Optimised Representativeness Sampling, which combines feature-based outliers with the most representative samples to enhance annotation efficiency.
Our findings demonstrate a substantial reduction in annotation costs, achieving a remarkable 99% upper bound performance while utilising only 20% of the labelled data. This equates to a reduction of 1680 images needing annotation within our dataset. When applied to a publicly available dataset, our approach yielded a remarkable 70% reduction in required annotation efforts, representing a significant advancement compared to baseline active learning strategies, which achieved only a 50% reduction. Our experiments highlight the nuanced performance of diverse sampling strategies across datasets within the same domain.
The study provides a cost-effective approach to tackle the challenges of limited expert annotations in echocardiography. By introducing a distinct dataset, made publicly available for research purposes, our work contributes to the field's understanding of efficient annotation strategies in medical image segmentation.
ConvMixer-based encoder and classification-based decoder architecture for breast lesion segmentation in ultrasound images
2024, Biomedical Signal Processing and Control
Automatic breast lesion segmentation in ultrasound images is an important research topic, as breast cancer is one of the most common and dangerous cancers. However, lesion segmentation is a difficult task due to the challenges encountered in ultrasound images. In this study, a new encoder-decoder network based on ConvMixer is designed for breast lesion segmentation in ultrasound images. This model, called the ConvMixer-based Encoder-Classification-Based Decoder (CE-CD), divides the pixel-level segmentation task into image-level classification and pixel-level detection, effectively combining them. ConvMixer and DenseNet121 are used in the encoder. While spatial and semantic details are obtained with DenseNet121, long-range-context details are obtained with ConvMixer. Then, these features are combined and transferred to the decoder. In addition, the decoder consists of a classification network and a detection network. The detection network obtains the lesion detection score at the pixel level, while the classification network obtains the lesion classification score at the image level. In the last section of CE-CD, the detected lesion class is determined using the classification output with the result generation algorithm. The BUSI dataset was used to analyze the performance of the CE-CD. As a result of experimental studies, the proposed model provided a superior performance than the state-of-the-art models with a Jaccard score of 69.23% and a Dice score of 80.23%. Furthermore, using ConvMixer together with DenseNet121 in the analyses performed for CE-CD effectively increased the success. On the other hand, although the mutual exclusion problem was encountered, the proposed decoder was found to be effective.
Distilling Knowledge From an Ensemble of Vision Transformers for Improved Classification of Breast Ultrasound
2024, Academic Radiology
To develop a deep learning model for the automated classification of breast ultrasound images as benign or malignant. More specifically, the application of vision transformers, ensemble learning, and knowledge distillation is explored for breast ultrasound classification.
Single view, B-mode ultrasound images were curated from the publicly available Breast Ultrasound Image (BUSI) dataset, which has categorical ground truth labels (benign vs malignant) assigned by radiologists and malignant cases confirmed by biopsy. The performance of vision transformers (ViT) is compared to convolutional neural networks (CNN), followed by a comparison between supervised, self-supervised, and randomly initialized ViT. Subsequently, the ensemble of 10 independently trained ViT, where the ensemble model is the unweighted average of the output of each individual model is compared to the performance of each ViT alone. Finally, we train a single ViT to emulate the ensembled ViT using knowledge distillation.
On this dataset that was trained using five-fold cross validation, ViT outperforms CNN, while self-supervised ViT outperform supervised and randomly initialized ViT. The ensemble model achieves an area under the receiver operating characteristics curve (AuROC) and area under the precision recall curve (AuPRC) of 0.977 and 0.965 on the test set, outperforming the average AuROC and AuPRC of the independently trained ViTs (0.958 ± 0.05 and 0.931 ± 0.016). The distilled ViT achieves an AuROC and AuPRC of 0.972 and 0.960.
Both transfer learning and ensemble learning can each offer increased performance independently and can be sequentially combined to collectively improve the performance of the final model. Furthermore, a single vision transformer can be trained to match the performance of an ensemble of a set of vision transformers using knowledge distillation.
A deep sift convolutional neural networks for total brain volume estimation from 3D ultrasound images
2023, Computer Methods and Programs in Biomedicine
Preterm infants are a highly vulnerable population. The total brain volume (TBV) of these infants can be accurately estimated by brain ultrasound (US) imaging which enables a longitudinal study of early brain growth during Neonatal Intensive Care (NICU) admission. Automatic estimation of TBV from 3D images increases the diagnosis speed and evades the necessity for an expert to manually segment 3D images, which is a sophisticated and time consuming task. We develop a deep-learning approach to estimate TBV from 3D ultrasound images. It benefits from deep convolutional neural networks (CNN) with dilated residual connections and an additional layer, inspired by the fuzzy c-Means (FCM), to further separate the features into different regions, i.e. sift layer. Therefore, we call this method deep-sift convolutional neural networks (DSCNN). The proposed method is validated against three state-of-the-art methods including AlexNet-3D, ResNet-3D, and VGG-3D, for TBV estimation using two datasets acquired from two different ultrasound devices. The results highlight a strong correlation between the predictions and the observed TBV values. The regression activation maps are used to interpret DSCNN, allowing TBV estimation by exploring those pixels that are more consistent and plausible from an anatomical standpoint. Therefore, it can be used for direct estimation of TBV from 3D images without needing further image segmentation.
Recent advancements in artificial intelligence for breast cancer: Image augmentation, segmentation, diagnosis, and prognosis approaches
2023, Seminars in Cancer Biology
Breast cancer is a significant global health burden, with increasing morbidity and mortality worldwide. Early screening and accurate diagnosis are crucial for improving prognosis. Radiographic imaging modalities such as digital mammography (DM), digital breast tomosynthesis (DBT), magnetic resonance imaging (MRI), ultrasound (US), and nuclear medicine techniques, are commonly used for breast cancer assessment. And histopathology (HP) serves as the gold standard for confirming malignancy. Artificial intelligence (AI) technologies show great potential for quantitative representation of medical images to effectively assist in segmentation, diagnosis, and prognosis of breast cancer. In this review, we overview the recent advancements of AI technologies for breast cancer, including 1) improving image quality by data augmentation, 2) fast detection and segmentation of breast lesions and diagnosis of malignancy, 3) biological characterization of the cancer such as staging and subtyping by AI-based classification technologies, 4) prediction of clinical outcomes such as metastasis, treatment response, and survival by integrating multi-omics data. Then, we then summarize large-scale databases available to help train robust, generalizable, and reproducible deep learning models. Furthermore, we conclude the challenges faced by AI in real-world applications, including data curating, model interpretability, and practice regulations. Besides, we expect that clinical implementation of AI will provide important guidance for the patient-tailored management.
A dual-stage transformer and MLP-based network for breast ultrasound image segmentation
2023, Biocybernetics and Biomedical Engineering
Automatic segmentation of breast lesions from ultrasound images plays an important role in computer-aided breast cancer diagnosis. Many deep learning methods based on convolutional neural networks (CNNs) have been proposed for breast ultrasound image segmentation. However, breast ultrasound image segmentation is still challenging due to ambiguous lesion boundaries. We propose a novel dual-stage framework based on Transformer and Multi-layer perceptron (MLP) for the segmentation of breast lesions. We combine the Swin Transformer block with an efficient pyramid squeezed attention block in a parallel design and introduce bi-directional interactions across branches, which can efficiently extract multi-scale long-range dependencies to improve the segmentation performance and robustness of the model. Furthermore, we introduce tokenized MLP block in the MLP stage to extract global contextual information while retaining fine-grained information to segment more complex breast lesions. We have conducted extensive experiments with state-of-the-art methods on three breast ultrasound datasets, including BUSI, BUL, and MT_BUS datasets. The dice coefficient reached 0.8127 ± 0.2178, and the intersection over union reached 0.7269 ± 0.2370 on benign lesions when the Hausdorff distance was maintained at 3.75 ± 1.83. The dice coefficient of malignant lesions is improved by 3.09% for BUSI dataset. The segmentation results on the BUL and MT_BUS datasets also show that our proposed model achieves better segmentation results than other methods. Moreover, the external experiments indicate that the proposed model provides better generalization capability for breast lesion segmentation. The dual-stage scheme and the proposed Transformer module achieve the fine-grained local information and long-range dependencies to relieve the burden of radiologists.

View all citing articles on Scopus

View full text

C-Net: Cascaded convolutional neural network with global guidance and refinement residuals for breast ultrasound images segmentation

Highlights

Abstract

Background and objective

Methods

Results

Conclusions

Introduction

Section snippets

Traditional methods for BUS image segmentation

Materials and method

Ablation study

Discussion

Conclusion

Declaration of Competing Interest

Acknowledgments

Pattern Recognit.

Med. Image Anal.

Pattern Recognit.

Engineering

Med. Image Anal.

Med. Image Anal.

Pattern Recognit.

Ultrasound Med. Biol.

Pattern Recognit.

Comput. Biol. Med.

Expert Syst. Appl.

Neurocomputing

Comput. Methods Progr. Biomed.

Data Brief

Comput. Methods Progr. Biomed.

Ultrasound image segmentation: a survey

IEEE Trans. Med. Imaging

Breast ultrasound image segmentation: a survey

Int. J. Comput. Assist. Radiol. Surg.

U-net: convolutional networks for biomedical image segmentation

UNet++: redesigning skip connections to exploit multiscale features in image segmentation

IEEE Trans. Med. Imaging