Multi-feature deep information bottleneck network for breast cancer classification in contrast enhanced spectral mammography

doi:10.1016/j.patcog.2022.108858

Pattern Recognition

Volume 131, November 2022, 108858

https://doi.org/10.1016/j.patcog.2022.108858 Get rights and content

Highlights

•
First information bottleneck based-deep learning study for classifying breast cancer.
•
Multi-feature deep information bottleneck is introduced.
•
The features between contrast enhanced spectral mammography are fully explored.
•
Multiple image features are considered simultaneously in relation to the labels.

Abstract

There is considerable variation in the size, shape and location of tumours, which makes it challenging for radiologists to diagnose breast cancer. Automated diagnosis of breast cancer from Contrast Enhanced Spectral Mammography (CESM) can support clinical decision making. However, existing methods fail to obtain an effective representation of the CESM and ignore the relationships between images. In this paper, we investigated for the first time a novel and flexible multimodal representation learning method, multi-feature deep information bottleneck (MDIB), for breast cancer classification in CESM. Specifically, the method incorporated an information bottleneck (IB)-based module to learn the prominent representation that provide concise input while informative for the classification. In addition, we creatively extended IB theory to multi-feature IB, which facilitates the learning of relevant features for classification between CESM images. To validate our method, experiments were conducted on our private and public datasets. The classification results of our method were also compared with those of state-of-the-art methods. The experiment results proved the effectiveness and the efficiency of the proposed method. We release our code at https://github.com/sjq5263/MDIB-for-CESM-classification.

Introduction

Breast cancer is the second leading cause of cancer death among women worldwide [1]. Early detection is a primary means of reducing breast cancer-related mortality [2]. Several medical imaging modalities are available for the diagnosis of breast cancer, including mammography, ultrasound, MRI and contrast enhanced spectral mammography (CESM). Compared to other imaging modalities, CESM provides low-energy (LE), dual-energy subtracted (DES) modalities images with craniocaudal (CC) and mediolateral oblique (MLO) view of the breast. A set of typical CESM images are presented in Fig. 1. Mammographic findings such as calcifications and masses can be seen from LE image. The DES provides images of enhanced areas due to increased perfusion of breast tissue at the site of the malignancy [3]. In addition, radiologists can view the breast area from multiple perspectives (CC and MLO views). The multi-view multi-modal images in CESM provide more information about the lesion area for the patient’s cancerous condition. Many preliminary studies also confirmed that the sensitivity and specificity of CESM are superior to mammography. Despite improved consistency, variations in the subjective description of imaging findings and limitations in expert experience and effort [4] remain a significant impediment to the accurate diagnosis of CESM.

An automated diagnosis system can be used to tackle the aforementioned problems, which has shown significant results in breast cancer classification. The method based on traditional machine learning (ML) algorithms can focus on radiomics features explicitly selected by domain experts, which relies on certain hand-crafted features. Recently, deep learning (DL) has been used for image classification for its superior performance with end-to-end pipeline automation, which can learn appropriate predictive features in images. After the success of convolutional neural networks (CNN), DL outperforms traditional ML algorithms in computer vision. As artificial intelligence (AI) evolves, DL methods have rivaled the performance of medical experts through learning from a large number of images.

Recently, many researchers utilized DL approaches for the classification of breast cancer in CESM images. Gao et al. [5] proposed a modified DL architecture to improve the classification performance of breast cancer using CESM images. Perek et al. [6] incorporated breast imaging reporting and data systems textual features into AI framework for improving classification accuracy. However, the classification performance of these studies did not achieve optimal results for the following reasons. Firstly, the designed methods only adopted general techniques which cannot accurately learn the representation of breast images, resulting in poor model performance. Secondly, most of these methods used unimodal or single-view breast images and did not utilize multi-featured CESM images by using explicit images as an input to the classification model. Finally, it is well known that the different modalities of breast images (LE and DES) are theoretically able to provide similar information for the lesion area of disease, improving the accuracy of malignant tumors. However, previous studies did not explore the relationship between the different modalities of CESM.

To break through the limitations of previous methods, we creatively conducted a multi-feature deep information bottleneck (MDIB) network for classification of breast cancer, which effectively represents CESM images. Our proposed model makes full use of the multimodal (LE and DES) and multiview (CC and MLO) feature characteristics of CESM images to achieve more accurate classification results. The proposed algorithm has outperformed general DL methods in many aspects. The contributions of this study are threefold. (1) We introduce a DL model, MDIB, which combines IB to learn the more accurate representation of CESM images for classification. To the best of our knowledge, this is the first study to classify CESM images using IB theory. (2) We theoretically extend the IB theory to multiple features. The proposed strategy can extract concise representations between inputs and labels as well as common features between modalities to achieve accurate classification. (3) We conduct an extensive evaluation on our private and a public CESM dataset, and the experimental results validate the effectiveness of the proposed multi-feature IB network for CESM image classification. Promising generalization capabilities are shown empirically compared to state-of-the-art methods.

Section snippets

Multimodal medical image classification

With the extension of clinical applications of various medical imaging systems, multimodal imaging starts playing an increasingly important role [7]. The multimodal medical image (CT, MRI, PET, etc.) typically involves different information with a partial aspect of patient’s condition, which can provide complementary information to another modality [8]. Methods for multimodal medical image classification can be broadly classified into ML-based [9], [10] and DL-based methods [11]. In [12],

Method

We start this section by motivating and giving intuition behind our approach. Then, we introduce the architecture of the proposed method. After that we explain the loss function that is based on IB theory.

Data

We report the performance of the proposed model on private and public datasets. We retrospectively collected CESM from 95 patients aged 21 to 74 years between July 2018 to April 2020 to form our private dataset. The dataset consists of 760 images. Before using the dataset, we preprocessed the data to remove the sensitive information about the patients. Eight images (4 images per unilateral breast) were collected per patient. One patient was considered as two cases because of the different

Discussion

In this study, we applied multi-feature deep information bottleneck theory to network for CESM image classification. The proposed method outperforms the previously available methods in several aspects. First, from Fig. 3, the proposed method with the information bottleneck strategy achieves better results than base models. In addition, the model with two losses of IB works significantly better. The IB strategy allows the model to learn the exact representation between CESM images and labels. We

Conclusion

Based on the analysis characteristics of CESM images, a novel multi-feature deep information bottleneck network is first proposed for CESM classification. The proposed method outperforms state-of-the-art methods for breast cancer classification using CESM images on both private and public datasets. The reason lies in the fact that it makes use of the information bottleneck theory to produce a concise and informative representation of the classification. In addition, the proposed multi-feature

Funding

This work was supported by the National Natural Science Foundation of China [grant numbers 81871508 and 61773246]; the Major Program of Shandong Province Natural Science Foundation [grant number ZR2018ZB0419]; the Major Basic Research Program of Natural Science Foundation of Shandong Province [grant number ZR2019ZD04]; Shandong Provincial Natural Science Foundation Joint Fund [grant number ZR2021LZL011]; and the Taishan Scholar Foundation of Shandong Province [grant number TSHW201502038].

Ethical approval and informed consent

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants included in the study.

Availability of data and materials

The dataset during the current study is available from the corresponding author on reasonable request.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Jingqi Song is currently a PhD candidate at the school of information science and engineering of Shandong Normal University. His current research interests include medical image processing, image classification, and deep learning.

References (32)

Y. Luo et al.
Segmentation information with attention integration for classification of breast tumor in ultrasound image
Pattern Recognit.
(2022)
F. Gao et al.
SD-CNN: a shallow-deep CNN for improved breast cancer diagnosis
Comput. Med. Imaging Graphics
(2018)
Y. Shi et al.
ASMFS: adaptive-similarity-based multi-modality feature selection for classification of Alzheimer’s disease
Pattern Recognit.
(2022)
A. Shapiro
Monte carlo sampling methods
Handb. Oper. Res. Manage.Sci.
(2003)
B.K. Patel et al.
Computer-aided diagnosis of contrast-enhanced spectral mammography: a feasibility study
Eur. J. Radiol.
(2018)
R. Siegel et al.
Cancer statistics, 2021
CA Cancer J. Clin.
(2021)
M. Lobbes
Interpretation of Contrast-Enhanced Mammography
(2019)
X. Qian et al.
Prospective assessment of breast cancer risk from multimodal multiview ultrasound images via clinically applicable deep learning
Nat. Biomed. Eng.
(2021)
S. Perek et al.
Classification of contrast-enhanced spectral mammography (CESM) images
Int. J. Comput. Assist. Radiol. Surg.
(2019)
S. Kavitha et al.
Dual channel pulse coupled neural network algorithm for fusion of multimodality brain images with quality analysis
Appl. Med. Inf.
(2014)

Z. Guo et al.

Deep learning-based image segmentation on multimodal medical imaging

IEEE Trans. Radiat. Plasma Med.Sci.

(2019)

S. Kavitha et al.

A classification system for fused brain images using support vector machine

Int. J. Appl. Eng.Res.

(2015)

S. Kavitha et al.

Efficient DWT-based fusion techniques using genetic algorithm for optimal parameter estimation

Soft Comput.

(2017)

W. Sutong et al.

Interpretability-based multimodal convolutional neural networks for skin lesion diagnosis

IEEE Trans. Cybern.

(2021)

S. Kavitha et al.

Analysis of multimodality brain images using machine learning techniques

International Conference on Communications and Signal Processing

(2015)

S. Kavitha et al.

Features based mammogram image classification using weighted feature support vector machine

Commun. Comput. Inf. Sci.

(2012)

Cited by (6)

Classification of tumor in one single ultrasound image via a novel multi-view learning strategy
2023, Pattern Recognition
Computer-aided diagnosis (CAD) technology has been widely used in the early diagnosis of breast cancer. Nowadays, most of the existing breast ultrasound classification methods need to crop a tumor-centered image (TCI) on each image as the input of the system. These methods ignore the fact that the tumor as well as its surrounding tissues can actually be viewed from multiple aspects, and it is difficult to extract multi-resolution information applying only a single view image. In addition, the current methods do not effectively extract fine-grained features, and subtle details play an important role in breast classification. In our research, we propose a novel strategy to generate multi-resolution TCIs in a single ultrasound image, resulting in a multi-data-input learning task. Hence, a conventional single image based learning task is converted into a multi-view learning task, and an improved combined style fusion method suitable for a deep network is proposed, which integrates the advantage of the decision-based and feature-based methods to fuse the information of different views. At the same time, we first attempt to introduce the fine-grained classification method into breast classifications and capture the pairwise correlation between feature channels at each position to extract subtle information. The comparative experimental results show that our method can effectively improve the classification performance and achieves the best results in five metrics.
RPI-CapsuleGAN: Predicting RNA-protein interactions through an interpretable generative adversarial capsule network
2023, Pattern Recognition
RNA-protein interactions (RPI) play a crucial regulatory role in cellular physiological processes. The study and prediction of RPIs can be insightful for exploring disease mechanisms and drug target design. Traditional RPI prediction methods relied mainly on tedious and expensive biological experiments, and there is an increasing interest in developing more cost-effective computational methods to predict RPIs. This work proposes an interpretable RPI-CapsuleGAN method for RPI prediction based on a generative adversarial capsule network with a convolutional block attention module. First, RPI-CapsuleGAN extracts and fuses multiple features to characterize RNA and protein sequences. Subsequently, the elastic net feature selection method is used to retain features that are highly informative to RPI prediction. Finally, we introduce a convolutional attention mechanism into the generative adversarial capsule network for the first time in order to construct the RPI prediction framework, which is shown to improve the model feature learning of interpretable and expression ability, and effectively solves the problem of the disappearance of the model spatial structure hierarchy. Based on a five-fold cross-validation test, the prediction accuracy of the RPI-CapsuleGAN method reaches 97.1%, 88.8%, 92.5%, 97.3%, and 87.8% for datasets RPI488, RPI369, RPI2241, RPI1807, and RPI1446. The RPI-CapsuleGAN method has higher accuracy than state-of-the-art RPI prediction methods that use the same datasets. In the test dataset NPInter227 constructed in this paper, five groups of test sets are composed of positive samples and five groups of negative samples, the prediction accuracy reaches 97.38%, 96.48%, 97.38%, 97.81%, and 97.15%, respectively, outperforming other mainstream deep learning algorithms. In addition, RPI-CapsuleGAN obtained better results for the prediction of independent test datasets. Extensive experiments detailed here show that RPI-CapsuleGAN can provide an efficient, accurate, and stable method for RPI prediction.
Triple-attention interaction network for breast tumor classification based on multi-modality images
2023, Pattern Recognition
Breast cancer can be diagnosed using medical imaging. Classification performance of medical imaging can be improved by multi-modality image fusion. However, existing fusion algorithm fail to consider the importance of modality interactions and cannot fully utilize multi-modality information. Attention mechanisms can effectively explore and combine multi-modality information. Thus, we propose a novel triple-attention interaction network for breast tumor classification based on diffusion-weighted imaging (DWI) and apparent dispersion coefficient (ADC) images. A triple inter-modality interaction mechanism is proposed to fully fuse the multi-modality information. Three modal interactions were performed through the developed inter-modality relation module, channel interaction module, and multi-level attention fusion module to explore the correlation, complementary, and discriminative information, respectively. Additionally, we introduce a novel dual parallel-attention module for the incorporation of spatial and channel attention to improve the discriminative ability of single-modality features. Using these mechanisms, the proposed algorithm can mine and explore useful multi-modality information fully, to improve classification performance. Experimental results demonstrate that our algorithm outperforms other multi-modality fusion algorithm, and extensive ablation studies were conducted to verify the advantages of our algorithm. The area under the receiver operating characteristic curve, accuracy, specificity, and sensitivity were 90.5%, 89.0%, 85.6%, and 92.4%, respectively.
Modality-correlation embedding model for breast tumor diagnosis with mammography and ultrasound images
2022, Computers in Biology and Medicine
Citation Excerpt :
Most breast tumor diagnosis methods using multi-modality images mainly focus on the classification step, and the diagnosis results can be obtained by combining the classification information of multiple modalities. However, problems arise for existing breast tumor classification methods with multimodality images, and the useful correlation information between mammography and ultrasound is ignored [11–14]. Generally, there exist inherent correlations between these two modalities for the same patient because the underlying pathology is the same, that is, the diagnosis results based on mammography and ultrasound for the same patient should be consistent [15].
The fusion of mammography and ultrasound images helps to improve tumor classification accuracy. However, traditional fusion models ignore the correlation between these two modalities, resulting in limited performance improvement. To address this problem, a modality-correlation embedding model was proposed for breast tumor diagnosis by combining mammography and ultrasound imaging. By jointly optimizing the correlation between mammography and ultrasound and classification loss of individual modalities, two mappings are learned to project mammography and ultrasound from their original feature spaces into a common label space. A novel modality-correlation term is introduced to maintain the pairwise closeness of multimodal data in the common label space. Contrary to previous studies that did not consider the correlation between multimodal data, the proposed term can exploit the learned correlation information in the fusion process, which guarantees the consistency of the diagnosis results of multimodal images from the same patient. The proposed method was evaluated on our dataset, which contained ultrasound and mammography images from 73 patients. The area under the ROC curve, accuracy, sensitivity, specificity, positive predictive value, and negative predictive value were 95.83, 95, 91.67, 95.83, 95.83, and 88.89%, respectively. The experimental results also demonstrate that the proposed method outperforms traditional fusion methods.
YOLO-based CAD framework with ViT transformer for breast mass detection and classification in CESM and FFDM images
2024, Neural Computing and Applications
Sublinear information bottleneck based two-stage deep learning approach to genealogy layout recognition
2023, Frontiers in Neuroscience

Yuanjie Zheng is a professor in the school of information science and engineering at the Shandong Normal University. He has a PhD degree fromiversity of Pennsylvania, USA.is research interests includemputer vision, medical image processing and pattern recognition areas.

Jing Wang is currently a PhD candidate at the school of information science and engineering of Shandong Normal University. Her current research interests include image classification and image detection.

Muhammad Zakir Ullah is currently a MSc candidate at the school of information science and engineering of Shandong Normal University. His current research interests include medical image processing, machine learning and deep learning.

Xuecheng Li is currently a MSc candidate at the school of information science and engineering of Shandong Normal University. His current research interests include medical image segmentation and detection.

Zhenxing Zou is a Radiologist at Medical Imaging Department of Yantai Yuhuangding Hospital.

Guocheng Ding is a Radiologist at Medical Imaging Department of Yantai Yuhuangding Hospital.

View full text

Multi-feature deep information bottleneck network for breast cancer classification in contrast enhanced spectral mammography

Highlights

Abstract

Introduction

Section snippets

Multimodal medical image classification

Method

Data

Discussion

Conclusion

Funding

Ethical approval and informed consent

Availability of data and materials

Declaration of Competing Interest

Pattern Recognit.

Comput. Med. Imaging Graphics

Pattern Recognit.

Handb. Oper. Res. Manage.Sci.

Eur. J. Radiol.

Cancer statistics, 2021

CA Cancer J. Clin.

Interpretation of Contrast-Enhanced Mammography

Prospective assessment of breast cancer risk from multimodal multiview ultrasound images via clinically applicable deep learning

Nat. Biomed. Eng.

Classification of contrast-enhanced spectral mammography (CESM) images

Int. J. Comput. Assist. Radiol. Surg.

Dual channel pulse coupled neural network algorithm for fusion of multimodality brain images with quality analysis

Appl. Med. Inf.

Deep learning-based image segmentation on multimodal medical imaging

IEEE Trans. Radiat. Plasma Med.Sci.

A classification system for fused brain images using support vector machine

Int. J. Appl. Eng.Res.

Efficient DWT-based fusion techniques using genetic algorithm for optimal parameter estimation

Soft Comput.

Interpretability-based multimodal convolutional neural networks for skin lesion diagnosis

IEEE Trans. Cybern.

Analysis of multimodality brain images using machine learning techniques

International Conference on Communications and Signal Processing

Features based mammogram image classification using weighted feature support vector machine

Commun. Comput. Inf. Sci.