Co-trained convolutional neural networks for automated detection of prostate cancer in multi-parametric MRI
Graphical abstract
Introduction
Prostate cancer (PCa) is one of the most lethal cancers with a high incidence rate. It has been reported that there are 180,890 newly diagnosed patients and 26,120 deaths caused by PCa in 2016 (Siegel et al., 2016). The number of PCa diagnoses is expected to increase to 1,700,000 globally by 2030, and could result in up to 500,000 related deaths annually (Maddams et al., 2012). Early detection and risk assessment of PCa with a proper treatment can effectively prevent it from progressing to advanced metastatic disease, greatly increasing the survival rate of patients and maintaining the quality of patients’ life.
Current clinical practice for PCa diagnosis is performed by prostate specific antigen (PSA) blood test and/or digital rectal examination (DRE), followed by a transrectal ultrasound (TRUS) biopsy if the result of PSA test is positive. However, due to a limited number of biopsy samples and/or low resolution of ultrasound for TRUS, lesions could be missed or the Gleason score (GS) determined from biopsy samples could vary between repeated biopsies and sometimes also differs from those determined by radical prostatectomy (Weinreb et al., 2014, de Rooij, Hamoen, Fütterer, Barentsz, Rovers, 2014). A recent report based on a large screening trial using PSA and TRUS biopsies has shown that a combination of PSA and TRUS biopsies has relatively low sensitivity and specificity (Schröder et al., 2009), which could lead to delayed diagnosis and treatment, as well as over-diagnosis and overtreatment, which in turn yields patients’ unnecessary discomfort and cost.
Recent studies (Fehr, Veeraraghavan, Wibmer, Gondo, Matsumoto, Vargas, Sala, Hricak, Deasy, 2015, Peng, Jiang, Yang, Brown, Antic, Sethi, Schmid-Tannwald, Giger, Eggener, Oto, 2013, Turkbey, Choyke, 2012) have demonstrated that multi-parametric magnetic resonance imaging (mp-MRI) could provide a noninvasive and more accurate way for PCa detection and cancer foci localization. Prostate Imaging Reporting and Data System Version 2.0 (PI-RADS 2.0) (Radiology et al., 2015) defines that mp-MRI for PCa detection and diagnosis typically include T2-weighted imaging (T2w), diffusion-weighted imaging (DWI), MR spectroscopy imaging (MRSI), and dynamic contrast-enhanced (DCE) MRI. Various combinations of MRI modalities have been investigated. For instance, Fehr et al. (2015) combined the apparent diffusion coefficient (ADC) from DWI and T2w for GS classification (i.e. vs. ≥ 7, and vs. ) and Peng et al. (2013) combined ADC, T2w and Ktrans from DCE-MRI to differentiate PCa from normal tissues. With a better PCa detection rate of mp-MRI, several studies proposed to fuse mp-MRI and ultrasound imaging for a target biopsy (Turkbey et al., 2011) in which mp-MRI images are used to effectively guide the biopsies towards suspicious lesions and in turn to improve the localization accuracy.
However, interpreting mp-MRI sequences manually is a demanding task which requires substantial expertise from radiologists and is quite time consuming. In addition, manual interpretation by radiologists usually suffers from large inter-/intra-observer variations, and low sensitivity (i.e. 67–81%) and specificity (i.e. 46–69%) especially for distinguishing PCa from benign prostate hyperplasia (BPH) at transition zones (TZ) as both PCa and BPH tissues have similarly abnormal and lower signal responses (Valerio et al., 2015). Therefore, automated and accurate PCa detection from mp-MRI sequences is of high demand for minimizing reading time, alleviating requirement for expertise in radiology reading, reducing risk of over-/under-treatment, and enabling large-scale PCa screening.
Prior work: Several computer-aided systems (Liu, Wang, Turkbey, Grant, Pinto, Choyke, Wood, Summers, 2013, Lemaitre, 2016, Litjens, Vos, Barentsz, Karssemeijer, Huisman, 2011, Litjens, Barentsz, Karssemeijer, Huisman, 2012, Litjens, Debats, Barentsz, Karssemeijer, Huisman, 2014, Fehr, Veeraraghavan, Wibmer, Gondo, Matsumoto, Vargas, Sala, Hricak, Deasy, 2015, Artan, Haider, Langer, van der Kwast, Evans, Yang, Wernick, Trachtenberg, Yetik, 2010, Niaf, Rouvière, Mège-Lechevallier, Bratan, Lartizien, 2012, Tiwari, Kurhanewicz, Madabhushi, 2013) have been developed in the past decade for accurate and automated PCa detection and diagnosis. A comprehensive survey of computer-aided detection and diagnosis (CADs) systems of PCa on mp-MRI is available in Lemaitre (2016); Wang et al. (2014). In general, CADs can be classified into two categories: (1) computer-aided detection (i.e. CADe), which primarily focuses on identifying the presence of PCa and localizing possible lesions in mp-MRI images given an entire mp-MRI data of a patient, and (2) computer-aided diagnosis (i.e. CADx), which mainly aims at assessing the aggressiveness of PCa given a set of patches selected manually by radiologists or automatically by a CADe system. In this work, we focus on an automated CADe system and in the following we mainly review related works for CADe and omit the survey of CADx. Existing PCa detection systems typically consist of four separate steps: (1) prostate segmentation, (2) voxel feature representation and classification within the segmented region to generate a malignancy likelihood map with each voxel in the map indicating the probability of being cancerous, (3) candidate lesion region detection based on the likelihood map, and (4) candidate regions representation and classification. Existing CADe systems differ in terms of features used for voxel and candidate region classification, MRI modalities, and methods used for classification and multimodal fusion. The first CADe system of PCa was developed by Chan et al. (2003), which extracted texture features from line-scan diffusion, T2 and T2w images using co-occurrence matrix and discrete cosine transform. An SVM classifier is then applied to generate a malignancy likelihood map at the peripheral zone (PZ) of a prostate. In Langer et al. (2009), the authors employed DCE-MRI images and pharmacokinetic parameter maps as features for PCa voxel classification at PZ of a prostate. Tiwari et al. (2012) investigated the use of magnetic resonance spectroscopy in combination with T2w imaging to identify the voxels that are affected by PCa. In Litjens et al. (2012), the authors applied Hessian matrix based blob detection to every voxel of an ADC map at multiple scales. Each voxel is first represented using a list of features including intensities and blobness of ADC, Ktrans, Kep, Late Wash, T2w, homogeneity and texture strength, and then classified to generate candidate regions. A list of texture-based features extracted from T1 and ADC maps are then used to represent each candidate region, followed by a two-stage classification procedure to detect lesions. In Litjens et al. (2014), the authors further improve their previous method (Litjens et al., 2012) by introducing anatomical voxel features and pharmacokinetic voxel features into their CAD system. In Artan et al. (2010), the authors proposed cost-sensitive SVMs to improve the accuracy of PCa localization. Additionally, they further extended their method for PCa region segmentation by integrating conditional random fields (CRF) into the localization framework. Tiwari et al. (2013) designed a semi-supervised multi-kernel graph embedding (SeSMiK-GE) method for fusing structural and metabolic imaging data from mp-MRI. Despite success of existing systems, most of them, if not all, rely on two-stage classification for PCa detection, i.e. voxel-level classification for candidate generation and region-level classification for verification. As a result, miss detection of lesions in voxel-level classification could propagate to the subsequent step, degrading the overall detection accuracy. In addition, features used for both voxel-level and region-level classification are ad-hoc and handcrafted which are empirically designed and validated based on a small dataset. The robustness and distinctiveness of these handcrafted features for PCa detection on a large population with high data variety remain unclear. Moreover, voxel-level classification in most existing systems completely ignores global information of a prostate, yielding a large amount of false positives. Although candidate regions are further verified using more information within the neighborhood of each candidate point, determination of the neighborhood size, which will greatly affects the final accuracy, is a non-trivial task. Most existing systems empirically chose this parameter via exhaustive search (Litjens et al., 2014), yielding inferior performance for mp-MRI data obtained using different acquisition settings.
In this work, we developed an automated CADe system that identifies the presence of PCa and localizes the lesion regions based on a single-stage image-level classifier, rather than two-stage classification used in most existing systems. Specifically, we trained deep convolutional neural networks (CNNs) for image-level classification of 2D mp-MRI slices which can best distinguish slices containing lesions from those not containing cancerous tissues. Different from most existing CNNs which consist of a batch of convolutional layers followed by a few fully connected layers (FC) and the softmax function for feature learning and image classification, we replace FC layers with a convolutional layer and apply a global average pooling for both training and final estimation. The last convolutional layer (i.e. the newly added convolutional layer) produces a single feature map in which the value of each pixel indicates the likelihood of this position to be cancerous. Therefore, we also consider the feature map as a PCa cancer response map (CRM). To further improve the detection rate of PCa lesions, we combine multimodal information from both ADC and T2w images of an mp-MRI sequence. Specifically, we propose co-trained CNNs whose loss function is designed not only for optimizing image-level classification for both ADC and T2w slices, but also for producing consistent CRMs from both modalities. Our co-trained CNNs provide a more in-depth multimodal fusion than most existing fusion methods as the entire feature learning process of one modality is guided by the other modality, in addition to its own, for producing consistent maps. In comparison, most existing methods (Ngiam, Khosla, Kim, Nam, Lee, Ng, 2011, Shiwen, Xinran, Willam, Alex, Holden, Michael, Steven, Daniel, Kyunghyun, 2016, Xinran, HungLe, Holden, Michael, Steven, William, Xin, Kyunghyun, 2017) directly concatenate high dimensional features or combine classification results of different modalities in a CNN network. For these methods, the feature learning process for each modality is completely independent from that of the other modalities, and the role of fusion is to simply assign weights to different features or results. Once a reliable cancer response map is obtained, a two-step post-processing is applied to remove false positives. In the first post-processing step, we perform global average pooling right before the last convolutional layer, yielding a high-dimensional sematic feature vector. Feature vectors from both modalities are concatenated, to which a SVM classifier is applied for filtering mis-classified image slices. Second, for slices which are classified as containing PCa we apply adaptive thresholding to the cancer response maps based on Otsu’s (1975) algorithm to remove false positives with relatively small responses. For each localized PCa lesion, we further extract co-trained CNN features and classify its GS score (i.e. GS ≤ 6, GS 8, and 9) using a SVM classifier.
The approaches most similar to ours are those reported in Oquab et al. (2015) and Zhou et al. (2015), which targeted natural image classification and object localization, and also trained weakly-supervised CNN classifiers based on image-level labels. In Oquab et al. (2015), the authors applied the global max-pooling (GMP) operation, based on which a single point lying in the boundary of a lesion can be localized. However, this method is not effective for patients with multiple lesions and cannot detect the extent of the lesion. In contrast, we apply the global average-pooling (GAP) operation based on which all discriminative regions of lesions can be identified. In Zhou et al. (2015), the authors also employed the GAP operation to generate a class activation map (CAM) which identifies the importance of image regions related to a particular class. Different from this method which applied GAP right before the FC layer and calculated a CAM after the final classification step by multiplying the weights of the output layer with all the convolutional feature maps, we replace the FC layer with a convolutional layer for generating a CRM and apply the GAP operation after the last convolutional layer just before the final output sigmoid function. As a result, our cancer response map is obtained explicitly before the final classification step. This modification facilitates the fusion of CRMs of the two modalities during the training phases in a co-trained fashion. In addition, both Oquab et al. (2015) and Zhou et al. (2015) are designed for images of a single modality, while our co-trained CNNs take input of multiple modalities and perform an in-depth multimodal fusion for better performance. More detailed theoretical and experimental comparison with the two methods will be presented in Sections 2.2 and 3.4.
To summarize, the main contributions of this work include:
- •
We developed a novel system for automated PCa detection, which relies on an image-level classifier to concurrently identify the presence of PCa in an image and to localize the lesion foci. Representative visual features of lesions are learned automatically during CNN training and highlighted in a feature map (i.e. the cancer response map). To the best of our knowledge, this is the first CADe system which applies a single-stage image-level classifier based on CNN for automated PCa detection, instead of using a voxel-level classifier followed by a region-level classifier as existing systems do.
- •
We propose novel co-trained CNNs for fusing ADC and T2w image of an mp-MRI sequence. In contrast to most multimodal fusion methods which directly concatenate feature vectors of different modalities or compute a weighted sum of results from different modalities (Ngiam, Khosla, Kim, Nam, Lee, Ng, 2011, Chan, Wells III, Mulkern, Haker, Zhang, Zou, Maier, Tempany, 2003), our co-trained CNNs enforce both consistent classification results and cancer response maps from ADC and T2w images and embed such enforcements in the back propagation of the network. Consequently, feature learning processes of different modalities mutually affect each other, yielding a better fusion result.
- •
We conducted extensive experimental evaluations and made comparison on a large dataset including 160 patients with 12-core systematic TRUS-guided prostate biopsy as the reference standard. This dataset contains 72 patients with prostate cancer and 88 patients with benign prostatic hyperplasia. Experimental results demonstrated that our method significantly outperforms the-state-of-the-art localization methods based on CNNs (Oquab, Bottou, Laptev, Sivic, 2015, Zhou, Khosla, Lapedriza, Oliva, Torralba) and the 6-core systematic prostate biopsy method. In particular, our system achieves sensitivity (i.e. detection rate of lesions) of 0.46, 0.92 and 0.97 at 0.1, 1 and 10 false positives per normal/benign patient. Additionally, our system shows good robustness to severe imbalanced dataset with much more normal/benign images than PCa images, which is a common clinical scenario. Specifically, as the ratio of the number of noncancerous images vs. the number of cancerous images increases from 1:1 to 15.5:1, the area under the curve of FROC decreases slightly from 0.9703 to 0.9576. To the best of our knowledge, our work is one of the very few studies (except for Litjens, Vos, Barentsz, Karssemeijer, Huisman, 2011, Litjens, Barentsz, Karssemeijer, Huisman, 2012, Litjens, Debats, Barentsz, Karssemeijer, Huisman, 2014) which conducted evaluation of automated PCa detection on a large dataset with over 100 patients meanwhile achieves a high detection rate of lesions, low false positive rate for noncancerous data and good robustness to severe data imbalance.
Section snippets
Method
Given an mp-MRI data of a patient including ADC and T2w image sequences, our goal is to (1) automatically classify whether each slice of the mp-MRI contains cancer or not, and (2) for slices classified as positive, localize the position of the cancerous tissues for further targeted prostate biopsies. Fig. 1 illustrates the framework of our automated PCa detection system, which consists of four main steps. First, the ADC and T2w images are aligned in the spatial domain to correct motion-caused
Experiments
This section presents data collection, reference standard generation, evaluation metrics, and training and testing data preparation for experimental evaluation, followed by results and analysis.
Conclusions and future work
This paper presents a novel PCa detection system which could automatically identify the presence of PCa in a patient and, once identified, further localize lesions. In contrast to most existing automated PCa detection system which rely on voxel-level classification followed by region-level classification based on empirically designed handcrafted features, our system employs weakly-supervised CNNs which learn representative lesion features from entire prostate images with only image-level labels
Acknowledgment
This work was supported by the National Natural Science Foundation of China grant 61502188.
References (49)
- et al.
Attribute openings, thinnings, and granulometries
Comput. Vision Image Understanding
(1996) A brief history of free-response receiver operating characteristic paradigm data analysis
Acad. Radiol.
(2013)- et al.
Random systematic versus directed ultrasound guided transrectal core biopsies of the prostate
J. Urol.
(1989) - et al.
Computer-aided detection and diagnosis for prostate cancer based on mono and multi-parametric mri: a review
Comput. Biol. Med.
(2015) - et al.
Multi-kernel graph embedding for detection, gleason grading of prostate cancer via mri/mrs
Med. Image Anal.
(2013) - et al.
Detection of clinically significant prostate cancer using magnetic resonance imaging–ultrasound fusion targeted biopsy: a systematic review
Eur. Urol.
(2015) - et al.
PI-RADS prostate imagingreporting and data system: 2015, version 2
Eur. Urol.
(2016) - et al.
Prostate cancer localization with multispectral mri using cost-sensitive support vector machines and conditional random fields
IEEE Trans. Image Process.
(2010) - et al.
Region-specific hierarchical segmentation of mr prostate using discriminative learning
MICCAI Grand Challenge: Prostate MR Image Segmentation, 2012
(2012) - et al.
Influence of imaging and histological factors on prostate cancer detection and localisation on multiparametric mri: a prospective study
Eur. Radiol.
(2013)
Detection of prostate cancer by integration of line-scan diffusion, t2-mapping and t2-weighted magnetic resonance imaging; a multichannel statistical classifier
Med. Phys.
Elastic registration of multimodal prostate mri and histology via multiattribute combined mutual information
Med. Phys.
Graph-based prostate extraction in t2-weighted images for prostate cancer detection
Fuzzy Systems and Knowledge Discovery (FSKD), 2015 12th International Conference on
Automatic classification of prostate cancer gleason scores from multiparametric magnetic resonance images
Proc. Natl. Acad. Sci.
Comparison of quantitative t2 mapping and diffusion-weighted imaging in the normal and pathologic prostate
Magn. Reson. Med.
Rich feature hierarchies for accurate object detection and semantic segmentation
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Prostate cancer detection with multi-parametric mri: logistic regression analysis of quantitative t2, diffusion-weighted imaging, and dynamic contrast-enhanced mri
J. Magn. Reson. Imaging
Computer-Aided Diagnosis for Prostate Cancer using Multi-Parametric Magnetic Resonance Imaging
Automated computer-aided detection of prostate cancer in mr images: from a whole-organ to a zone-based approach
SPIE Medical Imaging
Computer-aided detection of prostate cancer in mri
IEEE Trans. Med. Imaging
Automatic computer aided detection of abnormalities in multi-parametric prostate mri
SPIE Medical Imaging
A prostate cancer computer-aided diagnosis system using multimodal magnetic resonance imaging and targeted biopsy labels
SPIE medical imaging
Fully convolutional networks for semantic segmentation
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Cited by (132)
Research related to the diagnosis of prostate cancer based on machine learning medical images: A review
2024, International Journal of Medical InformaticsInter-reader agreement of the prostate imaging quality (PI-QUAL) score for basic readers in prostate MRI: A multi-center study
2023, European Journal of RadiologyThe role of AI in prostate MRI quality and interpretation: Opportunities and challenges
2023, European Journal of RadiologyArtificial intelligence-assisted selection and efficacy prediction of antineoplastic strategies for precision cancer therapy
2023, Seminars in Cancer Biology
- 1
The two authors have equal contributions to the paper.