Elsevier

Medical Image Analysis

Volume 42, December 2017, Pages 212-227
Medical Image Analysis

Co-trained convolutional neural networks for automated detection of prostate cancer in multi-parametric MRI

https://doi.org/10.1016/j.media.2017.08.006Get rights and content

Highlights

  • Our novel system localizes PCa lesions based on weakly-supervised co-trained CNNs.

  • Our fusion method guides CNNs to see true features and suppress irrelevant patterns.

  • We achieve sensitivity of 0.92 at 1 FP/normal patient on a dataset of 160 patients.

Abstract

Multi-parameter magnetic resonance imaging (mp-MRI) is increasingly popular for prostate cancer (PCa) detection and diagnosis. However, interpreting mp-MRI data which typically contains multiple unregistered 3D sequences, e.g. apparent diffusion coefficient (ADC) and T2-weighted (T2w) images, is time-consuming and demands special expertise, limiting its usage for large-scale PCa screening. Therefore, solutions to computer-aided detection of PCa in mp-MRI images are highly desirable. Most recent advances in automated methods for PCa detection employ a handcrafted feature based two-stage classification flow, i.e. voxel-level classification followed by a region-level classification. This work presents an automated PCa detection system which can concurrently identify the presence of PCa in an image and localize lesions based on deep convolutional neural network (CNN) features and a single-stage SVM classifier. Specifically, the developed co-trained CNNs consist of two parallel convolutional networks for ADC and T2w images respectively. Each network is trained using images of a single modality in a weakly-supervised manner by providing a set of prostate images with image-level labels indicating only the presence of PCa without priors of lesions’ locations. Discriminative visual patterns of lesions can be learned effectively from clutters of prostate and surrounding tissues. A cancer response map with each pixel indicating the likelihood to be cancerous is explicitly generated at the last convolutional layer of the network for each modality. A new back-propagated error E is defined to enforce both optimized classification results and consistent cancer response maps for different modalities, which help capture highly representative PCa-relevant features during the CNN feature learning process. The CNN features of each modality are concatenated and fed into a SVM classifier. For images which are classified to contain cancers, non-maximum suppression and adaptive thresholding are applied to the corresponding cancer response maps for PCa foci localization. Evaluation based on 160 patient data with 12-core systematic TRUS-guided prostate biopsy as the reference standard demonstrates that our system achieves a sensitivity of 0.46, 0.92 and 0.97 at 0.1, 1 and 10 false positives per normal/benign patient which is significantly superior to two state-of-the-art CNN-based methods (Oquab et al., 2015; Zhou et al., 2015) and 6-core systematic prostate biopsies.

Introduction

Prostate cancer (PCa) is one of the most lethal cancers with a high incidence rate. It has been reported that there are 180,890 newly diagnosed patients and 26,120 deaths caused by PCa in 2016 (Siegel et al., 2016). The number of PCa diagnoses is expected to increase to 1,700,000 globally by 2030, and could result in up to 500,000 related deaths annually (Maddams et al., 2012). Early detection and risk assessment of PCa with a proper treatment can effectively prevent it from progressing to advanced metastatic disease, greatly increasing the survival rate of patients and maintaining the quality of patients’ life.

Current clinical practice for PCa diagnosis is performed by prostate specific antigen (PSA) blood test and/or digital rectal examination (DRE), followed by a transrectal ultrasound (TRUS) biopsy if the result of PSA test is positive. However, due to a limited number of biopsy samples and/or low resolution of ultrasound for TRUS, lesions could be missed or the Gleason score (GS) determined from biopsy samples could vary between repeated biopsies and sometimes also differs from those determined by radical prostatectomy (Weinreb et al., 2014, de Rooij, Hamoen, Fütterer, Barentsz, Rovers, 2014). A recent report based on a large screening trial using PSA and TRUS biopsies has shown that a combination of PSA and TRUS biopsies has relatively low sensitivity and specificity (Schröder et al., 2009), which could lead to delayed diagnosis and treatment, as well as over-diagnosis and overtreatment, which in turn yields patients’ unnecessary discomfort and cost.

Recent studies (Fehr, Veeraraghavan, Wibmer, Gondo, Matsumoto, Vargas, Sala, Hricak, Deasy, 2015, Peng, Jiang, Yang, Brown, Antic, Sethi, Schmid-Tannwald, Giger, Eggener, Oto, 2013, Turkbey, Choyke, 2012) have demonstrated that multi-parametric magnetic resonance imaging (mp-MRI) could provide a noninvasive and more accurate way for PCa detection and cancer foci localization. Prostate Imaging Reporting and Data System Version 2.0 (PI-RADS 2.0) (Radiology et al., 2015) defines that mp-MRI for PCa detection and diagnosis typically include T2-weighted imaging (T2w), diffusion-weighted imaging (DWI), MR spectroscopy imaging (MRSI), and dynamic contrast-enhanced (DCE) MRI. Various combinations of MRI modalities have been investigated. For instance, Fehr et al. (2015) combined the apparent diffusion coefficient (ADC) from DWI and T2w for GS classification (i.e. 6(3+3) vs.  ≥ 7, and 7(3+4) vs. 7(4+3)) and Peng et al. (2013) combined ADC, T2w and Ktrans from DCE-MRI to differentiate PCa from normal tissues. With a better PCa detection rate of mp-MRI, several studies proposed to fuse mp-MRI and ultrasound imaging for a target biopsy (Turkbey et al., 2011) in which mp-MRI images are used to effectively guide the biopsies towards suspicious lesions and in turn to improve the localization accuracy.

However, interpreting mp-MRI sequences manually is a demanding task which requires substantial expertise from radiologists and is quite time consuming. In addition, manual interpretation by radiologists usually suffers from large inter-/intra-observer variations, and low sensitivity (i.e. 67–81%) and specificity (i.e. 46–69%) especially for distinguishing PCa from benign prostate hyperplasia (BPH) at transition zones (TZ) as both PCa and BPH tissues have similarly abnormal and lower signal responses (Valerio et al., 2015). Therefore, automated and accurate PCa detection from mp-MRI sequences is of high demand for minimizing reading time, alleviating requirement for expertise in radiology reading, reducing risk of over-/under-treatment, and enabling large-scale PCa screening.

Prior work: Several computer-aided systems (Liu, Wang, Turkbey, Grant, Pinto, Choyke, Wood, Summers, 2013, Lemaitre, 2016, Litjens, Vos, Barentsz, Karssemeijer, Huisman, 2011, Litjens, Barentsz, Karssemeijer, Huisman, 2012, Litjens, Debats, Barentsz, Karssemeijer, Huisman, 2014, Fehr, Veeraraghavan, Wibmer, Gondo, Matsumoto, Vargas, Sala, Hricak, Deasy, 2015, Artan, Haider, Langer, van der Kwast, Evans, Yang, Wernick, Trachtenberg, Yetik, 2010, Niaf, Rouvière, Mège-Lechevallier, Bratan, Lartizien, 2012, Tiwari, Kurhanewicz, Madabhushi, 2013) have been developed in the past decade for accurate and automated PCa detection and diagnosis. A comprehensive survey of computer-aided detection and diagnosis (CADs) systems of PCa on mp-MRI is available in Lemaitre (2016); Wang et al. (2014). In general, CADs can be classified into two categories: (1) computer-aided detection (i.e. CADe), which primarily focuses on identifying the presence of PCa and localizing possible lesions in mp-MRI images given an entire mp-MRI data of a patient, and (2) computer-aided diagnosis (i.e. CADx), which mainly aims at assessing the aggressiveness of PCa given a set of patches selected manually by radiologists or automatically by a CADe system. In this work, we focus on an automated CADe system and in the following we mainly review related works for CADe and omit the survey of CADx. Existing PCa detection systems typically consist of four separate steps: (1) prostate segmentation, (2) voxel feature representation and classification within the segmented region to generate a malignancy likelihood map with each voxel in the map indicating the probability of being cancerous, (3) candidate lesion region detection based on the likelihood map, and (4) candidate regions representation and classification. Existing CADe systems differ in terms of features used for voxel and candidate region classification, MRI modalities, and methods used for classification and multimodal fusion. The first CADe system of PCa was developed by Chan et al. (2003), which extracted texture features from line-scan diffusion, T2 and T2w images using co-occurrence matrix and discrete cosine transform. An SVM classifier is then applied to generate a malignancy likelihood map at the peripheral zone (PZ) of a prostate. In Langer et al. (2009), the authors employed DCE-MRI images and pharmacokinetic parameter maps as features for PCa voxel classification at PZ of a prostate. Tiwari et al. (2012) investigated the use of magnetic resonance spectroscopy in combination with T2w imaging to identify the voxels that are affected by PCa. In Litjens et al. (2012), the authors applied Hessian matrix based blob detection to every voxel of an ADC map at multiple scales. Each voxel is first represented using a list of features including intensities and blobness of ADC, Ktrans, Kep, Late Wash, T2w, homogeneity and texture strength, and then classified to generate candidate regions. A list of texture-based features extracted from T1 and ADC maps are then used to represent each candidate region, followed by a two-stage classification procedure to detect lesions. In Litjens et al. (2014), the authors further improve their previous method (Litjens et al., 2012) by introducing anatomical voxel features and pharmacokinetic voxel features into their CAD system. In Artan et al. (2010), the authors proposed cost-sensitive SVMs to improve the accuracy of PCa localization. Additionally, they further extended their method for PCa region segmentation by integrating conditional random fields (CRF) into the localization framework. Tiwari et al. (2013) designed a semi-supervised multi-kernel graph embedding (SeSMiK-GE) method for fusing structural and metabolic imaging data from mp-MRI. Despite success of existing systems, most of them, if not all, rely on two-stage classification for PCa detection, i.e. voxel-level classification for candidate generation and region-level classification for verification. As a result, miss detection of lesions in voxel-level classification could propagate to the subsequent step, degrading the overall detection accuracy. In addition, features used for both voxel-level and region-level classification are ad-hoc and handcrafted which are empirically designed and validated based on a small dataset. The robustness and distinctiveness of these handcrafted features for PCa detection on a large population with high data variety remain unclear. Moreover, voxel-level classification in most existing systems completely ignores global information of a prostate, yielding a large amount of false positives. Although candidate regions are further verified using more information within the neighborhood of each candidate point, determination of the neighborhood size, which will greatly affects the final accuracy, is a non-trivial task. Most existing systems empirically chose this parameter via exhaustive search (Litjens et al., 2014), yielding inferior performance for mp-MRI data obtained using different acquisition settings.

In this work, we developed an automated CADe system that identifies the presence of PCa and localizes the lesion regions based on a single-stage image-level classifier, rather than two-stage classification used in most existing systems. Specifically, we trained deep convolutional neural networks (CNNs) for image-level classification of 2D mp-MRI slices which can best distinguish slices containing lesions from those not containing cancerous tissues. Different from most existing CNNs which consist of a batch of convolutional layers followed by a few fully connected layers (FC) and the softmax function for feature learning and image classification, we replace FC layers with a convolutional layer and apply a global average pooling for both training and final estimation. The last convolutional layer (i.e. the newly added convolutional layer) produces a single feature map in which the value of each pixel indicates the likelihood of this position to be cancerous. Therefore, we also consider the feature map as a PCa cancer response map (CRM). To further improve the detection rate of PCa lesions, we combine multimodal information from both ADC and T2w images of an mp-MRI sequence. Specifically, we propose co-trained CNNs whose loss function is designed not only for optimizing image-level classification for both ADC and T2w slices, but also for producing consistent CRMs from both modalities. Our co-trained CNNs provide a more in-depth multimodal fusion than most existing fusion methods as the entire feature learning process of one modality is guided by the other modality, in addition to its own, for producing consistent maps. In comparison, most existing methods (Ngiam, Khosla, Kim, Nam, Lee, Ng, 2011, Shiwen, Xinran, Willam, Alex, Holden, Michael, Steven, Daniel, Kyunghyun, 2016, Xinran, HungLe, Holden, Michael, Steven, William, Xin, Kyunghyun, 2017) directly concatenate high dimensional features or combine classification results of different modalities in a CNN network. For these methods, the feature learning process for each modality is completely independent from that of the other modalities, and the role of fusion is to simply assign weights to different features or results. Once a reliable cancer response map is obtained, a two-step post-processing is applied to remove false positives. In the first post-processing step, we perform global average pooling right before the last convolutional layer, yielding a high-dimensional sematic feature vector. Feature vectors from both modalities are concatenated, to which a SVM classifier is applied for filtering mis-classified image slices. Second, for slices which are classified as containing PCa we apply adaptive thresholding to the cancer response maps based on Otsu’s (1975) algorithm to remove false positives with relatively small responses. For each localized PCa lesion, we further extract co-trained CNN features and classify its GS score (i.e. GS  ≤ 6, GS =3+4,4+3, 8, and 9) using a SVM classifier.

The approaches most similar to ours are those reported in Oquab et al. (2015) and Zhou et al. (2015), which targeted natural image classification and object localization, and also trained weakly-supervised CNN classifiers based on image-level labels. In Oquab et al. (2015), the authors applied the global max-pooling (GMP) operation, based on which a single point lying in the boundary of a lesion can be localized. However, this method is not effective for patients with multiple lesions and cannot detect the extent of the lesion. In contrast, we apply the global average-pooling (GAP) operation based on which all discriminative regions of lesions can be identified. In Zhou et al. (2015), the authors also employed the GAP operation to generate a class activation map (CAM) which identifies the importance of image regions related to a particular class. Different from this method which applied GAP right before the FC layer and calculated a CAM after the final classification step by multiplying the weights of the output layer with all the convolutional feature maps, we replace the FC layer with a convolutional layer for generating a CRM and apply the GAP operation after the last convolutional layer just before the final output sigmoid function. As a result, our cancer response map is obtained explicitly before the final classification step. This modification facilitates the fusion of CRMs of the two modalities during the training phases in a co-trained fashion. In addition, both Oquab et al. (2015) and Zhou et al. (2015) are designed for images of a single modality, while our co-trained CNNs take input of multiple modalities and perform an in-depth multimodal fusion for better performance. More detailed theoretical and experimental comparison with the two methods will be presented in Sections 2.2 and 3.4.

To summarize, the main contributions of this work include:

  • We developed a novel system for automated PCa detection, which relies on an image-level classifier to concurrently identify the presence of PCa in an image and to localize the lesion foci. Representative visual features of lesions are learned automatically during CNN training and highlighted in a feature map (i.e. the cancer response map). To the best of our knowledge, this is the first CADe system which applies a single-stage image-level classifier based on CNN for automated PCa detection, instead of using a voxel-level classifier followed by a region-level classifier as existing systems do.

  • We propose novel co-trained CNNs for fusing ADC and T2w image of an mp-MRI sequence. In contrast to most multimodal fusion methods which directly concatenate feature vectors of different modalities or compute a weighted sum of results from different modalities (Ngiam, Khosla, Kim, Nam, Lee, Ng, 2011, Chan, Wells III, Mulkern, Haker, Zhang, Zou, Maier, Tempany, 2003), our co-trained CNNs enforce both consistent classification results and cancer response maps from ADC and T2w images and embed such enforcements in the back propagation of the network. Consequently, feature learning processes of different modalities mutually affect each other, yielding a better fusion result.

  • We conducted extensive experimental evaluations and made comparison on a large dataset including 160 patients with 12-core systematic TRUS-guided prostate biopsy as the reference standard. This dataset contains 72 patients with prostate cancer and 88 patients with benign prostatic hyperplasia. Experimental results demonstrated that our method significantly outperforms the-state-of-the-art localization methods based on CNNs (Oquab, Bottou, Laptev, Sivic, 2015, Zhou, Khosla, Lapedriza, Oliva, Torralba) and the 6-core systematic prostate biopsy method. In particular, our system achieves sensitivity (i.e. detection rate of lesions) of 0.46, 0.92 and 0.97 at 0.1, 1 and 10 false positives per normal/benign patient. Additionally, our system shows good robustness to severe imbalanced dataset with much more normal/benign images than PCa images, which is a common clinical scenario. Specifically, as the ratio of the number of noncancerous images vs. the number of cancerous images increases from 1:1 to 15.5:1, the area under the curve of FROC decreases slightly from 0.9703 to 0.9576. To the best of our knowledge, our work is one of the very few studies (except for Litjens, Vos, Barentsz, Karssemeijer, Huisman, 2011, Litjens, Barentsz, Karssemeijer, Huisman, 2012, Litjens, Debats, Barentsz, Karssemeijer, Huisman, 2014) which conducted evaluation of automated PCa detection on a large dataset with over 100 patients meanwhile achieves a high detection rate of lesions, low false positive rate for noncancerous data and good robustness to severe data imbalance.

Section snippets

Method

Given an mp-MRI data of a patient including ADC and T2w image sequences, our goal is to (1) automatically classify whether each slice of the mp-MRI contains cancer or not, and (2) for slices classified as positive, localize the position of the cancerous tissues for further targeted prostate biopsies. Fig. 1 illustrates the framework of our automated PCa detection system, which consists of four main steps. First, the ADC and T2w images are aligned in the spatial domain to correct motion-caused

Experiments

This section presents data collection, reference standard generation, evaluation metrics, and training and testing data preparation for experimental evaluation, followed by results and analysis.

Conclusions and future work

This paper presents a novel PCa detection system which could automatically identify the presence of PCa in a patient and, once identified, further localize lesions. In contrast to most existing automated PCa detection system which rely on voxel-level classification followed by region-level classification based on empirically designed handcrafted features, our system employs weakly-supervised CNNs which learn representative lesion features from entire prostate images with only image-level labels

Acknowledgment

This work was supported by the National Natural Science Foundation of China grant 61502188.

References (49)

  • I. Chan et al.

    Detection of prostate cancer by integration of line-scan diffusion, t2-mapping and t2-weighted magnetic resonance imaging; a multichannel statistical classifier

    Med. Phys.

    (2003)
  • J. Chappelow et al.

    Elastic registration of multimodal prostate mri and histology via multiattribute combined mutual information

    Med. Phys.

    (2011)
  • W. Du et al.

    Graph-based prostate extraction in t2-weighted images for prostate cancer detection

    Fuzzy Systems and Knowledge Discovery (FSKD), 2015 12th International Conference on

    (2015)
  • D. Fehr et al.

    Automatic classification of prostate cancer gleason scores from multiparametric magnetic resonance images

    Proc. Natl. Acad. Sci.

    (2015)
  • P. Gibbs et al.

    Comparison of quantitative t2 mapping and diffusion-weighted imaging in the normal and pathologic prostate

    Magn. Reson. Med.

    (2001)
  • R. Girshick et al.

    Rich feature hierarchies for accurate object detection and semantic segmentation

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2014)
  • D.L. Langer et al.

    Prostate cancer detection with multi-parametric mri: logistic regression analysis of quantitative t2, diffusion-weighted imaging, and dynamic contrast-enhanced mri

    J. Magn. Reson. Imaging

    (2009)
  • G. Lemaitre

    Computer-Aided Diagnosis for Prostate Cancer using Multi-Parametric Magnetic Resonance Imaging

    (2016)
  • Lin, M., Chen, Q., Yan, S., 2013. Network in network. arXiv preprint...
  • G. Litjens et al.

    Automated computer-aided detection of prostate cancer in mr images: from a whole-organ to a zone-based approach

    SPIE Medical Imaging

    (2012)
  • G. Litjens et al.

    Computer-aided detection of prostate cancer in mri

    IEEE Trans. Med. Imaging

    (2014)
  • G. Litjens et al.

    Automatic computer aided detection of abnormalities in multi-parametric prostate mri

    SPIE Medical Imaging

    (2011)
  • P. Liu et al.

    A prostate cancer computer-aided diagnosis system using multimodal magnetic resonance imaging and targeted biopsy labels

    SPIE medical imaging

    (2013)
  • J. Long et al.

    Fully convolutional networks for semantic segmentation

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    (2015)
  • Cited by (132)

    View all citing articles on Scopus
    1

    The two authors have equal contributions to the paper.

    View full text