3D Cardiac Shape Prediction with Deep Neural Networks: Simultaneous Use of Images and Patient Metadata

Abstract

Large prospective epidemiological studies acquire cardiovascular magnetic resonance (CMR) images for pre-symptomatic populations and follow these over time. To support this approach, fully automatic large-scale 3D analysis is essential. In this work, we propose a novel deep neural network using both CMR images and patient metadata to directly predict cardiac shape parameters. The proposed method uses the promising ability of statistical shape models to simplify shape complexity and variability together with the advantages of convolutional neural networks for the extraction of solid visual features. To the best of our knowledge, this is the first work that uses such an approach for 3D cardiac shape prediction. We validated our proposed CMR analytics method against a reference cohort containing 500 3D shapes of the cardiac ventricles. Our results show broadly significant agreement with the reference shapes in terms of the estimated volume of the cardiac ventricles, myocardial mass, 3D Dice, and mean and Hausdorff distance.

1 Introduction

Cardiovascular disease (CVD) is the most prevalent cause of death worldwide [1]. Early quantitative assessment of cardiac function and structure allow for proper preventive care, and early cardiovascular treatment. To support such an approach, analysis and interpretation of large-scale population-based cardiovascular magnetic resonance (CMR) imaging studies are of high importance in the medical image analysis community. This helps to identify patterns and trends across population groups, and accordingly, reveal insights into key risk factors before CVDs fully develop.

We believe that true 3D analysis is essential for the structural assessment of global and regional cardiac function. We propose a new approach that ensures the global coherence of the cardiac anatomy and naturally lends itself to further analysis in which full 3D anatomy is necessary; for example, in mechanical and flow simulations, or modelling the relationship between cardiac morphology and patient information such as: socio-demographic, lifestyle and environmental, family history, genetic, and omics data.

Though fully automatic 3D segmentation is required for further analysis, the complexity of anatomical structures and their local intensity variation across a population cohort make it challenging. Statistical 3D shape model-based approaches such as [1] have been successfully used for automatically segmenting cardiac structures and generating associated function indexes. This is mainly attributed to the inclusion of prior knowledge of the cardiac shape into the segmentation method. These segmentation approaches typically use very simple features such as gradients on intensity profiles to fit a 3D model. This is an iterative process in which the goal is to minimise the Mahalanobis distance between an intensity profile sampled at a candidate position and its corresponding intensity appearance model by deforming the shape within its range of normal variation to match the image data. On the other hand, in the last decade, fully convolutional networks (FCN) have shown great potential in image-based pattern recognition in a variety of tasks, including cardiac segmentation. However, their output results are, by nature, 2D segmentation masks for every short axis (SAX) and long axis (LAX) CMR slices. Although these 2D masks are sometimes extended via a further step of non-rigid registration to a 3D atlas to produce a 3D cardiac shape [2], this is not efficient for learning topological shape information.

In this paper, we propose to exploit image features obtained using deep FCNs trained on both SAX and LAX views, along with the rich shape priors learned using statistical shape models, to jointly and simultaneously predict the parameters of 3D cardiac shapes, instead of a pixel-wise classification across each 2D slice. Another significant aspect of this work is the integration of patient metadata into the process of shape prediction using a Multilayer Perceptron (MLP). This information, which is currently ignored in cardiac segmentation or shape generation, has been shown in different clinical studies to have an impact on cardiac morphology and structure [3]. We hope this work inspires other researchers to exploit the priors offered by patient metadata in other applications for potentially more accurate and patient-specific models.

The Contributions of this Paper are Three-Fold: we propose (1) an innovative end-to-end deep neural network that directly predicts 3D shape parameters derived from a Principal Component Analysis (PCA) space; (2) a novel approach using two CMR image views and patient metadata simultaneously to predict cardiac shape; (3) a creative loss function defined in the domain of 3D shape parameters which weights each PCA mode of variation independently, prioritising the more significant modes and leading to more accurate shape prediction.

2 Methods

2.1 Reference 3D Shapes of Cardiac Ventricles

We generated a reference cohort of 3D shapes through the non-rigid registration of a 3D biventricular model to a set of 3D points obtained from manual delineations using the Coherent Point Drift (CPD) method [4]. The 3D model is comprised of two structures; the Left Ventricle (LV) and the Right Ventricle (RV). The LV is a closed water-tight mesh comprising both endo and epicardial walls. The RV is an open mesh representing only the RV endocardium. The RV has two openings, the atrioventricular valve opening, and the pulmonary valve opening. Figure 1 shows a sample of manual 2D contours and its corresponding 3D shape obtained from the CPD method.

Fig. 1.

An example 3D shape of cardiac ventricles constructed from a stack of 2D manual contours on SAX view slices.

2.2 Point Distribution Model (PDM)

The PDM encodes the mean and variance of the 3D cardiac shapes. The PDM is constructed during training using PCA on a set of generalised Procrustes-aligned shapes. Assume a training set of M shapes, each described by N points in ${\mathcal {R}}^3$, i.e., $({\mathbf {x}}_j^i, {\mathbf {y}}_j^i, {\mathbf {z}}_j^i)$ with $i=1, ..., M$ and $j=1, ..., N$. Further, let ${\mathbf {s}}_i = ({\mathbf {x}}_1^i, {\mathbf {y}}_1^i, {\mathbf {z}}_1^i, ..., {\mathbf {x}}_N^i, {\mathbf {y}}_N^i, {\mathbf {z}}_N^i)^T$ be the i-th vector representing the i-th shape. Finally, let ${\mathbf {S}} = [{\mathbf {s}}^1, ..., {\mathbf {s}}^M]$ be the set of all training shapes in matrix form. The shape class mean and covariance of ${\mathbf {S}}$ is calculated as follows:

$$\begin{aligned} \bar{{\mathbf {s}}} = \dfrac{1}{M}\sum _{i=1}^{M}{\mathbf {s}}_i ~~~~~\mathrm {and}~~~~~ {\mathbf {C}} = \dfrac{1}{M-1}\sum _{i=1}^{M}({\mathbf {s}}_i-\bar{{\mathbf {s}}})({\mathbf {s}}_i-\bar{{\mathbf {s}}})^T \end{aligned}$$

(1)

The shape covariance is represented in a low-dimensional PCA space. This provides l eigenvectors $\varvec{\Phi }= [{\varphi }_1{\varphi }_2...{\varphi }_l]$, and corresponding eigenvalues $\varvec{\Lambda } = diag({\lambda }_1, {\lambda }_2,...,{\lambda }_l)$ computed through the Singular Value Decomposition of the covariance matrix. Hence, assuming the shape class follows a multi-dimensional Gaussian probability distribution, any shape in the shape class can be approximated from the following linear generative model:

$$\begin{aligned} {\mathbf {s}} \approx \bar{{\mathbf {s}}} + \varvec{\Phi }{\mathbf {b}} \end{aligned}$$

(2)

where ${\mathbf {b}}$ are shape parameters restricted to $|{\mathbf {b}}_i| \le \beta \sqrt{\lambda _i}$; we typically set $\beta = 3$ to capture $99.7\%$ of shape variability. The shape parameters of ${\mathbf {s}}$ can then be estimated as follows:

$$\begin{aligned} {\mathbf {b}} = \varvec{\Phi }_l^T ({\mathbf {s}}-\bar{{\mathbf {s}}}). \end{aligned}$$

(3)

Here, the entries of b are the projection coefficients of mean-centred shapes $({\mathbf {s}} - \bar{{\mathbf {s}}})$ along the columns of $\varvec{\Phi }$.

2.3 Images and Metadata

Each CMR image volume was pre-processed as follows. Each 9-slice SAX stack was intensity normalised by saturating the top 0.2% of intensities and scaling between 0 and 1, and spatially normalised by aligning a fixed point, defined as the average point of intersection between the three LAX views and each SAX slice, and angle, defined as the angle of the 4-chamber LAX, to a standard location and angle respectively. A 64 $\times$ 64 px region of interest (ROI) was then sampled from each slice at a 2 mm isotropic resolution. The corresponding 4-chamber LAX views were similarly intensity normalised, with a 80 $\times$ 60 px ROI sampled at the same 2 mm resolution around the point of intersection between the three LAX views and the basel SAX slice, and zero-padded to 80 $\times$ 80 px. Table 1 shows the summary of the metadata available for every image volume including both continuous and categorical variables. All variables were scaled to the range [0, 1], including categorical variables (viz. sex/alcohol $\in (0,1)$, smoking $\in (0,0.5,1)$).

Table 1. Summary of the subject metadata used in this study.

2.4 Network Architecture and Loss Function

Figure 2 shows a diagram of the proposed method. The network has three inputs: SAX view images, LAX view image, and metadata. The output is the predicted shape parameters $b^{P} = \{b^P_i |i=1, ..., k \}$. To train the proposed architecture, we introduce the following loss function for training:

$$\begin{aligned} \mathbf {E(\theta )} = \sum _{i=1}^{k} ~f(b^P_i (\theta ),~b_i^{R})~.~w(i, k) ~~~~\mathrm {where} ~~~~ w(i, k)=\sqrt{\frac{k-i+1}{k}} \end{aligned}$$

(4)

where k is the number of shape parameters, $\theta$ denotes the network parameters, f(.) denotes the absolute error of the difference between the reference value ($b_i^{R}$) and the value ($b^P_i (\theta )$) predicted by the network. w(.) denotes a weighting function depending on the importance of the $i-th$ mode of variation on shape prediction, i.e. it assigns a higher weight to first modes of variation in the shape’s PCA space. The first modes of variations in the PCA space are critical as they are the main parameters to affect the shape structure. Predicting these accurately is, therefore, more important as they have the greatest control over the final predicted shape. Ultimately, having the mean shape, eigenvectors and predicted shape parameters, the final shape can be predicted using Eq. 2.

Fig. 2.

The proposed method extracts a high-level representation of the image from SAX and LAX views using two FCNs, and concatenates the image features together along with the output of an MLP network applied to the metadata. Four fully connected layers with ReLU or Sigmoid activation functions and batch normalisation then produce the k parameters in PCA space which describe the 3D shape of the cardiac ventricles.

Fig. 3.

The architecture of the two FCNs used in this study to obtain a vector of features representing the image information derived from the LAX and SAX views. A separate network is used for each view, with the LAX and SAX networks containing 9 and 15 layers respectively. The FCNs are composed of convolutional layers, with ReLUs and max-pooling. W is the image width and height, $n_S$ and $n_F$ are the number of slices in each image volume and the number of activation maps respectively.

As illustrated in Fig. 3, the FCN used in this work has been adapted from the down-sampling path of a U-Net [5] architecture with an encoder depth of 2 for the LAX and 4 for the SAX images. The last layer of the FCNs have an extra convolutional layer with the kernel size of the current feature map dimensions to produce a vector of features - 1024 for SAX and 256 for LAX. The MLP has 11 inputs (size of metadata feature vector), 3 hidden layers (with 16, 32, and 64 neurons), and an output layer (with 128 neurons). ReLU is used in hidden and output layers.

The outputs of the three sub-networks are concatenated to construct one feature vector (with the size of 1024 + 256 + 128 = 1408 neurons) that contains the behavioural, phenotypic, and demographic information derived from the metadata in addition to visual information from the imaging data. This information is fed into four fully connected layers, with ReLU (first two layers) and Sigmoid (last two layers) activation functions and batch normalisation, so that, by minimising $E(\theta )$ from Eq. 4, they produce the first k parameters in PCA space which describe the 3D shape of the cardiac ventricles. To capture 99.7% of shape variability in the training dataset we set $k=28$ and regress only those parameters from randomly initialised weights.

3 Experiments and Results

3.1 Data and Annotations

We performed experiments on 3500 CMR image volumes from the UK Biobank (UKB) using both end-diastolic and end-systolic time points. In terms of population sample size, experimental setup, and quality control, the most reliable reference annotations of cardiovascular structure and function found in the literature are those reported by [6], in which CMR scans were manually delineated and analysed by a team of eight expert observers. These delineations were used to generate the reference 3D shapes, as explained in Sect. 2.1. The dataset was randomly split into a training (3000) and test set (500). The performance is reported on the test set with mean ± standard deviation.

3.2 Implementation and Training

The method was implemented using Python and Tensorflow. The network was trained using Adam for optimising the loss function (Eq. 4) with the learning rate of 0.001 and iteration number of 50,000 with a batch size of 10 subjects, all of which were determined empirically. There was no data augmentation. Training took $\sim$10 h on Nvidia Tesla V100 GPUs hosted by Amazon Web Service and accessed using the MULTI-X platform [7]. At test time, it took less than a second to predict the shape parameters.

3.3 Accuracy of Predicted Shapes

Figure 4 shows some samples of ventricular shapes generated by our proposed method (in purple) overlaid with the corresponding reference shapes (in grey). It confirms that the network is capable of predicting accurate shape parameters to generate shapes very similar to the reference shapes obtained by manual delineations. To quantify the amount of similarity, we evaluated the performance of the proposed method by computing the Dice index (${\mathcal {D}}$), and the mean (${\mathcal {M}}$) and Hausdorff distance (${\mathcal {H}}$) between reference and predicted shapes. Since this method outputs the parameters of a shape in the space, we first align the two shapes by removing their orientation and translation before computing the aforementioned metrics. ${\mathcal {D}}$ is between 0 and 1, with a higher ${\mathcal {D}}$ indicating a better match between the two shapes. ${\mathcal {M}}$ and ${\mathcal {H}}$ measure the mean and maximum distance, respectively, between the two surfaces, with a lower value indicating a better the agreement. Moreover, we report the effect of including the metadata in Table 2. As expected, the use of the metadata alongside the image information improves the network, leading to a more accurate prediction in all cardiac substructures. In addition to comparing against reference measurements, we also compare against one baseline method proposed by Attar et al. [1] in which the authors carried out 3D analysis of the UKB CMR images using a shape model-based approach where the model is fitted during an iterative process using traditional intensity profiles.

Fig. 4.

Three samples of the generated 3D shapes of LV and RV. The grey shape is the reference whereas the purple shape is the predicted. (Color figure online)

Table 2. Comparison of shape prediction accuracy using only images (IMG) or images with metadata (IMG+MTDT) in terms of ${\mathcal {D}}$ (%), ${\mathcal {M}}$ (mm) and ${\mathcal {H}}$ (mm) for LV endo-/epicardium and RV endocardium. Bold indicates best performing method.

As shown in Table 2, ${\mathcal {D}}$ values show excellent agreement between reference and predicted shapes (${\ge }0.90$). ${\mathcal {M}}$ values are comparable to the in-plane pixel spacing range of 1.8 mm to 2.3 mm found in the UKB. Although ${\mathcal {H}}$ is larger, it is still within an acceptable range when compared with the distance range seen in [1] or [2]. Note that the performance of the proposed method on RV is consistently better than the other approaches. Furthermore, we report the absolute and relative difference of the main cardiac function indexes (viz. LV and RV volume (ml) and myocardium mass (g)) derived from the predicted and the reference shapes in Table 3. The proposed method achieved significantly lower error in volume and mass estimation, with p < 0.001 in paired t-tests.

Table 3. Comparison of the absolute and relative difference between the reference and predicted shapes. Bold indicates best performing method.

Overall, the proposed method (IMG+MTDT) has superior accuracy to reference shapes than [1], while being on average $\sim$30 times faster during test time. This can be attributed to the combined use of image and patient metadata within a single network to directly predict shape parameters. The introduction of the metadata yielded a substantial positive impact on shape prediction with a $\sim$15% average improvement across all metrics. We believe that including this information provides the network with a variable prior by allowing it to learn the likely distributions of shape parameters across different populations.

4 Conclusion

In this study, we presented a fully automatic framework capable of producing 3D cardiac shapes via the simultaneous use of images and patient metadata. We validated our workflow on a reference cohort of 500 subjects for which ground truth shapes exist with promising results. In particular, we showed a significant positive impact from including the metadata. As future work, in addition to investigating the effect of other clinical variables on shape prediction, we would like to increase the robustness of our pipeline to locate the shape in the image space, and handle severe pathological morphology, variable image quality, and alternative modalities. We also plan to explore the use of patient metadata in other deep learning applications.