Abstract
Cardiac Magnetic Resonance (CMR) imaging serves as the gold-standard for evaluating cardiac morphology and function. Typically, a multi-view CMR stack, covering short-axis (SA) and 2/3/4-chamber long-axis (LA) views, is acquired for a thorough cardiac assessment. However, efficiently streamlining the complex, high-dimensional 3D+T CMR data and distilling compact, coherent representation remains a challenge. In this work, we introduce a whole-heart self-supervised learning framework that utilizes masked imaging modeling to automatically uncover the correlations between spatial and temporal patches throughout the cardiac stacks. This process facilitates the generation of meaningful and well-clustered heart representations without relying on the traditionally required, and often costly, labeled data. The learned heart representation can be directly used for various downstream tasks. Furthermore, our method demonstrates remarkable robustness, ensuring consistent representations even when certain CMR planes are missing/flawed. We train our model on 14,000 unlabeled CMR data from UK BioBank and evaluate it on 1,000 annotated data. The proposed method demonstrates superior performance to baselines in tasks that demand comprehensive 3D+T cardiac information, e.g. cardiac phenotype (ejection fraction and ventricle volume) prediction and multi-plane/multi-frame CMR segmentation, highlighting its effectiveness in extracting comprehensive cardiac features that are both anatomically and pathologically relevant. The code is available at https://github.com/Yundi-Zhang/WholeHeartRL.git.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bai, W., Suzuki, H., Huang, J., Francis, C., et al.: A population-based phenome-wide association study of cardiac and aortic structure and function. Nat. Med. 26(10), 1654–1662 (2020)
Biffi, C., et al.: Learning interpretable anatomical features through deep generative models: application to cardiac remodeling. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 464–471. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_52
Campello, V.M., Gkontra, P., Izquierdo, C., Martin-Isla, C., et al.: Multi-centre, multi-vendor and multi-disease cardiac segmentation: the M &MS challenge. IEEE Trans. Med. Imaging 40(12), 3543–3554 (2021)
Chartsias, A., et al.: Factorised spatial representation learning: application in semi-supervised myocardial segmentation. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 490–498. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_55
Chen, C., Biffi, C., Tarroni, G., Petersen, S., Bai, W., Rueckert, D.: Learning shape priors for robust cardiac MR segmentation from multi-view images. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11765, pp. 523–531. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32245-8_58
Chen, C., Qin, C., Qiu, H., Tarroni, G., et al.: Deep learning for cardiac image segmentation: a review. Front. Cardiovasc. Med. 7, 25 (2020)
Chen, X., He, K.: Exploring simple siamese representation learning. In: CVPR, pp. 15750–15758 (2021)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
Grill, J.B., Strub, F., Altché, F., Tallec, C., et al.: Bootstrap your own latent-a new approach to self-supervised learning. In: NeurIPS, vol. 33, pp. 21271–21284 (2020)
Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., et al.: UNETR: transformers for 3D medical image segmentation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 574–584 (2022)
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: CVPR, pp. 16000–16009 (2022)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)
Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18(2), 203–211 (2021)
Khened, M., Kollerathu, V.A., Krishnamurthi, G.: Fully convolutional multi-scale residual densenets for cardiac segmentation and automated cardiac diagnosis using ensemble of classifiers. Med. Image Anal. 51, 21–45 (2019)
von Knobelsdorff-Brenkenhoff, F., Pilz, G., Schulz-Menger, J.: Representation of cardiovascular magnetic resonance in the AHA/ACC guidelines. J. Cardiovasc. Magn. Reson. 19(1), 1–21 (2017)
Luo, G., Sun, G., Wang, K., Dong, S., Zhang, H.: A novel left ventricular volumes prediction method based on deep learning network in cardiac MRI. In: Computing in Cardiology Conference, pp. 89–92 (2016)
Meng, Q., Qin, C., Bai, W., Liu, T., et al.: Mulvimotion: shape-aware 3D myocardial motion tracking from multi-view cardiac MRI. IEEE TMI 41(8), 1961–1974 (2022)
Pan, J., Huang, W., Rueckert, D., Küstner, T., Hammernik, K.: Reconstruction-driven motion estimation for motion-compensated MR cine imaging. IEEE TMI (2024)
Pan, J., Rueckert, D., Küstner, T., Hammernik, K.: Efficient image registration network for non-rigid cardiac motion estimation. In: Haq, N., Johnson, P., Maier, A., Würfl, T., Yoo, J. (eds.) MLMIR 2021. LNCS, vol. 12964, pp. 14–24. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88552-6_2
Pan, J., Shit, S., Turgut, Ö., Huang, W., et al.: Global k-space interpolation for dynamic MRI reconstruction using masked image modeling. In: Greenspan, H., et al. (eds.) MICCAI 2023. LNCS, vol. 14229, pp. 228–238. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-43999-5_22
Petersen, S.E., Matthews, P.M., Francis, J.M., Robson, M.D., et al.: UK Biobank’s cardiovascular magnetic resonance protocol. JCMR 1–7 (2015)
Qin, C., et al.: Joint learning of motion estimation and segmentation for cardiac MR image sequences. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 472–480. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_53
Qiu, J., Huang, P., Nakashima, M., Lee, J., et al.: Multimodal representation learning of cardiovascular magnetic resonance imaging. arXiv preprint arXiv:2304.07675 (2023)
Radhakrishnan, A., Friedman, S.F., Khurshid, S., Ng, K., et al.: Cross-modal autoencoder framework learns holistic representations of cardiovascular state. Nat. Commun. 14(1), 2436 (2023)
Schlemper, J., et al.: Cardiac MR segmentation from undersampled k-space using deep latent representation learning. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 259–267. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_30
Stolt-Ansó, N., McGinnis, J., Pan, J., Hammernik, K., Rueckert, D.: NISF: neural implicit segmentation functions. In: Greenspan, H., et al. (eds.) MICCAI 2023. LNCS, vol. 14223, pp. 734–744. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-43901-8_70
Sun, X., Liu, Z., Zheng, S., Lin, C., et al.: Attention-enhanced disentangled representation learning for unsupervised domain adaptation in cardiac segmentation. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) MICCAI 2022. LNCS, vol. 13437, pp. 745–754. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16449-1_71
Turgut, Ö., Müller, P., Hager, P., Shit, S., et al.: Unlocking the diagnostic potential of ECG through knowledge transfer from cardiac MRI. arXiv preprint arXiv:2308.05764 (2023)
Wang, H., Amini, A.A.: Cardiac motion and deformation recovery from MRI: a review. IEEE TMI 31(2), 487–503 (2011)
Wang, S., et al.: Joint motion correction and super resolution for cardiac segmentation via latent optimisation. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12903, pp. 14–24. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87199-4_2
Xue, W., Islam, A., Bhaduri, M., Li, S.: Direct multitype cardiac indices estimation via joint representation and regression learning. IEEE TMI 36(10), 2057–2067 (2017)
Zhang, Q., Wang, Y., Wang, Y.: How mask matters: towards theoretical understandings of masked autoencoders. In: NeurIPS, vol. 35, pp. 27127–27139 (2022)
Zhang, Y., Stolt-Ansó, N., Pan, J., Huang, W., Hammernik, K., Rueckert, D.: Direct cardiac segmentation from undersampled k-space using transformers. arXiv preprint arXiv:2406.00192 (2024)
Zhen, X., Wang, Z., Islam, A., Bhaduri, M., et al.: Multi-scale deep networks and regression forests for direct bi-ventricular volume estimation. Med. Image Anal. 30, 120–129 (2016)
Zhou, L., Liu, H., Bae, J., He, J., et al.: Self pre-training with masked autoencoders for medical image classification and segmentation. In: IEEE ISBI, pp. 1–6 (2023)
Acknowledgments
This research has been conducted using the UK Biobank Resource under Application Number 87802. This work is funded by the European Research Council (ERC) project Deep4MI (884622).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, Y., Chen, C., Shit, S., Starck, S., Rueckert, D., Pan, J. (2024). Whole Heart 3D+T Representation Learning Through Sparse 2D Cardiac MR Images. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15001. Springer, Cham. https://doi.org/10.1007/978-3-031-72378-0_34
Download citation
DOI: https://doi.org/10.1007/978-3-031-72378-0_34
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72377-3
Online ISBN: 978-3-031-72378-0
eBook Packages: Computer ScienceComputer Science (R0)
