Skip to main content
Springer Nature Link
Log in

Point2Mesh-Net: Combining Point Cloud and Mesh-Based Deep Learning for Cardiac Shape Reconstruction

  • Conference paper
  • First Online:
Statistical Atlases and Computational Models of the Heart. Regular and CMRxMotion Challenge Papers (STACOM 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13593))

Abstract

Cine magnetic resonance imaging (MRI) is the gold standard modality for the assessment of cardiac anatomy and function. However, a standard cine acquisition typically consists of only a set of intersecting 2D image slices to represent the true 3D geometry of the human heart, thus limiting its utility in various clinical and research settings. In this work, we present a novel geometric deep learning method, Point2Mesh-Net, to directly and efficiently transform a set of 2D MRI slices into 3D cardiac surface meshes. Its architecture consists of an encoder and a decoder, which are based on recent advances in point cloud and mesh-based deep learning, respectively. This allows the network to not only directly process point cloud data, which represents the sparse MRI contours obtained from image segmentation, but also to output 3D triangular surface meshes, which are highly suitable for a variety of follow-up tasks. Furthermore, the Point2Mesh-Net’s hierarchical setup with multiple downsampling and upsampling steps enables multi-scale feature learning and helps the network to successfully overcome the two main challenges of cardiac surface reconstruction: data sparsity and slice misalignment. We evaluate the model on a synthetic dataset derived from a 3D MRI-based statistical shape model and find surface distances between reconstructed and gold standard meshes below the underlying image resolution for multiple anatomical substructures of the heart. In addition, we apply the pre-trained Point2Mesh-Net as part of a multi-step pipeline to cine MRI acquisitions of the UK Biobank dataset and observe realistic mesh reconstructions with various clinical metrics in line with corresponding findings of large-scale population studies.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Bai, W., et al.: A bi-ventricular cardiac atlas built from 1000+ high resolution MR images of healthy subjects and an analysis of shape and motion. Med. Image Anal. 26(1), 133–145 (2015)

    Article  Google Scholar 

  2. Bai, W., et al.: Automated cardiovascular magnetic resonance image analysis with fully convolutional networks. J. Cardiovasc. Magn. Reson. 20(65), 1–12 (2018)

    Google Scholar 

  3. Banerjee, A., et al.: A completely automated pipeline for 3D reconstruction of human heart from 2D cine magnetic resonance slices. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 379(2212), 20200257 (2021)

    Article  Google Scholar 

  4. Banerjee, A., Zacur, E., Choudhury, R.P., Grau, V.: Optimised misalignment correction from cine MR slices using statistical shape model. In: Papież, B.W., Yaqub, M., Jiao, J., Namburete, A.I.L., Noble, J.A. (eds.) MIUA 2021. LNCS, vol. 12722, pp. 201–209. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-80432-9_16

    Chapter  Google Scholar 

  5. Banerjee, A., Zacur, E., Choudhury, R.P., Grau, V.: Automated 3D whole-heart mesh reconstruction from 2D cine MR slices using statistical shape model. In: 44th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 1702–1706 (2022)

    Google Scholar 

  6. Beetz, M., Banerjee, A., Grau, V.: Biventricular surface reconstruction from cine MRI contours using point completion networks. In: 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pp. 105–109 (2021)

    Google Scholar 

  7. Beetz, M., Banerjee, A., Grau, V.: Generating subpopulation-specific biventricular anatomy models using conditional point cloud variational autoencoders. In: Puyol Antón, E., et al. (eds.) STACOM 2021. LNCS, vol. 13131, pp. 75–83. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-93722-5_9

    Chapter  Google Scholar 

  8. Beetz, M., Banerjee, A., Grau, V.: Multi-domain variational autoencoders for combined modeling of MRI-based biventricular anatomy and ECG-based cardiac electrophysiology. Front. Physiol. 13, 991 (2022)

    Article  Google Scholar 

  9. Beetz, M., Banerjee, A., Sang, Y., Grau, V.: Combined generation of electrocardiogram and cardiac anatomy models using multi-modal variational autoencoders. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pp. 1–4 (2022)

    Google Scholar 

  10. Beetz, M., Ossenberg-Engels, J., Banerjee, A., Grau, V.: Predicting 3D cardiac deformations with point cloud autoencoders. In: Puyol Antón, E., et al. (eds.) STACOM 2021. LNCS, vol. 13131, pp. 219–228. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-93722-5_24

    Chapter  Google Scholar 

  11. Chen, X., et al.: Shape registration with learned deformations for 3D shape reconstruction from sparse and incomplete point clouds. Med. Image Anal. 74, 102228 (2021)

    Article  Google Scholar 

  12. Corral Acero, J., et al.: Understanding and improving risk assessment after myocardial infarction using automated left ventricular shape analysis. JACC Cardiovasc. Imaging 15, 1563–1574 (2022)

    Article  Google Scholar 

  13. Defferrard, M., Bresson, X., Vandergheynst, P.: Convolutional neural networks on graphs with fast localized spectral filtering. In: Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 3844–3852 (2016)

    Google Scholar 

  14. Di Folco, M., Moceri, P., Clarysse, P., Duchateau, N.: Characterizing interactions between cardiac shape and deformation by non-linear manifold learning. Med. Image Anal. 75, 102278 (2022)

    Article  Google Scholar 

  15. Fey, M., Lenssen, J.E.: Fast graph representation learning with PyTorch geometric. In: ICLR Workshop on Representation Learning on Graphs and Manifolds (2019)

    Google Scholar 

  16. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  17. Lamata, P., et al.: An automatic service for the personalization of ventricular cardiac meshes. J. R. Soc. Interface 11(91), 20131023 (2014)

    Article  Google Scholar 

  18. Li, L., Camps, J., Banerjee, A., Beetz, M., Rodriguez, B., Grau, V.: Deep computational model for the inference of ventricular activation properties. arXiv preprint arXiv:2208.04028 (2022)

  19. Mauger, C., et al.: An iterative diffeomorphic algorithm for registration of subdivision surfaces: application to congenital heart disease. In: 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 596–599. IEEE (2018)

    Google Scholar 

  20. Mauger, C., et al.: Right ventricular shape and function: cardiovascular magnetic resonance reference morphology and biventricular risk factor morphometrics in UK Biobank. J. Cardiovasc. Magn. Reson. 21(1), 1–13 (2019)

    Article  Google Scholar 

  21. O’Dell, W.G.: Accuracy of left ventricular cavity volume and ejection fraction for conventional estimation methods and 3D surface fitting. J. Am. Heart Assoc. 8(6), e009124 (2019)

    Article  Google Scholar 

  22. Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 8026–8037 (2019)

    Google Scholar 

  23. Petersen, S.E., et al.: Reference ranges for cardiac structure and function using cardiovascular magnetic resonance (CMR) in Caucasians from the UK Biobank population cohort. J. Cardiovasc. Magn. Reson. 19(18), 1–19 (2017)

    Google Scholar 

  24. Petersen, S.E., et al.: Imaging in population science: cardiovascular magnetic resonance in 100,000 participants of UK Biobank - rationale, challenges and approaches. J. Cardiovasc. Magn. Reson. 15(46), 1–10 (2013)

    Google Scholar 

  25. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: deep learning on point sets for 3D classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652–660 (2017)

    Google Scholar 

  26. Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems, pp. 5099–5108 (2017)

    Google Scholar 

  27. Ranjan, A., Bolkart, T., Sanyal, S., Black, M.J.: Generating 3D faces using convolutional mesh autoencoders. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 725–741. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_43

    Chapter  Google Scholar 

  28. Rodero, C., et al.: Linking statistical shape models and simulated function in the healthy adult human heart. PLoS Comput. Biol. 17(4), e1008851 (2021)

    Article  Google Scholar 

  29. Sinclair, M., Bai, W., Puyol-Antón, E., Oktay, O., Rueckert, D., King, A.P.: Fully automated segmentation-based respiratory motion correction of multiplanar cardiac magnetic resonance images for large-scale datasets. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10434, pp. 332–340. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66185-8_38

    Chapter  Google Scholar 

  30. Stokes, M.B., Roberts-Thomson, R.: The role of cardiac imaging in clinical practice. Aust. Prescr. 40(4), 151 (2017)

    Article  Google Scholar 

  31. Suinesiaputra, A., et al.: Statistical shape modeling of the left ventricle: myocardial infarct classification challenge. IEEE J. Biomed. Health Inform. 22(2), 503–515 (2017)

    Article  Google Scholar 

  32. Villard, B., Grau, V., Zacur, E.: Surface mesh reconstruction from cardiac MRI contours. J. Imaging 4(1), 16 (2018)

    Article  Google Scholar 

  33. Wang, Z.-Y., Zhou, X.-Y., Li, P., Theodoreli-Riga, C., Yang, G.-Z.: Instantiation-Net: 3D mesh reconstruction from single 2D image for right ventricle. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12264, pp. 680–691. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59719-1_66

    Chapter  Google Scholar 

  34. Xu, H., Zacur, E., Schneider, J.E., Grau, V.: Ventricle surface reconstruction from cardiac MR slices using deep learning. In: Coudière, Y., Ozenne, V., Vigmond, E., Zemzemi, N. (eds.) FIMH 2019. LNCS, vol. 11504, pp. 342–351. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-21949-9_37

    Chapter  Google Scholar 

  35. Yuan, W., Khot, T., Held, D., Mertz, C., Hebert, M.: PCN: point completion network. In: 2018 International Conference on 3D Vision (3DV), pp. 728–737 (2018)

    Google Scholar 

  36. Zhou, X.-Y., Wang, Z.-Y., Li, P., Zheng, J.-Q., Yang, G.-Z.: One-stage shape instantiation from a single 2D image to 3D point cloud. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11767, pp. 30–38. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32251-9_4

    Chapter  Google Scholar 

Download references

Acknowledgments

This research has been conducted using the UK Biobank Resource under Application Number ‘40161’. The authors express no conflict of interest. The work of M. Beetz is supported by the Stiftung der Deutschen Wirtschaft (Foundation of German Business). A. Banerjee is a Royal Society University Research Fellow and is supported by the Royal Society (Grant No. URF\({\backslash }\)R1\({\backslash }\)221314). The work of A. Banerjee and V. Grau is supported by the British Heart Foundation (BHF) Project under Grant PG/20/21/35082. The work of V. Grau is supported by the CompBioMed 2 Centre of Excellence in Computational Biomedicine (European Commission Horizon 2020 research and innovation programme, grant agreement No. 823712).

Author information

Authors and Affiliations

  1. Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, OX3 7DQ, UK

    Marcel Beetz, Abhirup Banerjee & Vicente Grau

  2. Division of Cardiovascular Medicine, Radcliffe Department of Medicine, University of Oxford, Oxford, OX3 9DU, UK

    Abhirup Banerjee

Authors
  1. Marcel Beetz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abhirup Banerjee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vicente Grau
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marcel Beetz .

Editor information

Editors and Affiliations

  1. Pompeu Fabra University, Barcelona, Spain

    Oscar Camara

  2. King’s College London, London, UK

    Esther Puyol-Antón

  3. Imperial College London, London, UK

    Chen Qin

  4. Inria, Sophia Antipolis, France

    Maxime Sermesant

  5. King’s College London, London, UK

    Avan Suinesiaputra

  6. Fudan University, Shanghai, China

    Shuo Wang

  7. King’s College London, London, UK

    Alistair Young

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Beetz, M., Banerjee, A., Grau, V. (2022). Point2Mesh-Net: Combining Point Cloud and Mesh-Based Deep Learning for Cardiac Shape Reconstruction. In: Camara, O., et al. Statistical Atlases and Computational Models of the Heart. Regular and CMRxMotion Challenge Papers. STACOM 2022. Lecture Notes in Computer Science, vol 13593. Springer, Cham. https://doi.org/10.1007/978-3-031-23443-9_26

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-23443-9_26

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-23442-2

  • Online ISBN: 978-3-031-23443-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships