Abstract
During the last years tens of challenges have been organized to benchmark computational techniques with shared data. Historically, most challenges in conferences such as MICCAI have been devoted to medical image processing, especially on object recognition or segmentation tasks. Due to the increasing popularity and easy access to machine (deep) learning methods, as part of our current Artificial Intellingence (AI) summer, the number of AI-related challenges has exploded. In parallel, the community of biophysical models also has a valuable history of organizing challenges, including synthetic and experimental data, to assess the accuracy of the resulting simulations. In this paper, the similarities and differences in computational challenges organized by these communities are reviewed, suggesting best practices and what to avoid when organizing a challenge on biophysical models. Specifically, details will be given about the preparation of the CRT-EPiggy19 challenge.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24.
- 25.
- 26.
- 27.
- 28.
- 29.
- 30.
- 31.
- 32.
- 33.
- 34.
- 35.
- 36.
- 37.
- 38.
- 39.
- 40.
- 41.
- 42.
- 43.
References
Ashburner, J.: SPM: a history. NeuroImage 62(2), 791–800 (2012)
ASME: ASME V&V 20–2009: Standard for Verification and Validation in Computational Fluid Dynamics and Heat Transfer. American Society of Mechanical Engineers, New York, NY (2009)
ASME: ASME V&V 40–2018: Assessing Credibility of Computational Modeling through Verification and Validation: Application to Medical Devices. American Society of Mechanical Engineers, New York, NY (2018)
Bradley, C., Bowery, A., Britten, R., Budelmann, V., Camara, O., et al.: OpenCMISS: a multi-physics & multi-scale computational infrastructure for the VPH/Physiome project. Prog. Biophys. Mol. Biol. 107(1), 32–47 (2011)
Camara, O., et al.: Inter-model consistency and complementarity: learning from ex-vivo imaging and electrophysiological data towards an integrated understanding of cardiac physiology. Prog. Biophys. Mol. Biol. 107(1), 122–133 (2011)
Cox, R.W.: AFNI: what a long strange trip it’s been. NeuroImage 62(2), 743–747 (2012)
Cox, S.R., Lyall, D.M., Ritchie, S.J., Bastin, M.E., et al.: Associations between vascular risk factors and brain MRI indices in UK Biobank. Eur. Heart J. 40(28), 2290–2300 (2019)
Doste, R., Soto-Iglesias, D., Bernardino, G., Alcaine, A., Sebastian, R., et al.: A rule-based method to model myocardial fiber orientation in cardiac biventricular geometries with outflow tracts. Int. J. Numer. Meth. Biomed. Eng. 35(4), e3185 (2019)
Evans, A.C., Janke, A.L., Collins, D.L., Baillet, S.: Brain templates and atlases. NeuroImage 62(2), 911–922 (2012)
Fischl, B.: FreeSurfer. NeuroImage 62(2), 774–781 (2012)
Fonseca, C.G., Backhaus, M., Bluemke, D.A., Britten, R.D., et al.: The Cardiac Atlas Project-an imaging database for computational modeling and statistical atlases of the heart. Bioinformatics 27(16), 2288–2295 (2011)
Friston, K., Ashburner, J., Kiebel, S., Nichols, T., Penny, W.: Statistical Parametric Mapping: The Analysis of Functional Brain Images. Elsevier Academic Press, Amsterdam (2007)
Geffroy, D., Rivire, D., Denghien, I., Souedet, N., Laguitton, S., Cointepas, Y.: BrainVISA: a complete software platform for neuroimaging. In: Python in Neuroscience workshop, Paris, August 2011
Goldberger, A.L., Amaral, L.A.N., Glass, L., Hausdorff, J.M., et al.: PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation 101(23), e215–e220 (2000)
Gorgolewski, K.J., Auer, T., Calhoun, V.D., Craddock, R.C., Das, S., et al.: The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Sci. Data 3, 160044 EP (2016)
Gray, R.A., Pathmanathan, P.: Patient-specific cardiovascular computational modeling: diversity of personalization and challenges. J. Cardiovasc. Transl. Res. 11(2), 80–88 (2018)
Jenkinson, M., Beckmann, C.F., Behrens, T.E., Woolrich, M.W., Smith, S.M.: FSL. NeuroImage 62(2), 782–790 (2012)
Land, S., et al.: Verification of cardiac mechanics software: benchmark problems and solutions for testing active and passive material behaviour. Proc. Roy. Soc. London A: Math. Phys. Eng. Sci. 471(2184), 20150641 (2015)
Lee, A.W.C., Costa, C.M., Strocchi, M., Rinaldi, C.A., Niederer, S.A.: Computational modeling for cardiac resynchronization therapy. J. Cardiovasc. Transl. Res. 11(2), 92–108 (2018)
Maier-Hein, L., Eisenmann, M., Reinke, A., Onogur, S., Stankovic, M., et al.: Why rankings of biomedical image analysis competitions should be interpreted with care. Nat. Commun. 9(1), 5217 (2018)
Mirams, G.R., Arthurs, C.J., Bernabeu, M.O., Bordas, R., Cooper, J., et al.: Chaste: an open source C++ library for computational physiology and biology. PLOS Comput. Biol. 9(3), 1–8 (2013)
Niederer, S.A., Kerfoot, E., Benson, A.P., Bernabeu, M.O., Bernus, O., et al.: Verification of cardiac tissue electrophysiology simulators using an n-version benchmark. Philos. Trans. Ser. A Math. Phys. Eng. Sci. 369(1954), 4331–4351 (2011)
Nielsen, P.M., Le Grice, I.J., Smaill, B.H., Hunter, P.J.: Mathematical model of geometry and fibrous structure of the heart. Am. J. Physiol.-Heart C. Physiol. 260(4), H1365–H1378 (1991)
Parvinian, B., Pathmanathan, P., Daluwatte, C., Yaghouby, F., et al.: Credibility evidence for computational patient models used in the development of physiological closed-loop controlled devices for critical care medicine. Front. Physiol. 10, 220 (2019)
Pathmanathan, P., Cordeiro, J.M., Gray, R.A.: Comprehensive uncertainty quantification and sensitivity analysis for cardiac action potential models. Front. Physiol. 10, 721 (2019)
Pathmanathan, P., Gray, R.A.: Validation and trustworthiness of multiscale models of cardiac electrophysiology. Front. Physiol. 9, 106 (2018)
Pop, M., et al.: Fusion of optical imaging and MRI for the evaluation and adjustment of macroscopic models of cardiac electrophysiology: a feasibility study. Med. Image Anal. 13(2), 370–380 (2009)
Pop, M., et al.: Construction of 3D MR image-based computer models of pathologic hearts, augmented with histology and optical fluorescence imaging to characterize action potential propagation. Med. Image Anal. 16(2), 505–523 (2012)
Rigol, M., Solanes, N., Fernandez-Armenta, J., Silva, E., Doltra, A., et al.: Development of a swine model of left bundle branch block for experimental studies of cardiac resynchronization therapy. J. Cardiovasc. Transl. Res. 6(4), 616–622 (2013)
Samper-González, J., Burgos, N., Bottani, S., Fontanella, S., Lu, P., et al.: Reproducible evaluation of classification methods in Alzheimer’s disease: framework and application to mri and pet data. NeuroImage 183, 504–521 (2018)
Shepard, L.M., Sommer, K.N., Angel, E., Iyer, V., Wilson, M.F., et al.: Initial evaluation of three-dimensionally printed patient-specific coronary phantoms for CT-FFR software validation. J. Med. Imaging 6(2), 1–10 (2019)
Soto-Iglesias, D., Duchateau, N., Butakov, C.B.K., Andreu, D., et al.: Quantitative analysis of electro-anatomical maps: application to an experimental model of left bundle branch block/cardiac resynchronization therapy. IEEE J. Transl. Eng. Health Med. 5, 1–15 (2017)
Acknowledgments
Most of the organization of the CRT-EPiggy19 challenge has occurred during an academic visit of the author to the University of Auckland, which was partially funded by a Salvador de Madariaga fellowship by the Spanish Ministry of Science, Innovation and Universities and an expert visit grant from the European Union’s Horizon 2020 project EPIC (grant agreement No 687794). The CRT-EPiggy19 challenge is also partially funded by the Maria de Maeztu Units of Excellence Program (MDM-2015-0502) from the Spanish Ministry of Economy and Competitiveness of the Department of Information and Communication Technologies at the Universitat Pompeu Fabra, which is focused on data-driven knowledge extraction and promotes reproducible research and open science initiatives (https://www.upf.edu/web/mdm-dtic/reproducibility-in-research). I would like to thank all researchers participating in the challenge but also those who kindly explained me their justifiable reasons for not doing it. Special acknowledgments are given to all contributors of the CRT-EPiggy19 challenge, notably data collectors and clinical researchers (M. Sitges, A. Berruezo, M. Rigol, N. Solanes, A. Doltra, J. Fernández-Armenta), data curators (D. Soto, E. Silva, D. Andreu, C. Albors), data processors and scientific researchers (T. Mansi, E. Castañeda, B. Bijnens, G. Jiménez, N. Duchateau, J. Mill) and IT support (C. Yagüe) from Hospital Clínic de Barcelona, Siemens Healthineers and Universitat Pompeu Fabra. Finally, I would like to thank the anonymous reviewer of this paper for his fruitful comments and desire to initially reject it, which highly contributed to improve the manuscript.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Camara, O. (2020). Best (and Worst) Practices for Organizing a Challenge on Cardiac Biophysical Models During AI Summer: The CRT-EPiggy19 Challenge. In: Pop, M., et al. Statistical Atlases and Computational Models of the Heart. Multi-Sequence CMR Segmentation, CRT-EPiggy and LV Full Quantification Challenges. STACOM 2019. Lecture Notes in Computer Science(), vol 12009. Springer, Cham. https://doi.org/10.1007/978-3-030-39074-7_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-39074-7_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-39073-0
Online ISBN: 978-3-030-39074-7
eBook Packages: Computer ScienceComputer Science (R0)
