Skip to main content

Advertisement

SpringerLink
Log in

Validation of White-Matter Lesion Change Detection Methods on a Novel Publicly Available MRI Image Database

  • Original Article
  • Published:
Neuroinformatics Aims and scope Submit manuscript

Abstract

Changes of white-matter lesions (WMLs) are good predictors of the progression of neurodegenerative diseases like multiple sclerosis (MS). Based on longitudinal magnetic resonance (MR) imaging the changes can be monitored, while the need for their accurate and reliable quantification led to the development of several automated MR image analysis methods. However, an objective comparison of the methods is difficult, because publicly unavailable validation datasets with ground truth and different sets of performance metrics were used. In this study, we acquired longitudinal MR datasets of 20 MS patients, in which brain regions were extracted, spatially aligned and intensity normalized. Two expert raters then delineated and jointly revised the WML changes on subtracted baseline and follow-up MR images to obtain ground truth WML segmentations. The main contribution of this paper is an objective, quantitative and systematic evaluation of two unsupervised and one supervised intensity based change detection method on the publicly available datasets with ground truth segmentations, using common pre- and post-processing steps and common evaluation metrics. Besides, different combinations of the two main steps of the studied change detection methods, i.e. dissimilarity map construction and its segmentation, were tested to identify the best performing combination.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Avants, B. B., Tustison, N. J., Wu, J., Cook, P. A., & Gee, J. C. (2011). An open source multivariate framework for n-tissue segmentation with evaluation on public data. Neuroinformatics, 9(4), 381–400. doi:10.1007/s12021-011-9109-y.

    Article  PubMed  PubMed Central  Google Scholar 

  • Avants, B. B., Tustison, N. J., Stauffer, M., Song, G., Wu, B., & Gee, J. C. (2014). The Insight ToolKit image registration framework. Frontiers in Neuroinformatics, 8. doi:10.3389/fninf.2014.00044.

  • Avants, B. B., Tustison, N. J., & Johnson, H. J. (n.d.). Advanced normalization tools (ANTs). http://stnava.github.io/ANTs/. Accessed 16 Mar 2016.

  • Battaglini, M., Rossi, F., Grove, R. A., Stromillo, M. L., Whitcher, B., Matthews, P. M., & De Stefano, N. (2014). Automated identification of brain new lesions in multiple sclerosis using subtraction images. Journal of Magnetic Resonance Imaging, 39(6), 1543–1549. doi:10.1002/jmri.24293.

    Article  PubMed  Google Scholar 

  • Bosc, M., Heitz, F., Armspach, J. P., Namer, I., Gounot, D., & Rumbach, L. (2003). Automatic change detection in multimodal serial MRI: application to multiple sclerosis lesion evolution. NeuroImage, 20(2), 643–656. doi:10.1016/S1053-8119(03)00406-3.

    Article  PubMed  Google Scholar 

  • Cocosco, C. A., Kollokian, V., Kwan, R. K.-S., Pike, G. B., & Evans, A. C. (1997). BrainWeb: online interface to a 3D MRI simulated brain database. NeuroImage, 5, 425.

    Google Scholar 

  • Diez, Y., Oliver, A., Cabezas, M., Valverde, S., Martí, R., Vilanova, J. C., et al. (2013). Intensity based methods for brain MRI longitudinal registration. a study on multiple sclerosis patients. Neuroinformatics, 12(3), 365–379. doi:10.1007/s12021-013-9216-z.

    Article  Google Scholar 

  • Duan, Y., Hildenbrand, P. G., Sampat, M. P., Tate, D. F., Csapo, I., Moraal, B., et al. (2008). Segmentation of subtraction images for the measurement of lesion change in multiple sclerosis. AJNR. American Journal of Neuroradiology, 29(2), 340–346. doi:10.3174/ajnr.A0795.

    Article  CAS  PubMed  Google Scholar 

  • Elliott, C., Arnold, D. L., Collins, D. L., & Arbel, T. (2013). Temporally consistent probabilistic detection of new multiple sclerosis lesions in brain MRI. IEEE Transactions on Medical Imaging, 32(8), 1490–1503. doi:10.1109/TMI.2013.2258403.

    Article  PubMed  Google Scholar 

  • Ganiler, O., Oliver, A., Diez, Y., Freixenet, J., Vilanova, J. C., Beltran, B., et al. (2014). A subtraction pipeline for automatic detection of new appearing multiple sclerosis lesions in longitudinal studies. Neuroradiology, 56(5), 363–374. doi:10.1007/s00234-014-1343-1.

    Article  PubMed  Google Scholar 

  • García-Lorenzo, D., Francis, S., Narayanan, S., Arnold, D. L., & Collins, D. L. (2013). Review of automatic segmentation methods of multiple sclerosis white matter lesions on conventional magnetic resonance imaging. Medical Image Analysis, 17(1), 1–18. doi:10.1016/j.media.2012.09.004.

    Article  PubMed  Google Scholar 

  • Ge, Y. (2006). Multiple sclerosis: the role of MR imaging. American Journal of Neuroradiology, 27(6), 1165–1176.

    CAS  PubMed  Google Scholar 

  • Giorgio, A., Stromillo, M. L., Bartolozzi, M. L., Rossi, F., Battaglini, M., De Leucio, A., … & Amato, M. P. (2014). Relevance of hypointense brain MRI lesions for long-term worsening of clinical disability in relapsing multiple sclerosis. Multiple Sclerosis Journal, 20(2), 214–219.

  • Goldberg‐Zimring, D., Achiron, A., Guttmann, C. R., & Azhari, H. (2003). Three‐dimensional analysis of the geometry of individual multiple sclerosis lesions: detection of shape changes over time using spherical harmonics. Journal of Magnetic Resonance Imaging, 18(3), 291–301.

    Article  PubMed  Google Scholar 

  • Klein, A., Andersson, J., Ardekani, B. A., Ashburner, J., Avants, B., Chiang, M. C., … & Song, J. H. (2009). Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. Neuroimage, 46(3), 786–802.

  • Lebrun, C., Bensa, C., Debouverie, M., De Seze, J., Wiertlievski, S., Brochet, B., et al. (2008). Unexpected multiple sclerosis: follow-up of 30 patients with magnetic resonance imaging and clinical conversion profile. Journal of Neurology, Neurosurgery, and Psychiatry, 79(2), 195–198. doi:10.1136/jnnp.2006.108274.

    Article  CAS  PubMed  Google Scholar 

  • Llado, X., Oliver, A., Cabezas, M., Freixenet, J., Vilanova, J. C., Quiles, A., et al. (2012). Segmentation of multiple sclerosis lesions in brain MRI: a review of automated approaches. Information Sciences, 186(1), 164–185. doi:10.1016/j.ins.2011.10.011.

    Article  Google Scholar 

  • Lladó, X., Ganiler, O., Oliver, A., Martí, R., Freixenet, J., Valls, L., et al. (2012). Automated detection of multiple sclerosis lesions in serial brain MRI. Neuroradiology, 54(8), 787–807. doi:10.1007/s00234-011-0992-6.

    Article  PubMed  Google Scholar 

  • Maes, F., Collignon, A., Vandermeulen, D., Marchal, G., & Suetens, P. (1997). Multimodality image registration by maximization of mutual information. IEEE Transactions on Medical Imaging, 16(2), 187–198. doi:10.1109/42.563664.

    Article  CAS  PubMed  Google Scholar 

  • Moraal, B., Meier, D. S., Poppe, P. A., Geurts, J. J. G., Vrenken, H., Jonker, W. M. A., et al. (2009). Subtraction MR images in a multiple sclerosis multicenter clinical trial setting. Radiology, 250(2), 506–514. doi:10.1148/radiol.2501080480.

    Article  PubMed  Google Scholar 

  • Moraal, B., van den Elskamp, I. J., Knol, D. L., Uitdehaag, B. M. J., Geurts, J. J. G., Vrenken, H., et al. (2010a). Long-interval T2-weighted subtraction magnetic resonance imaging: a powerful new outcome measure in multiple sclerosis trials. Annals of Neurology, 67(5), 667–675. doi:10.1002/ana.21958.

    PubMed  Google Scholar 

  • Moraal, B., Wattjes, M. P., Geurts, J. J. G., Knol, D. L., van Schijndel, R. A., Pouwels, P. J. W., et al. (2010b). Improved detection of active multiple sclerosis lesions: 3D subtraction imaging. Radiology, 255(1), 154–163. doi:10.1148/radiol.09090814.

    Article  PubMed  Google Scholar 

  • Nika, V., Babyn, P., & Zhu, H. (2014). EigenBlock algorithm for change detection – an application of adaptive dictionary learning techniques. Journal of Computational Science, 5(3), 527–535. doi:10.1016/j.jocs.2013.10.008.

    Article  Google Scholar 

  • Patriarche, J., & Erickson, B. (2004). A review of the automated detection of change in serial imaging studies of the brain. Journal of Digital Imaging, 17(3), 158–174. doi:10.1007/s10278-004-1010-x.

    Article  PubMed  PubMed Central  Google Scholar 

  • Patti, F., De Stefano, M., Lavorgna, L., Messina, S., Chisari, C. G., Ippolito, D., et al. (2015). Lesion load may predict long-term cognitive dysfunction in multiple sclerosis patients. PloS One, 10(3), e0120754. doi:10.1371/journal.pone.0120754.

    Article  PubMed  PubMed Central  Google Scholar 

  • Pham, D. (n.d.). Longitudinal Multiple Sclerosis Lesion Segmentation Challenge | ISBI 2015. http://biomedicalimaging.org/2015/3d-segmentation-of-neurites-in-em-images/. Accessed 10 Apr 2015.

  • Popescu, V., Agosta, F., Hulst, H. E., Sluimer, I. C., Knol, D. L., Sormani, M. P., et al. (2013). Brain atrophy and lesion load predict long term disability in multiple sclerosis. Journal of Neurology, Neurosurgery, and Psychiatry, 84(10), 1082–1091. doi:10.1136/jnnp-2012-304094.

    Article  PubMed  Google Scholar 

  • Ramirez, J., McNeely, A. A., Scott, C. J., Stuss, D. T., & Black, S. E. (2014). Subcortical hyperintensity volumetrics in Alzheimer’s disease and normal elderly in the Sunnybrook Dementia Study: correlations with atrophy, executive function, mental processing speed, and verbal memory. Alzheimer’s Research & Therapy, 6(4), 49. doi:10.1186/alzrt279.

    Article  Google Scholar 

  • Rey, D., Subsol, G., Delingette, H., & Ayache, N. (2002). Automatic detection and segmentation of evolving processes in 3D medical images: application to multiple sclerosis. Medical Image Analysis, 6(2), 163–179. doi:10.1016/S1361-8415(02)00056-7.

    Article  PubMed  Google Scholar 

  • Risacher, S. L., Saykin, A. J., West, J. D., Shen, L., Firpi, H. A., McDonald, B. C., & Alzheimer’s Disease Neuroimaging Initiative (ADNI). (2009). Baseline MRI predictors of conversion from MCI to probable AD in the ADNI cohort. Current Alzheimer Research, 6(4), 347–361.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Rocca, M. A., Anzalone, N., Falini, A., & Filippi, M. (2013). Contribution of magnetic resonance imaging to the diagnosis and monitoring of multiple sclerosis. La Radiologia Medica, 118(2), 251–264. doi:10.1007/s11547-012-0811-3.

    Article  CAS  PubMed  Google Scholar 

  • Roura, E., Oliver, A., Cabezas, M., Vilanova, J. C., Rovira, À., Ramió-Torrentà, L., & Lladó, X. (2014). MARGA: multispectral adaptive region growing algorithm for brain extraction on axial MRI. Computer Methods and Programs in Biomedicine, 113(2), 655–673. doi:10.1016/j.cmpb.2013.11.015.

    Article  PubMed  Google Scholar 

  • Rousseau, F., Faisan, S., Heitz, F., Armspach, J.-P., Chevalier, Y., & Blanc, F., et al. (2007). An A Contrario Approach for Change Detection in 3D Multimodal Images: Application to Multiple Sclerosis in MRI. In 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2007. EMBS 2007 (pp. 2069–2072). Presented at the 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2007. EMBS 2007. doi:10.1109/IEMBS.2007.4352728.

  • Rovira, À., Wattjes, M. P., Tintoré, M., Tur, C., Yousry, T. A., Sormani, M. P., … & Barkhof, F. (2015). Evidence-based guidelines: MAGNIMS consensus guidelines on the use of MRI in multiple sclerosis – clinical implementation in the diagnostic process. Nature Reviews Neurology.

  • Seo, H. J., & Milanfar, P. (2009). A non-parametric approach to automatic change detection in MRI images of the brain. In IEEE International Symposium on Biomedical Imaging: From Nano to Macro, 2009. ISBI ’09 (pp. 245–248). Presented at the IEEE International Symposium on Biomedical Imaging: From Nano to Macro, 2009. ISBI ’09. doi:10.1109/ISBI.2009.5193029.

  • Shah, M., Xiao, Y., Subbanna, N., Francis, S., Arnold, D. L., Collins, D. L., & Arbel, T. (2011). Evaluating intensity normalization on MRIs of human brain with multiple sclerosis. Medical Image Analysis, 15(2), 267–282. doi:10.1016/j.media.2010.12.003.

    Article  PubMed  Google Scholar 

  • Shinohara, R. T., Sweeney, E. M., Goldsmith, J., Shiee, N., Mateen, F. J., Calabresi, P. A., et al. (2014). Statistical normalization techniques for magnetic resonance imaging. NeuroImage Clinical, 6, 9–19. doi:10.1016/j.nicl.2014.08.008.

    Article  PubMed  PubMed Central  Google Scholar 

  • Simoes, R., & Slump, C. (2011). Change detection and classification in brain MR images using change vector analysis. In 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society,EMBC (pp. 7803–7807). Presented at the 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society,EMBC. doi:10.1109/IEMBS.2011.6091923.

  • Smith, S. M. (2002). Fast robust automated brain extraction. Human Brain Mapping, 17(3), 143–155. doi:10.1002/hbm.10062.

    Article  PubMed  Google Scholar 

  • Studholme, C., Drapaca, C., Iordanova, B., & Cardenas, V. (2006). Deformation-based mapping of volume change from serial brain MRI in the presence of local tissue contrast change. IEEE Transactions on Medical Imaging, 25(5), 626–639. doi:10.1109/TMI.2006.872745.

    Article  PubMed  Google Scholar 

  • Styner, M., Lee, J., Chin, B., Chin, M., Commowick, O., Tran, H., et al. (2008). 3D segmentation in the clinic: a grand challenge II: MS lesion segmentation. MIDAS Journal, 1–6. Accessed 9 Mar 2015.

  • Susanto, T. A. K., Pua, E. P. K., & Zhou, J. (2015). Cognition, brain atrophy, and cerebrospinal fluid biomarkers changes from preclinical to dementia stage of Alzheimer’s disease and the influence of apolipoprotein e. Journal of Alzheimer’s Disease, 45(1), 253–268. doi:10.3233/JAD-142451.

    CAS  PubMed  Google Scholar 

  • Sweeney, E. M., Shinohara, R. T., Shea, C. D., Reich, D. S., & Crainiceanu, C. M. (2013). Automatic lesion incidence estimation and detection in multiple sclerosis using multisequence longitudinal MRI. American Journal of Neuroradiology, 34(1), 68–73. doi:10.3174/ajnr.A3172.

    Article  CAS  PubMed  Google Scholar 

  • Tustison, N. J., Avants, B. B., Cook, P. A., Zheng, Y., Egan, A., Yushkevich, P. A., & Gee, J. C. (2010). N4ITK: improved N3 bias correction. IEEE Transactions on Medical Imaging, 29(6), 1310–1320. doi:10.1109/TMI.2010.2046908.

    Article  PubMed  PubMed Central  Google Scholar 

  • Vrenken, H., Jenkinson, M., Horsfield, M. A., Battaglini, M., van Schijndel, R. A., Rostrup, E., et al. (2013). Recommendations to improve imaging and analysis of brain lesion load and atrophy in longitudinal studies of multiple sclerosis. Journal of Neurology, 260(10), 2458–2471. doi:10.1007/s00415-012-6762-5.

    Article  CAS  PubMed  Google Scholar 

  • Wang, H., Das, S. R., Suh, J. W., Altinay, M., Pluta, J., Craige, C., … & Alzheimer’s Disease Neuroimaging Initiative (2011). A learning-based wrapper method to correct systematic errors in automatic image segmentation: consistently improved performance in hippocampus, cortex and brain segmentation. NeuroImage, 55(3), 968–985.

  • Wei, X., Guttmann, C. R. G., Warfield, S. K., Eliasziw, M., & Mitchell, J. R. (2004). Has your patient’s multiple sclerosis lesion burden or brain atrophy actually changed? Multiple Sclerosis (Houndmills, Basingstoke, England), 10(4), 402–406.

    Article  Google Scholar 

Download references

Acknowledgments

This research was supported by the Ministry of Education, Science and Sport, Slovenia, under grants J2-5473, L2-5472, and J7-6781. The authors would also like to acknowledge A.K. and M.L. from the University Medical Centre Ljubljana for creating the reference segmentations.

Author information

Authors and Affiliations

  1. University of Ljubljana, Faculty of Electrical Engineering, Tržaška 25, 1000, Ljubljana, Slovenia

    Žiga Lesjak, Franjo Pernuš, Boštjan Likar & Žiga Špiclin

  2. Computer Vision Systems, Sensum, Tehnološki Park 21, 1000, Ljubljana, Slovenia

    Boštjan Likar & Žiga Špiclin

Authors
  1. Žiga Lesjak
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Franjo Pernuš
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Boštjan Likar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Žiga Špiclin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Žiga Lesjak.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lesjak, Ž., Pernuš, F., Likar, B. et al. Validation of White-Matter Lesion Change Detection Methods on a Novel Publicly Available MRI Image Database. Neuroinform 14, 403–420 (2016). https://doi.org/10.1007/s12021-016-9301-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12021-016-9301-1

Keywords

Search

Navigation