Abstract
Segmentation of Multiple Sclerosis (MS) lesions in longitudinal brain MR scans is performed for monitoring the progression of MS lesions. We hypothesize that the spatio-temporal cues in longitudinal data can aid the segmentation algorithm. Therefore, we propose a multi-task learning approach by defining an auxiliary self-supervised task of deformable registration between two time-points to guide the neural network toward learning from spatio-temporal changes. We show the efficacy of our method on a clinical dataset comprised of 70 patients with one follow-up study for each patient. Our results show that spatio-temporal information in longitudinal data is a beneficial cue for improving segmentation. We improve the result of current state-of-the-art by 2.6% in terms of overall score (p < 0.05). Code is publicly available (https://github.com/StefanDenn3r/Spatio-temporal-MS-Lesion-Segmentation).
S. Denner and A. Khakzar—First two authors contributed equally to this work.
S. T. Kim and N. Navab—Share senior authorship.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Andermatt, S., Pezold, S., Cattin, P.C.: Automated segmentation of multiple sclerosis lesions using multi-dimensional gated recurrent units. In: Crimi, A., Bakas, S., Kuijf, H., Menze, B., Reyes, M. (eds.) BrainLes 2017. LNCS, vol. 10670, pp. 31–42. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75238-9_3
Aslani, S., Dayan, M., Storelli, L., Filippi, M., Murino, V., Rocca, M.A., Sona, D.: Multi-branch convolutional neural network for multiple sclerosis lesion segmentation. NeuroImage 196, 1–15 (2019)
Balakrishnan, G., Zhao, A., Sabuncu, M.R., Dalca, A.V., Guttag, J.: An unsupervised learning model for deformable medical image registration. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2018). https://doi.org/10.1109/CVPR.2018.00964
Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: VoxelMorph: a learning framework for deformable medical image registration. IEEE Trans. Med. Imaging (2019). https://doi.org/10.1109/TMI.2019.2897538
Birenbaum, A., Greenspan, H.: Longitudinal multiple sclerosis lesion segmentation using multi-view convolutional neural networks. In: Carneiro, G., et al. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 58–67. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46976-8_7
Carass, A., et al.: Longitudinal multiple sclerosis lesion segmentation: resource and challenge. NeuroImage 148, 77–102 (2017)
Cardoso, M.J., et al.: Geodesic information flows: spatially-variant graphs and their application to segmentation and fusion. IEEE Trans. Med. Imaging 34(9), 1976–1988 (2015)
Chen, Z., Badrinarayanan, V., Lee, C.Y., Rabinovich, A.: GradNorm: gradient normalization for adaptive loss balancing in deep multitask networks. In: 35th International Conference on Machine Learning, ICML 2018 (2018)
Compston, A., Coles, A.: Multiple sclerosis (2008). https://doi.org/10.1016/S0140-6736(08)61620-7
Galimzianova, A., Pernuš, F., Likar, B., Špiclin, Ž: Stratified mixture modeling for segmentation of white-matter lesions in brain MR images. NeuroImage 124, 1031–1043 (2016)
Ghafoorian, M., Platel, B.: Convolutional neural networks for MS lesion segmentation, method description of diag team. In: Proceedings of the 2015 Longitudinal Multiple Sclerosis Lesion Segmentation Challenge, pp. 1–2 (2015)
Hashemi, S.R., Salehi, S.S.M., Erdogmus, D., Prabhu, S.P., Warfield, S.K., Gholipour, A.: Asymmetric loss functions and deep densely-connected networks for highly-imbalanced medical image segmentation: application to multiple sclerosis lesion detection. IEEE Access 7, 1721–1735 (2018)
Jégou, S., Drozdzal, M., Vazquez, D., Romero, A., Bengio, Y.: The one hundred layers tiramisu: fully convolutional densenets for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 11–19 (2017)
Klein, S., Staring, M., Murphy, K., Viergever, M.A., Pluim, J.P.W.: elastix: a toolbox for intensity-based medical image registration. IEEE Trans. Med. Imaging 29(1), 196–205 (2010)
Lesjak, Ž, et al.: A novel public MR image dataset of multiple sclerosis patients with lesion segmentations based on multi-rater consensus. Neuroinformatics 16(1), 51–63 (2018)
Manjón, J.V., Coupé, P., Buades, A., Louis Collins, D., Robles, M.: New methods for MRI denoising based on sparseness and self-similarity. Med. Image Anal. 16(1), 18–27 (2012). https://doi.org/10.1016/j.media.2011.04.003. http://www.sciencedirect.com/science/article/pii/S1361841511000491
Paszke, A., et al.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, pp. 8024–8035 (2019)
Reddi, S.J., Kale, S., Kumar, S.: On the convergence of Adam and beyond (2018)
Roth, H.R., et al.: A new 2.5D representation for lymph node detection using random sets of deep convolutional neural network observations. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8673, pp. 520–527. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10404-1_65
Roy, A.G., Conjeti, S., Navab, N., Wachinger, C., Initiative, A.D.N., et al.: QuickNAT: a fully convolutional network for quick and accurate segmentation of neuroanatomy. NeuroImage 186, 713–727 (2019)
Stangel, M., Penner, I.K., Kallmann, B.A., Lukas, C., Kieseier, B.C.: Towards the implementation of ‘no evidence of disease activity’ in multiple sclerosis treatment: the multiple sclerosis decision model. Therap. Adv. Neurol. Disord. 8(1), 3–13 (2015)
Steinman, L.: Multiple sclerosis: A coordinated immunological attack against myelin in the central nervous system (1996). https://doi.org/10.1016/S0092-8674(00)81107-1
Styner, M., et al.: 3D segmentation in the clinic: a grand challenge II: MS lesion segmentation. Midas J. 2008, 1–6 (2008)
Tustison, N.J., et al.: N4ITK: improved N3 bias correction. IEEE Trans. Med. Imaging 29(6), 1310–1320 (2010)
Uher, T., et al.: Combining clinical and magnetic resonance imaging markers enhances prediction of 12-year disability in multiple sclerosis. Multiple Sclerosis 23(1), 51–61 (2017)
Valverde, S., et al.: Improving automated multiple sclerosis lesion segmentation with a cascaded 3D convolutional neural network approach. NeuroImage 155, 159–168 (2017)
Wachinger, C., Reuter, M., Klein, T.: DeepNAT: deep convolutional neural network for segmenting neuroanatomy. NeuroImage 170, 434–445 (2018)
Xu, Z., Niethammer, M.: DeepAtlas: joint semi-supervised learning of image registration and segmentation. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11765, pp. 420–429. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32245-8_47
Zhang, H., et al.: Multiple sclerosis lesion segmentation with tiramisu and 2.5D stacked slices. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11766, pp. 338–346. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32248-9_38
Acknowledgements
The authors acknowledge the financial support for this work by Siemens Healthineers and Munich Center for Machine Learning (MCML). Ziga Spiclin was supported by the Slovenian Research Agency (research core funding No. P2-0232, and research grant No. J2-2500).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Denner, S. et al. (2021). Spatio-Temporal Learning from Longitudinal Data for Multiple Sclerosis Lesion Segmentation. In: Crimi, A., Bakas, S. (eds) Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2020. Lecture Notes in Computer Science(), vol 12658. Springer, Cham. https://doi.org/10.1007/978-3-030-72084-1_11
Download citation
DOI: https://doi.org/10.1007/978-3-030-72084-1_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-72083-4
Online ISBN: 978-3-030-72084-1
eBook Packages: Computer ScienceComputer Science (R0)
