Abstract
Longitudinal magnetic resonance imaging (MRI) is essential in neuroimaging studies of early brain development. However, incomplete data is an inevitable problem in longitudinal studies because of participant attrition and scan failure. Data imputation is a possible way to address such missing data. Here, we propose a novel 3D multi-modal perceptual adversarial network (MPGAN) to predict a missing MR image from an existing longitudinal image of the same subject. To the best of our knowledge, this is the first application of deep generative methods for longitudinal image prediction of structural MRI in the first year of life, where brain volume and image intensities are changing dramatically. In order to produce sharper and more realistic images, we incorporate the perceptual loss into the adversarial training process. To leverage complementary information contained in the multi-modality data, MPGAN predicts T1w and T2w images jointly in the prediction process. We evaluated MPGAN versus six alternative approaches based on visual as well as quantitative assessment. The results indicate that our MPGAN predicts missing MR images in an accurate and visually realistic fashion, and shows better performance than the alternative methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Gilmore, J.H., et al.: Imaging structural and functional brain development in early childhood. Nat. Rev. Neurosci. 19(3), 123–137 (2018)
Hazlett, H.C., et al.: Early brain development in infants at high risk for autism spectrum disorder. Nature 542(7641), 348–351 (2017)
Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)
Wolterink, J.M., Dinkla, A.M., Savenije, M.H.F., Seevinck, P.R., van den Berg, C.A.T., Išgum, I.: Deep MR to CT synthesis using unpaired data. In: Tsaftaris, S.A., Gooya, A., Frangi, A.F., Prince, J.L. (eds.) SASHIMI 2017. LNCS, vol. 10557, pp. 14–23. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68127-6_2
Pan, Y., Liu, M., Lian, C., Zhou, T., Xia, Y., Shen, D.: Synthesizing missing PET from MRI with cycle-consistent generative adversarial networks for Alzheimer’s disease Diagnosis. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11072, pp. 455–463. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00931-1_52
Qu, L., Wang, S., Yap, P.-T., Shen, D.: Wavelet-based semi-supervised adversarial learning for synthesizing realistic 7T from 3T MRI. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11767, pp. 786–794. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32251-9_86
Zhao, F., et al.: Harmonization of infant cortical thickness using surface-to-surface cycle-consistent adversarial networks. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11767, pp. 475–483. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32251-9_52
Xia, T., Chartsias, A., Tsaftaris, S.A.: Consistent brain ageing synthesis. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11767, pp. 750–758. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32251-9_82
Ravi, D., Alexander, D.C., Oxtoby, N.P.: Degenerative adversarial NeuroImage nets: generating images that mimic disease progression. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11766, pp. 164–172. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32248-9_19
Pathak, D., et al.: Context encoders: feature learning by inpainting. In: CVPR, pp. 2536–2544 (2016)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
Zhou, Z., et al.: Models genesis: generic autodidactic models for 3D medical image analysis. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11767, pp. 384–393. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32251-9_42
Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 424–432. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46723-8_49
Zhu, J.Y., et al.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: ICCV, pp. 2223–2232 (2017)
Wang, Z., et al.: Mean squared error: love it or leave it? A new look at signal fidelity measures. Signal Process. Mag. 26(1), 98–117 (2009)
Huynh-Thu, Q., et al.: Scope of validity of PSNR in image/video quality assessment. Electron. Lett. 44(13), 800–801 (2008)
Wang, J., et al.: Multi-atlas segmentation of subcortical brain structures via the AutoSeg software pipeline. Front. Neuroinformatics 8, 7 (2014)
Zhang, R., et al.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR, pp. 586–595 (2018)
Simonyan K., et al.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)
Acknowledgement
This study was supported by grants from the Major Scientific Project of Zhejiang Lab (No. 2018DG0ZX01), the National Institutes of Health (R01-HD055741, T32-HD040127, U54-HD079124, U54-HD086984, R01-EB021391), Autism Speaks, and the Simons Foundation (140209). MDS is supported by a U.S. National Institutes of Health (NIH) career development award (K12-HD001441). The sponsors had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Peng, L. et al. (2020). Multi-modal Perceptual Adversarial Learning for Longitudinal Prediction of Infant MR Images. In: Hu, Y., et al. Medical Ultrasound, and Preterm, Perinatal and Paediatric Image Analysis. ASMUS PIPPI 2020 2020. Lecture Notes in Computer Science(), vol 12437. Springer, Cham. https://doi.org/10.1007/978-3-030-60334-2_28
Download citation
DOI: https://doi.org/10.1007/978-3-030-60334-2_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-60333-5
Online ISBN: 978-3-030-60334-2
eBook Packages: Computer ScienceComputer Science (R0)
