Abstract
Conditional Generative Adversarial Networks (cGANs) are a set of methods able to synthesize images that match a given condition. However, existing models designed for natural images are impractical to generate high-quality 3D medical images due to enormous computation. To address this issue, most cGAN models used in the medical field process either 2D slices or small 3D crops and join them together in subsequent steps to reconstruct the full-size 3D image. However, these approaches often cause spatial inconsistencies in adjacent slices or crops, and the changes specified by the target condition may not consider the 3D image as a whole. To address these problems, we propose a novel cGAN that can synthesize high-quality 3D MR images at different stages of the Alzheimer’s disease (AD). First, our method generates a sequence of 2D slices using an attention-based 2D generator with a disease condition for efficient transformations depending on brain regions. Then, consistency in 3D space is enforced by the use of a set of 2D and 3D discriminators. Moreover, we propose an adaptive identity loss based on the attention scores to properly transform features relevant to the target condition. Our experiments show that the proposed method can generate smooth and realistic 3D images at different stages of AD, and the image change with respect to the condition is better than the images generated by existing GAN-based methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ben-Cohen, A., Klang, E., Raskin, S.P., Amitai, M.M., Greenspan, H.: Virtual PET images from CT data using deep convolutional networks: initial results. In: Simulation and Synthesis in Medical Imaging, pp. 49–57 (2017)
Bińkowski, M., Sutherland, D., Arbel, M., Gretton, A.: Demystifying MMD GANs. ICML (2018)
Bowles, C., Gunn, R., Hammers, A., Rueckert, D.: Modelling the progression of Alzheimer’s disease in mri using generative adversarial networks. SPIE Medical Imaging, p. 55 (2018)
Choi, H., Kang, H., Lee, D.S., T.A.D.N.I.: Predicting aging of brain metabolic topography using variational autoencoder. Front. Aging Neurosc. 10, 212 (2018)
Choi, Y., Choi, M.J., Kim, M., Ha, J.W., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)
Gretton, A., Borgwardt, K.M., Rasch, M.J., Schölkopf, B., Smola, A.: A kernel two-sample test. JMLR (2012)
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved Training of Wasserstein GANs. In: NIPS, pp. 5767–5777 (2017)
He, Z., Zuo, W., Kan, M., Shan, S., Chen, X.: Attgan: facial attribute editing by only changing what you want. IEEE Trans. Image Process. 28(11), 5464–5478 (2019)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: NIPS (2017)
Hinton, G., Salakhutdinov, R.: Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006)
Jack, C., et al.: The Alzheimer’s Disease Neuroimaging Initiative (ADNI): MRI methods. J. Magn. Reson. Imaging 27(4), 685–691 (2008)
Jung, E., Luna, M., Park, S.H.: Conditional generative adversarial network for predicting 3d medical images affected by alzheimer’s diseases. In: International Workshop on PRedictive Intelligence in MEdicine, pp. 79–90 (2020)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: 3rd International Conference for Learning Representations (2014)
LaMontagne, P.J., et al.: Marcus, D.: Oasis-3: Longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and alzheimer disease. medRxiv (2019)
Lei, Y., et al.: MRI-only based synthetic CT generation using dense cycle consistent generative adversarial networks. Med. Phys. 46(8), 3565–3581 (2019)
Muhammad, S., Muhammad, Naveed, R., Jing, W., Chengnian, L., Shaoyuan, L.: Unpaired multi-contrast MR image synthesis using generative adversarial networks. In: Simulation and Synthesis in Medical Imaging. pp. 22–31. Springer International Publishing (2019). https://doi.org/10.1007/978-3-030-32778-1_3
Pan, Y., Liu, M., Lian, C., Zhou, T., Xia, Y., Shen, D.: Synthesizing missing PET from MRI with cycle-consistent generative adversarial networks for Alzheimer’s disease diagnosis. In: Medical Image Computing and Computer Assisted Intervention, pp. 455–463 (2018)
Prokopenko, D., Stadelmann, J., Schulz, H., Renisch, S., Dylov, D.: Synthetic CT generation from MRI using improved dualgan. arXiv:1909.08942 (2019)
Pumarola, A., Agudo, A., Martinez, A., Sanfeliu, A., Moreno-Noguer, F.: GANimation: one-shot anatomically consistent facial animation. In: International Journal of Computer Vision (IJCV) (2019)
Ravi, D., Alexander, D.C., Oxtoby, N.P.: Degenerative adversarial neuroimage nets: generating images that mimic disease progression. In: Medical Image Computing and Computer Assisted Intervention, pp. 164–172 (2019)
Reitz, C.: Toward precision medicine in Alzheimer’s disease. Ann. Transl. Med. 4(6), 107 (2016)
Roychowdhury, S., Roychowdhury, S.: A modular framework to predict alzheimer’s disease progression using conditional generative adversarial networks. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2020)
Salman, Ul, H.D., Mahmut, Y., Levent, K., Aykut, E., Erkut, E., Tolga, C.: Image synthesis in multi-contrast mri with conditional generative adversarial networks. IEEE Trans. Med. Imaging 38(10), 2375–2388 (2019)
Shmelkov, K., Schmid, C., Alahari, K.: How good is my gan? The European Conference on Computer Vision (2018)
Wegmayr, V., Horold, M., Buhmann, J.: Generative aging of brain mri for early prediction of mci-ad conversion. In: International Symposium on Biomedical Imaging, pp. 1042–1046 (2019)
Welander, P., Karlsson, S., Eklund, A.: Generative adversarial networks for image-to-image translation on multi-contrast mr images - a comparison of cyclegan and unit. arXiv:1806.07777 (2018)
Wolterink, J.M., Dinkla, A.M., Savenije, M.H.F., Seevinck, P.R., van den Berg, C.A.T., Išgum, I.: Deep MR to CT synthesis using unpaired data. In: Simulation and Synthesis in Medical Imaging, pp. 14–23 (2017)
Zeng, G., Zheng, G.: Hybrid generative adversarial networks for deep MR to CT synthesis using unpaired data. Medical Image Computing and Computer Assisted Intervention, pp. 59–767 (2019)
Zhang, Z., Song, Y., Qi, H.: Age progression/regression by conditional adversarial autoencoder. CVPR (2017)
Zhao, Q., Adeli, E., Honnorat, N., Leng, T., Pohl, K.M.: Variational autoencoder for regression: Application to brain aging analysis. In: Medical Image Computing and Computer Assisted Intervention, pp. 823–831 (2019)
Acknowledgement
This research was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean Government (MSIT) (No. 2019R1C1C1008727)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Jung, E., Luna, M., Park, S.H. (2021). Conditional GAN with an Attention-Based Generator and a 3D Discriminator for 3D Medical Image Generation. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12906. Springer, Cham. https://doi.org/10.1007/978-3-030-87231-1_31
Download citation
DOI: https://doi.org/10.1007/978-3-030-87231-1_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87230-4
Online ISBN: 978-3-030-87231-1
eBook Packages: Computer ScienceComputer Science (R0)
