Abstract
We propose an unsupervised deep learning method to reconstruct a 3D tomographic image from biplanar X-rays, to reduce the number of required projections, the patient dose, and the acquisition time. To address this ill-posed problem, we introduce prior knowledge of anatomic structures by training a generative model on 3D CTs of head and neck. We optimize the latent vectors of the generative model to recover a volume that both integrates this prior knowledge and ensures consistency between the reconstructed image and input projections. Our method outperforms recent methods in terms of reconstruction error while being faster and less radiating than current clinical workflow. We evaluate our method in a clinical configuration for radiotherapy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Beichel, R.R., et al.: Data from QIN-HEADNECK (2015)
Ellis, S., et al.: Evaluation of 3D GANs for Lung Tissue Modelling in Pulmonary CT. arXiv (2022)
Feldkamp, L.A., Davis, L.C., Kress, J.W.: Practical cone-beam algorithm. J. Opt. Soc. Am. A-Opt. Image Sci. Vis. (1984)
Glocker, B., Komodakis, N., Tziritas, G., Navab, N., Paragios, N.: Dense image registration through MRFs and efficient linear programming. Med. Image Anal. 12(6), 731–741 (2008)
Goodfellow, I., et al.: Generative adversarial networks. Commun. ACM 63(11) (2020)
Grossberg, A., et al.: Anderson Cancer Center Head and Neck Quantitative Imaging Working Group. HNSCC (2020)
Henzler, P., Rasche, V., Ropinski, T., Ritschel, T.: Single-image tomography: 3D volumes from 2D cranial X-rays. In: Computer Graphics Forum (2018)
Herman, G.T.: Fundamentals of Computerized Tomography: Image Reconstruction from Projections. Springer, London (2009). https://doi.org/10.1007/978-1-84628-723-7
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: NeurIPS (2017)
Hong, S., et al.: 3D-StyleGAN: a style-based generative adversarial network for generative modeling of three-dimensional medical images. In: Engelhardt, S., et al. (eds.) DGM4MICCAI/DALI -2021. LNCS, vol. 13003, pp. 24–34. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88210-5_3
Hubbell, J.H.: Tables of X-Ray Mass Attenuation Coefficients 1 keV to 20 MeV for Elements Z=1 to 92 and 48 Additional Substance of Dosimetric Interest. NISTIR 5632 (1995)
Jiang, Y.: MFCT-GAN: multi-information network to reconstruct CT volumes for security screening. J. Intell. Manuf. Spec. Equip. 3(1), 17–30 (2022)
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., Aila, T.: Training generative adversarial networks with limited data. In: NeurIPS (2020)
Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: CVPR (2020)
Kinahan, P., Muzi, M., Bialecki, B., Coombs, L.: Data from the ACRIN 6685 Trial HNSCC-FDG-PET/CT (2020)
Kwan, J.Y.Y., et al.: Data from Radiomic Biomarkers to Refine Risk Models for Distant Metastasis in Oropharyngeal Carcinoma (2019)
Marinescu, R.V., Moyer, D., Golland, P.: Bayesian Image Reconstruction Using Deep Generative Models. arXiv (2020)
Mescheder, L., Geiger, A., Nowozin, S.: Which training methods for GANs do actually converge? In: ICML (2018)
Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. Commun. ACM 65(1) (2021)
Peng, C., Liao, H., Wong, G., Luo, J., Zhou, S.K., Chellappa, R.: XraySyn: realistic view synthesis from a single radiograph through CT priors. In: AAAI (2021)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Shen, L., Pauly, J., Xing, L.: NeRP: implicit neural representation learning with prior embedding for sparsely sampled image reconstruction. IEEE Trans. Neural Netw. (2022)
Shen, L., Zhao, W., Capaldi, D., Pauly, J., Xing, L.: A geometry-informed deep learning framework for ultra-sparse 3D tomographic image reconstruction. Comput. Biol. Med. 148, 105710 (2022)
Shen, L., Zhao, W., Xing, L.: Patient-specific reconstruction of volumetric computed tomography images from a single projection view via deep learning. Nature 3(11), 880–888 (2019)
Shibata, H., et al.: On the simulation of ultra-sparse-view and ultra-low-dose computed tomography with maximum a posteriori reconstruction using a progressive flow-based deep generative model. Tomography 8(5), 2129–2152 (2022)
Unberath, M., et al.: DeepDRR – a catalyst for machine learning in fluoroscopy-guided procedures. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 98–106. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_12
Vallières, M., et al.: Data from Head-Neck-PET-CT (2020)
Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers (2003)
Ying, X., Guo, H., Ma, K., Wu, J., Weng, Z., Zheng, Y.: X2CT-GAN: reconstructing CT from biplanar X-rays with generative adversarial networks. In: CVPR (2019)
Zha, R., Zhang, Y., Li, H.: NAF: neural attenuation fields for sparse-view CBCT reconstruction. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) MICCAI 2022. LNCS, vol. 13436, pp. 442–452. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16446-0_42
Zuley, M.L., et al.: The Cancer Genome Atlas Head-Neck Squamous Cell Carcinoma Collection TCGA-HNSC) (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Cafaro, A. et al. (2023). X2Vision: 3D CT Reconstruction from Biplanar X-Rays with Deep Structure Prior. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14229. Springer, Cham. https://doi.org/10.1007/978-3-031-43999-5_66
Download citation
DOI: https://doi.org/10.1007/978-3-031-43999-5_66
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43998-8
Online ISBN: 978-3-031-43999-5
eBook Packages: Computer ScienceComputer Science (R0)
