Abstract
In this paper, we propose a deep neural network (DNN)-based volumetric image reconstruction framework, enabling an end-to-end volume inference from a 2D X-ray image. The proposed framework is built upon the vector quantised-variational autoencoders of the 2D X-ray images and the 3D cone-beam CT (CBCT) images. We present a bi-directional cross-modal transition module between the discrete latent variables of the 2D and 3D images. In the X-ray-to-CBCT pass, the transition module takes the feature maps of the X-ray images as an input and outputs the discrete volumetric embedding, which is combined with the codebook in the embedding space for the reconstruction of CBCT images. On the other hand, the CBCT-to-X-ray pass realizes the DNN-based volume rendering. Given real clinically-obtained X-ray images without paired volumetric images, we devise an unsupervised learning scheme to optimize the discrete embedding transition module. Our approach utilizes the quantised latent representation and the cross-modality transition module to infer volumetric images. The proposed method has been applied to the lateral X-ray-based CBCT image reconstruction in clinical orthodontics, achieving performance improvements over compared methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bier, B., et al.: Learning to detect anatomical landmarks of the pelvis in x-rays from arbitrary views. Int. J. Comput. Assist. Radiol. Surg. 14, 1463–1473 (2019). https://doi.org/10.1007/s11548-019-01975-5
Bui, M., Albarqouni, S., Schrapp, M., Navab, N., Ilic, S.: X-Ray PoseNet: 6 DoF pose estimation for mobile x-Ray devices. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1036–1044 (2017)
Chen, H., et al.: LEARN: learned experts assessment-based reconstruction network for sparse-data CT. IEEE Trans. Med. Imaging 37, 1333–1347 (2018)
Chou, C.R., et al.: 2D/3D image registration using regression learning. Comput. Vis. Image Underst. 117(9), 1095–1106 (2013)
Grupp, R.B., et al.: Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration. Int. J. Comput. Assist. Radiol. Surg. 15, 759–769 (2020). https://doi.org/10.1007/s11548-020-02162-7
Henzler, P., Rasche, V., Ropinski, T., Ritschel, T.: Single-image tomography: 3D volumes from 2D X-Rays. arXiv preprint arXiv:1710.04867 (2017)
Hou, B., et al.: Predicting slice-to-volume transformation in presence of arbitrary subject motion. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10434, pp. 296–304. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66185-8_34
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Kügler, D., Stefanov, A., Mukhopadhyay, A.: i3PosNet: instrument pose estimation from X-ray. arXiv preprint arXiv:1802.09575 (2018)
Li, R., et al.: 3D tumor localization through real-time volumetric X-ray imaging for lung cancer radiotherapy. Med. Phys. 38(5), 2783–94 (2011)
Likar, B.: A review of 3D/2D registration methods for image-guided interventions. Med. Image Anal. 16(3), 642–661 (2012)
Miao, S., et al.: Dilated FCN for multi-agent 2D/3D medical image registration. In: AAAI (2018)
Miao, S., Wang, Z.J., Liao, R.: A CNN regression approach for real-time 2D/3D registration. IEEE Trans. MI 35(5), 1352–1363 (2016)
Montoya, J., Zhang, C., Li, K., Chen, G.H.: Volumetric scout CT images reconstructed from conventional two-view radiograph localizers using deep learning (Conference Presentation) (2019)
van den Oord, A., Vinyals, O., et al.: Neural discrete representation learning. In: Advances in Neural Information Processing Systems, pp. 6306–6315 (2017)
Pei, Y., Dai, F., Xu, T., Zha, H., Ma, G.: Volumetric reconstruction of craniofacial structures from 2D lateral cephalograms by regression forest. In: IEEE ICIP, pp. 4052–4056 (2016)
Pei, Y., et al.: Non-rigid craniofacial 2D-3D registration using CNN-based regression. In: Cardoso, M.J., et al. (eds.) DLMIA/ML-CDS -2017. LNCS, vol. 10553, pp. 117–125. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67558-9_14
Razavi, A., van den Oord, A., Vinyals, O.: Generating diverse high-fidelity images with VQ-VAE-2. In: Advances in Neural Information Processing Systems, pp. 14837–14847 (2019)
Shen, L., Zhao, W., Xing, L.: Patient-specific reconstruction of volumetric computed tomography images from a single projection view via deep learning. Nat. Biomed. Eng. 3, 880–888 (2019)
Syben, C., Michen, M., Stimpel, B., Seitz, S., Ploner, S.B., Maier, A.K.: PYRO-NN: python reconstruction operators in neural networks. Med. Phys. 46, 5110–5115 (2019)
Toth, D., Cimen, S., Ceccaldi, P., Kurzendorfer, T., Rhode, K.S., Mountney, P.: Training deep networks on domain randomized synthetic X-ray data for cardiac interventions. In: MIDL (2019)
Toth, D., et al.: 3D/2D model-to-image registration by imitation learning for cardiac procedures. Int. J. Comput. Assist. Radiol. Surg. 13(8), 1141–1149 (2018). https://doi.org/10.1007/s11548-018-1774-y
Wu, Y., et al.: Incorporating prior knowledge via volumetric deep residual network to optimize the reconstruction of sparsely sampled MRI. Magn. Reson. Imaging 66, 93–103 (2019)
Würfl, T., et al.: Deep learning computed tomography: learning projection-domain weights from image domain in limited angle problems. IEEE Trans. Med. Imaging 37, 1454–1463 (2018)
Xu, Y., et al.: A method for volumetric imaging in radiotherapy using single X-ray projection. Med. Phys. 42(5), 2498–509 (2014)
Ying, X., Guo, H., Ma, K., Wu, J.Y., Weng, Z., Zheng, Y.: X2CT-GAN: reconstructing CT from biplanar X-rays with generative adversarial networks. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10611–10620 (2019)
Zhang, Y., Miao, S., Mansi, T., Liao, R.: Task driven generative modeling for unsupervised domain adaptation: application to X-ray image segmentation. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 599–607. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_67
Zhang, Y., et al.: Temporal consistent 2D-3D registration of lateral cephalograms and cone-beam computed tomography images. In: Shi, Y., Suk, H.-I., Liu, M. (eds.) MLMI 2018. LNCS, vol. 11046, pp. 371–379. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00919-9_43
Zheng, G.: 3D volumetric intensity reconsturction from 2D X-ray images using partial least squares regression. In: IEEE ISBI, pp. 1268–1271 (2013)
Zheng, J., Miao, S., Liao, R.: Learning CNNs with pairwise domain adaption for real-time 6DoF ultrasound transducer detection and tracking from X-ray images. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10434, pp. 646–654. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66185-8_73
Zhu, B., Liu, J.Z., Rosen, B.R., Rosen, M.S.: Image reconstruction by domain-transform manifold learning. Nature 555, 487–492 (2018)
Acknowledgments
This work was supported by NSFC 61876008.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Jiang, Y. et al. (2020). 3D Volume Reconstruction from Single Lateral X-Ray Image via Cross-Modal Discrete Embedding Transition. In: Liu, M., Yan, P., Lian, C., Cao, X. (eds) Machine Learning in Medical Imaging. MLMI 2020. Lecture Notes in Computer Science(), vol 12436. Springer, Cham. https://doi.org/10.1007/978-3-030-59861-7_33
Download citation
DOI: https://doi.org/10.1007/978-3-030-59861-7_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59860-0
Online ISBN: 978-3-030-59861-7
eBook Packages: Computer ScienceComputer Science (R0)
