Skip to main content
Springer Nature Link
Log in

Generalizable Structure-Aware INF: Biplanar-View CT Reconstruction via Disentangled Implicit Neural Field

  • Conference paper
  • First Online:
Computer Vision – ACCV 2024 (ACCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15472))

Included in the following conference series:

  • 131 Accesses

Abstract

Structure-aware CT reconstruction from a single or biplanar X-rays produces patient-specific 3D insights into underlying structures, pushing the radiation hazards during diagnosis and treatment to a minimum. Existing implicit neural fields (INF) methods have shown impressive performance in CT reconstruction, though they rely on multi-view X-rays and conduct subject-specific reconstruction. Additional online adaptation is required to handle novel subjects. In this paper, we present the generalizable structure-aware implicit neural fields (GSA-INF), a unified model that learns a generalizable structure-aware volume prior to INF decoding from sparse-view X-rays. Previous CoderNeRF views the latent code as a holistic shape prior to 3D reconstruction. In contrast, we present a new triplane generative model to learn a generalizable volume prior distribution, where the sampled triplane latent code produces voxel-level representation for INF decoding and CT reconstruction. Moreover, we introduce anatomical structure mask supervision by building a parallel INF-based decoding framework that enhances structure disentanglements when popping up a variety of structures from 2D X-rays. Our approach entails simultaneous INF-based CT reconstruction and volume-prior learning. In the online inference process, we can conditionally reconstruct CT from single- or biplanar-view X-rays and unconditionally generate CTs via sampling in the latent space. GSA-INF demonstrates robust and superior results over the compared methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Armato, S.G., Mclennan, G., Bidaut, L., McNitt-Gray, M.F., Meyer, C.R., Reeves, A.P., Zhao, B., Aberle, D., Henschke, C.I., Hoffman, E.A., Kazerooni, E.A., MacMahon, H., van Beek, E.J.R., Yankelevitz, D., Biancardi, A.M., Bland, P.H., Brown, M.S., Engelmann, R.M., Laderach, G.E., Max, D., Pais, R.C., Qing, D.P.Y., Roberts, R.Y., Smith, A.R., Starkey, A., Batra, P., Caligiuri, P., Farooqi, A.O., Gladish, G.W., Jude, C.M., Munden, R., Petkovska, I., Quint, L.E., Schwartz, L.H., Sundaram, B., Dodd, L.E., Fenimore, C., Gur, D., Petrick, N., Freymann, J.B., Kirby, J., Hughes, B., Casteele, A.V., Gupte, S., Sallam, M., Heath, M.D., Kuhn, M.H., Dharaiya, E., Burns, R., Fryd, D., Salganicoff, M., Anand, V., Shreter, U., Vastagh, S., Croft, B.Y., Clarke, L.P.: The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans. Med. Phys. 38(2), 915–31 (2011)

    Article  Google Scholar 

  2. Athar, S., Shu, Z., Samaras, D.: Flame-in-nerf : Neural control of radiance fields for free view face animation. ArXiv abs/2108.04913 (2021)

    Google Scholar 

  3. Binkowski, M., Sutherland, D.J., Arbel, M., Gretton, A.: Demystifying mmd gans. ArXiv abs/1801.01401 (2018)

    Google Scholar 

  4. Chen, H., Zhang, Y., Chen, Y., Zhang, J., Zhang, W., Sun, H., Lv, Y., Liao, P., Zhou, J., Wang, G.: Learn: Learned experts assessment-based reconstruction network for sparse-data ct. IEEE Trans. Med. Imag. 37, 1333–1347 (2018)

    Article  Google Scholar 

  5. Chen, H., Gu, J., Chen, A., Tian, W., Tu, Z., Liu, L., Su, H.: Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction. IEEE/CVF International Conference on Computer Vision (ICCV) pp. 2416–2425 (2023)

    Google Scholar 

  6. Chen, Z., Zhang, H.: Learning implicit fields for generative shape modeling. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 5932–5941 (2019)

    Google Scholar 

  7. Corona-Figueroa, A., Frawley, J., Bond-Taylor, S., Bethapudi, S., Shum, H.P.H., Willcocks, C.G.: Mednerf: Medical neural radiance fields for reconstructing 3d-aware ct-projections from a single x-ray. 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) pp. 3843–3848 (2022)

    Google Scholar 

  8. Eyndhoven, G.V., Batenburg, K.J., Kazantsev, D., Nieuwenhove, V.V., Lee, P., Dobson, K., Sijbers, J.: An iterative ct reconstruction algorithm for fast fluid flow imaging. IEEE Trans. Image Process. 24, 4446–4458 (2015)

    Article  MathSciNet  Google Scholar 

  9. Fang, Y., Mei, L., Li, C., Liu, Y., Wang, W., Cui, Z., Shen, D.: Snaf: Sparse-view cbct reconstruction with neural attenuation fields. ArXiv abs/2211.17048 (2022)

    Google Scholar 

  10. Folkerts, M.M.: drrsuite. In: https://code.google.com/archive/p/drrsuite/

  11. Fu, J., Dong, J., Zhao, F.: A deep learning reconstruction framework for differential phase-contrast computed tomography with incomplete data. IEEE Trans. Image Process. 29, 2190–2202 (2020)

    Article  Google Scholar 

  12. Gafni, G., Thies, J., Zollhofer, M., Nießner, M.: Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 8645–8654 (2021)

    Google Scholar 

  13. Gao, C., Shih, Y., Lai, W.S., Liang, C.K., Huang, J.B.: Portrait neural radiance fields from a single image. ArXiv abs/2012.05903 (2020)

    Google Scholar 

  14. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 770–778 (2016)

    Google Scholar 

  15. Henzler, P., Rasche, V., Ropinski, T., Ritschel, T.: Single-image tomography: 3d volumes from 2d cranial x-rays. Computer Graphics Forum 37 (2018)

    Google Scholar 

  16. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Neural Information Processing Systems (2017)

    Google Scholar 

  17. Jang, W.J., de Agapito, L.: Codenerf: Disentangled neural radiance fields for object categories. IEEE/CVF International Conference on Computer Vision (ICCV) pp. 12929–12938 (2021)

    Google Scholar 

  18. Jiang, Y., Pei, Y., Li, P., Zhang, Y., Guo, Y., Fan, Y., Chen, G., Dai, F., Xu, T., Yuan, X., yan Zha, H.: Craniofacial volumetric image estimation from a lateral cephalogram using cross-dimensional discrete embedding mapping. IEEE Transactions on Computational Imaging 8, 972–985 (2022)

    Google Scholar 

  19. Jiang, Y., Yuan, X., Pei, Y.: Spatially-consistent implicit volumetric function for uni- and bi-planar x-ray-based computed tomography reconstruction. IEEE 20th International Symposium on Biomedical Imaging (ISBI) pp. 1–5 (2023)

    Google Scholar 

  20. Liu, J., Anirudh, R., Thiagarajan, J.J., He, S., Mohan, K.A., Kamilov, U.S., Kim, H.: Dolce: A model-based probabilistic diffusion framework for limited-angle ct reconstruction. IEEE/CVF International Conference on Computer Vision (ICCV) pp. 10464–10474 (2022)

    Google Scholar 

  21. Liu, S., Li, T., Chen, W., Li, H.: Soft rasterizer: A differentiable renderer for image-based 3d reasoning. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 7707–7716 (2019)

    Google Scholar 

  22. Montoya, J., Zhang, C., Li, K., Chen, G.H.: Volumetric scout ct images reconstructed from conventional two-view radiograph localizers using deep learning. In: Physics of Medical Imaging. vol. 10948 (2019)

    Google Scholar 

  23. Muller, N., Siddiqui, Y., Porzi, L., Bulò, S.R., Kontschieder, P., Nießner, M.: Diffrf: Rendering-guided 3d radiance field diffusion. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 4328–4338 (2022)

    Google Scholar 

  24. Müller, N., Simonelli, A., Porzi, L., Bulò, S.R., Nießner, M., Kontschieder, P.: Autorf: Learning 3d object radiance fields from single view observations. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 3961–3970 (2022)

    Google Scholar 

  25. Niemeyer, M., Mescheder, L.M., Oechsle, M., Geiger, A.: Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 3501–3512 (2020)

    Google Scholar 

  26. Preiswerk, F., Toews, M., Cheng, C.C., yuan George Chiou, J., Mei, C.S., Schaefer, L., Hoge, W.S., Schwartz, B.M., Panych, L., Madore, B.: Hybrid mri ultrasound acquisitions, and scannerless realtime imaging. Magnetic Resonance in Medicine 78, 897”C908 (2017)

    Google Scholar 

  27. Reed, A.W., Kim, H., Anirudh, R., Mohan, K.A., Champley, K.M., Kang, J., Jayasuriya, S.: Dynamic ct reconstruction from limited views with implicit neural representations and parametric motion fields. IEEE/CVF International Conference on Computer Vision (ICCV) pp. 2238–2248 (2021)

    Google Scholar 

  28. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. ArXiv abs/1505.04597 (2015)

    Google Scholar 

  29. Rückert, D., Wang, Y., Li, R., Idoughi, R., Heidrich, W.: Neat: Neural adaptive tomography. ACM Trans. Graph. 41, 55:1–55:13 (2022)

    Google Scholar 

  30. Setio, A.A.A., et al.: Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: The luna16 challenge. Med. Image Anal. 42, 1–13 (2016)

    Article  Google Scholar 

  31. Shen, L., Pauly, J.M., Xing, L.: Nerp: Implicit neural representation learning with prior embedding for sparsely sampled image reconstruction. IEEE transactions on neural networks and learning systems (2021)

    Google Scholar 

  32. Shen, L., Zhao, W., Xing, L.: Patient-specific reconstruction of volumetric computed tomography images from a single projection view via deep learning. Nat. Biomed. Eng. 3, 880–888 (2019)

    Article  Google Scholar 

  33. Sitzmann, V., Zollhoefer, M., Wetzstein, G.: Scene representation networks: Continuous 3d-structure-aware neural scene representations. ArXiv abs/1906.01618 (2019)

    Google Scholar 

  34. Song, B., Shen, L., Xing, L.: Piner: Prior-informed implicit neural representation learning for test-time adaptation in sparse-view ct reconstruction. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) pp. 1928–1937 (2023)

    Google Scholar 

  35. Syben, C., Michen, M., Stimpel, B., Seitz, S., Ploner, S.B., Maier, A.K.: Pyro-nn: Python reconstruction operators in neural networks. Med. Phys. 46, 5110–5115 (2019)

    Article  Google Scholar 

  36. Vasconcelos, F., He, B., Singh, N., Teh, Y.W.: Uncertainr: Uncertainty quantification of end-to-end implicit neural representations for computed tomography. ArXiv abs/2202.10847 (2022)

    Google Scholar 

  37. Wang, D., Chandran, P., Zoss, G., Bradley, D., Gotardo, P.F.U.: Morf: Morphable radiance fields for multiview neural head modeling. ACM SIGGRAPH 2022 Conference Proceedings (2022)

    Google Scholar 

  38. Wu, Y., Ma, Y., Capaldi, D.P.I., Liu, J., Zhao, W., Du, J., Xing, L.: Incorporating prior knowledge via volumetric deep residual network to optimize the reconstruction of sparsely sampled mri. Magnetic resonance imaging (2019)

    Google Scholar 

  39. Würfl, T., Hoffmann, M., Christlein, V., Breininger, K., Huang, Y., Unberath, M., Maier, A.K.: Deep learning computed tomography: Learning projection-domain weights from image domain in limited angle problems. IEEE Trans. Med. Imag. 37, 1454–1463 (2018)

    Article  Google Scholar 

  40. Xu, Q., Wang, W., Ceylan, D., Mech, R., Neumann, U.: Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. In: NeurIPS (2019)

    Google Scholar 

  41. Ying, X., Guo, H., Ma, K., Wu, J.Y., Weng, Z., Zheng, Y.: X2ct-gan: Reconstructing ct from biplanar x-rays with generative adversarial networks. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 10611–10620 (2019)

    Google Scholar 

  42. Yu, A., Ye, V., Tancik, M., Kanazawa, A.: pixelnerf: Neural radiance fields from one or few images. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

    Google Scholar 

  43. Zang, G., Idoughi, R., Li, R., Wonka, P., Heidrich, W.: Intratomo: Self-supervised learning-based tomography via sinogram synthesis and prediction. IEEE/CVF International Conference on Computer Vision (ICCV) pp. 1940–1950 (2021)

    Google Scholar 

  44. Zha, R., Zhang, Y., Li, H.: Naf: Neural attenuation fields for sparse-view cbct reconstruction. ArXiv abs/2209.14540 (2022)

    Google Scholar 

  45. Zhu, B., Liu, J.Z., Rosen, B.R., Rosen, M.S.: Image reconstruction by domain-transform manifold learning. Nature 555, 487–492 (2018)

    Article  Google Scholar 

  46. Zhuang, Y., Zhu, H., Sun, X., Cao, X.: Mofanerf: Morphable facial neural radiance field. In: European Conference on Computer Vision (2021)

    Google Scholar 

Download references

Acknowledgments

This work was supported in part by National Natural Science Foundation of China under Grant 62272011 and 61876008, Beijing Natural Science Foundation 7232337.

Author information

Authors and Affiliations

  1. School of Intelligence Science and Technology, Key Laboratory of Machine Perception (MOE), State Key Laboratory of General Artificial Intelligence, Peking University, Beijing, 100871, China

    Bei Huang & Yuru Pei

Authors
  1. Bei Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuru Pei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuru Pei .

Editor information

Editors and Affiliations

  1. Pohang University of Science and Technology (POSTECH), Pohang, Korea (Republic of)

    Minsu Cho

  2. Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates

    Ivan Laptev

  3. Google, Mountain View, CA, USA

    Du Tran

  4. National University of Singapore, Singapore, Singapore

    Angela Yao

  5. Peking University, Beijing, China

    Hongbin Zha

Ethics declarations

Disclosure of Interests

The authors have no competing interests to declare that are relevant to the content of this article.

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Huang, B., Pei, Y. (2025). Generalizable Structure-Aware INF: Biplanar-View CT Reconstruction via Disentangled Implicit Neural Field. In: Cho, M., Laptev, I., Tran, D., Yao, A., Zha, H. (eds) Computer Vision – ACCV 2024. ACCV 2024. Lecture Notes in Computer Science, vol 15472. Springer, Singapore. https://doi.org/10.1007/978-981-96-0885-0_9

Download citation

  • DOI: https://doi.org/10.1007/978-981-96-0885-0_9

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-96-0884-3

  • Online ISBN: 978-981-96-0885-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics