Abstract
We propose a method to address the challenge of generating a 3D digital twin of a patient during an X-ray guided medical procedure from a single 2D X-ray projection image, a problem that is inherently ill-posed. To tackle this issue, we aim to infer the parameters of Bones, Organs and Skin Shape (BOSS) model, a deformable human shape and pose model. There are currently two main approaches for model-based estimation. Optimization-based methods try to iteratively fit a body model to 2D measurements, they produce accurate 2D alignments but are slow and sensitive to initialization. On the other hand, regression-based methods use neural networks to estimate the model parameters directly, resulting in faster predictions but often with misalignments. Our approach combines the benefits of both techniques by implementing a fully differentiable paradigm through the use of higher-order optimizers that only require the Jacobian, which can be determined implicitly. The network was trained on synthetic CT and real CBCT image data, ensuring view independence. We demonstrate the potential clinical applicability of our method by validating it on multiple datasets covering diverse anatomical regions, and achieving an error of 27.98 mm.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ehlke, M., Ramm, H., Lamecker, H., Hege, H.C., Zachow, S.: Fast generation of virtual X-ray images for reconstruction of 3D anatomy. IEEE Trans. Vis. Comput. Graph. 19(12), 2673–2682 (2013)
Fotouhi, J., Liu, X., Armand, M., Navab, N., Unberath, M.: From perspective X-ray imaging to parallax-robust orthographic stitching. arXiv preprint arXiv:2003.02959 (2020)
Jaganathan, S., Wang, J., Borsdorf, A., Shetty, K., Maier, A.: Deep iterative 2D/3D registration. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12904, pp. 383–392. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87202-1_37
Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.: End-to-end recovery of human shape and pose. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7122–7131. IEEE Computer Society (2018)
Kasten, Y., Doktofsky, D., Kovler, I.: End-to-end convolutional neural network for 3D reconstruction of knee bones from bi-planar X-ray images. In: Deeba, F., Johnson, P., Würfl, T., Ye, J.C. (eds.) MLMIR 2020. LNCS, vol. 12450, pp. 123–133. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61598-7_12
Keller, M., Zuffi, S., Black, M.J., Pujades, S.: OSSO: obtaining skeletal shape from outside. In: Proceedings IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 20492–20501 (2022)
Kinahan, P., Muzi, M., Bialecki, B., Herman, B., Coombs, L.: Data from the ACRIN 6668 trial NSCLC-FDG-PET (2019). https://doi.org/10.7937/TCIA.2019.30ILQFCL
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Kolotouros, N., Pavlakos, G., Black, M.J., Daniilidis, K.: Learning to reconstruct 3D human pose and shape via model-fitting in the loop. In: ICCV (2019)
Lamecker, H., Wenckebach, T.H., Hege, H.: Atlas-based 3D-shape reconstruction from X-ray images. In: 18th International Conference on Pattern Recognition, ICPR 2006, vol. 1, pp. 371–374 (2006). https://doi.org/10.1109/ICPR.2006.279
Li, J., Xu, C., Chen, Z., Bian, S., Yang, L., Lu, C.: HybrIK: a hybrid analytical-neural inverse kinematics solution for 3D human pose and shape estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3383–3393 (2021)
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: a skinned multi-person linear model. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 34(6), 248:1–248:16 (2015)
Moon, G., Lee, K.M.: I2L-MeshNet: image-to-lixel prediction network for accurate 3D human pose and mesh estimation from a single RGB image. arXiv arXiv:2008.03713 (2020)
Pavlakos, G., et al.: Expressive body capture: 3D hands, face, and body from a single image (2019)
Pineda, L., et al.: Theseus: a library for differentiable nonlinear optimization. In: Advances in Neural Information Processing Systems (2022)
Roser, P., et al.: Physics-driven learning of x-ray skin dose distribution in interventional procedures. Med. Phys. 46(10), 4654–4665 (2019)
Sekuboyina, A., et al.: VerSe: a vertebrae labelling and segmentation benchmark for multi-detector CT images. Med. Image Anal. 73, 102166 (2021)
Shen, L., Zhao, W., Xing, L.: Patient-specific reconstruction of volumetric computed tomography images from a single projection view via deep learning. Nat. Biomed. Eng. 3(11), 880–888 (2019)
Shetty, K., et al.: BOSS: Bones, organs and skin shape model. Comput. Biol. Med. 165, 107383 (2023). https://doi.org/10.1016/j.compbiomed.2023.107383. ISSN 0010-4825
Shetty, K., et al.: PLIKS: a pseudo-linear inverse kinematic solver for 3D human body estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 574–584 (2023)
Tan, M., Le, Q.: EfficientNetV2: smaller models and faster training. In: International Conference on Machine Learning, pp. 10096–10106. PMLR (2021)
Tretschk, E., et al.: State of the art in dense monocular non-rigid 3D reconstruction (2022). https://doi.org/10.48550/ARXIV.2210.15664. https://arxiv.org/abs/2210.15664
Unberath, M., et al.: DeepDRR – a catalyst for machine learning in fluoroscopy-guided procedures. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 98–106. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_12
Vávra, P., et al.: Recent development of augmented reality in surgery: a review. J. Healthc. Eng. 2017, 4574172 (2017)
Ying, X., Guo, H., Ma, K., Wu, J., Weng, Z., Zheng, Y.: X2CT-GAN: reconstructing CT from biplanar X-rays with generative adversarial networks (2019)
Zheng, C., et al.: Deep learning-based human pose estimation: a survey. arXiv arXiv:2012.13392 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
The concepts and information presented in this article are based on research and are not commercially available.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Shetty, K. et al. (2024). HOOREX: Higher Order Optimizers for 3D Recovery from X-Ray Images. In: Maier, A.K., Schnabel, J.A., Tiwari, P., Stegle, O. (eds) Machine Learning for Multimodal Healthcare Data. ML4MHD 2023. Lecture Notes in Computer Science, vol 14315. Springer, Cham. https://doi.org/10.1007/978-3-031-47679-2_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-47679-2_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47678-5
Online ISBN: 978-3-031-47679-2
eBook Packages: Computer ScienceComputer Science (R0)