Skip to main content
SpringerLink
Log in

Predictive online 3D target tracking with population-based generative networks for image-guided radiotherapy

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Respiratory motion of thoracic organs poses a severe challenge for the administration of image-guided radiotherapy treatments. Providing online and up-to-date volumetric information during free breathing can improve target tracking, ultimately increasing treatment efficiency and reducing toxicity to surrounding healthy tissue. In this work, a novel population-based generative network is proposed to address the problem of 3D target location prediction from 2D image-based surrogates during radiotherapy, thus enabling out-of-plane tracking of treatment targets using images acquired in real time.

Methods

The proposed model is trained to simultaneously create a low-dimensional manifold representation of 3D non-rigid deformations and to predict, ahead of time, the motion of the treatment target. The predictive capabilities of the model allow correcting target location errors that can arise due to system latency, using only a baseline volume of the patient anatomy. Importantly, the method does not require supervised information such as ground-truth registration fields, organ segmentation, or anatomical landmarks.

Results

The proposed architecture was evaluated on both free-breathing 4D MRI and ultrasound datasets. Potential challenges present in a realistic therapy, like different acquisition protocols, were taken into account by using an independent hold-out test set. Our approach enables 3D target tracking from single-view slices with a mean landmark error of 1.8 mm, 2.4 mm and 5.2 mm in volunteer MRI, patient MRI and US datasets, respectively, without requiring any prior subject-specific 4D acquisition.

Conclusions

This model presents several advantages over state-of-the-art approaches. Namely, it benefits from an explainable latent space with explicit respiratory phase discrimination. Thanks to the strong generalization capabilities of neural networks, it does not require establishing inter-subject correspondences. Once trained, it can be quickly deployed with an inference time of only 8 ms. The results show the capability of the network to predict future anatomical changes and track tumors in real time, yielding statistically significant improvements over related methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Arnold P, Preiswerk F, Fasel B, Salomir R, Scheffler K, Cattin PC (2011) 3D organ motion prediction for mr-guided high intensity focused ultrasound. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 623–630

  2. Balakrishnan G, Zhao A, Sabuncu MR, Guttag J, Dalca AV (2019) Voxelmorph: a learning framework for deformable medical image registration. IEEE Trans Med Imaging 38(8):1788–1800

    Article  Google Scholar 

  3. Boye D, Samei G, Schmidt J, Székely G, Tanner C (2013) Population based modeling of respiratory lung motion and prediction from partial information. In: Medical imaging 2013: image processing. International Society for Optics and Photonics, p 86690U

  4. Coupé P, Hellier P, Kervrann C, Barillot C (2009) Nonlocal means-based speckle filtering for ultrasound images. IEEE Trans Image Process 18(10):2221–2229

    Article  Google Scholar 

  5. Ehrhardt J, Werner R, Schmidt-Richberg A, Handels H (2009) 4D motion modeling: estimation of respiratory motion for radiation therapy. Biological and medical physics, biomedical engineering, vol 25. Springer, Berlin

    Google Scholar 

  6. Fayad HJ, Buerger C, Tsoumpas C, Cheze-Le-Rest C, Visvikis D (2012) A generic respiratory motion model based on 4D MRI imaging and 2D image navigators. In: 2012 IEEE nuclear science symposium and medical imaging conference record (NSS/MIC)

  7. Feng L, Axel L, Chandarana H, Block KT, Sodickson DK, Otazo R (2016) Xd-grasp: golden-angle radial MRI with reconstruction of extra motion-state dimensions using compressed sensing. Magn Reson Med 75(2):775–788

    Article  Google Scholar 

  8. Garau N, Via R, Meschini G, Lee D, Keall PJ, Riboldi M, Baroni G, Paganelli C (2019) A ROI-based global motion model established on 4DCT and 2D cine-MRI data for MRI-guidance in radiation therapy. Phys Med Biol 64:045002

    Article  Google Scholar 

  9. Giger A, Sandkühler R, Jud C, Bauman G, Bieri O, Salomir R, Cattin C (2018) Respiratory motion modelling using cgans. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 81–88

  10. Girdhar R, Fouhey DF, Rodriguez M, Gupta A (2016) Learning a predictable and generative vector representation for objects. In: European conference on computer vision. Springer, pp 484–499

  11. Wendy H, Fang-Fang Y, Jing C, Lei R (2020) Volumetric cine magnetic resonance imaging (VC-MRI) using motion modeling, free-form deformation and multi-slice undersampled 2D cine MRI reconstructed with spatio-temporal low-rank decomposition. Quant Imaging Med Surg 10(2):432

    Article  Google Scholar 

  12. Lauren H, Rojano K, Clifford R, Austen C, Todd DW, Jeffrey B, Olga G, Jeff M, Sasa M, Parag P (2018) Phase I trial of stereotactic MR-guided online adaptive radiation therapy (smart) for the treatment of oligometastatic or unresectable primary malignancies of the abdomen. Radiother Oncol 126(3):519–526

    Article  Google Scholar 

  13. Jud C, Preiswerk F, Cattin PC (2015) Respiratory motion compensation with topology independent surrogates. In: Workshop on imaging and computer assistance in radiation therapy

  14. Klein S, Staring M, Murphy K, Viergever MA, Pluim JPW (2009) Elastix: a toolbox for intensity-based medical image registration. IEEE Trans Med Imaging 29(1):196–205

    Article  Google Scholar 

  15. Kurenkov A, Ji J, Garg A, Mehta V, Gwak JY, Choy C, Savarese S (2018) Deformnet: free-form deformation network for 3d shape reconstruction from a single image. In: 2018 IEEE winter conference on applications of computer vision (WACV). IEEE, pp 858–866

  16. Kurz C, Buizza G, Landry G, Kamp F, Rabe M, Paganelli C, Baroni G, Reiner M, Keall PJ, van den Berg CAT (2020) Medical physics challenges in clinical MR-guided radiotherapy. Radiat Oncol 15:1–16

    Article  Google Scholar 

  17. Küstner T, Fuin N, Hammernik K, Bustin A, Qi H, Hajhosseiny R, Masci PG, Neji R, Rueckert D, Botnar RM, Prieto C (2020) Cinenet: deep learning-based 3D cardiac cine MRI reconstruction with multi-coil complex-valued 4D spatio-temporal convolutions. Sci Rep 10(1):1–13

    Article  Google Scholar 

  18. Mezheritsky T, Romaguera LV, Kadoury S (2020) 3D ultrasound generation from partial 2D observations using fully convolutional and spatial transformation networks. In: 2020 IEEE 17th international symposium on biomedical imaging (ISBI). IEEE, pp 1808–1811

  19. Paganelli C, Lee D, Kipritidis J, Whelan B, Greer PB, Baroni G, Riboldi M, Keall P (2018) Feasibility study on 3D image reconstruction from 2D orthogonal cine-MRI for MRI-guided radiotherapy. J Med Imaging Radiat Oncol 62(3):389–400

    Article  Google Scholar 

  20. Preiswerk F, De Luca V, Arnold P, Celicanin Z, Petrusca L, Tanner C, Bieri O, Salomir R, Cattin PC (2014) Model-guided respiratory organ motion prediction of the liver from 2D ultrasound. Med Image Anal 18(5):740–751

    Article  Google Scholar 

  21. Romaguera LV, Plantefève R, Romero FP, Hébert F, Carrier J-F, Kadoury S (2020) Prediction of in-plane organ deformation during free-breathing radiotherapy via discriminative spatial transformer networks. Med Image Anal 64:101754

    Article  Google Scholar 

  22. Seregni M, Paganelli C, Kipritidis J, Baroni G, Riboldi M (2017) Out-of-plane motion correction in orthogonal cine-MRI registration. Radiother Oncol 123:S147–S148

    Article  Google Scholar 

  23. Stemkens B, Paulson ES, Tijssen RHN (2018) Nuts and bolts of 4D-MRI for radiotherapy. Phys Med Biol 63(21):21TR01

    Article  CAS  Google Scholar 

  24. Stemkens B, Tijssen RHN, De Senneville BD, Lagendijk JJW, Van Den Berg CAT (2016) Image-driven, model-based 3d abdominal motion estimation for MR-guided radiotherapy. Phys Med Biol 61(14):5335

    Article  CAS  Google Scholar 

  25. Tanner C, Yang M, Samei G, Székely G (2016) Influence of inter-subject correspondences on liver motion predictions from population models. In: 2016 IEEE 13th international symposium on biomedical imaging (ISBI). IEEE, pp 286–289

  26. Tanner C, Zur Y, French K, Samei G, Strehlow J, Sat G, McLeod H, Houston G, Kozerke, Székely G, Melzer A Preusser T (2016) In vivo validation of spatio-temporal liver motion prediction from motion tracked on mr thermometry images. Int J Comput Assist Radiol Surg 11(6:1143–1152

  27. von Siebenthal M, Székely G, Lomax A, Cattin PC (2007) Inter-subject modelling of liver deformation during radiation therapy. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 659–666

  28. von Siebenthal M, Szekely G, Gamper U, Boesiger P, Lomax A, Cattin PC (2007) 4D MR imaging of respiratory organ motion and its variability. Phys Med Biol 52(6):1547

    Article  Google Scholar 

  29. Vorontsov E, Molchanov P, Byeon W, De Mello S, Jampani V, Liu M-Y, Kadoury S, Kautz J (2019) Boosting segmentation with weak supervision from image-to-image translation. arXiv preprint arXiv:1904.01636

  30. Wilms M, Werner R, Yamamoto T, Handels H, Ehrhardt J (2017) Subpopulation-based correspondence modelling for improved respiratory motion estimation in the presence of inter-fraction motion variations. Phys Med Biol 62(14):5823

    Article  Google Scholar 

Download references

Funding

This work was partly funded by an NSERC collaborative research and development project (CRDPJ-517413-17) and by the Canada First Research Excellence Fund through the TransMedTech Institute.

Author information

Authors and Affiliations

  1. École Polytechnique de Montréal, Montréal, Canada

    Liset Vázquez Romaguera & Tal Mezheritsky

  2. Centre de Recherche du Centre Hospitalier de l’Université de Montréal, Montréal, Canada

    Rihab Mansour

  3. Département de Radiologie, Radio-Oncologie et Médecine Nucléaire, Faculté de médecine, Université de Montréal, Montréal, Canada

    William Tanguay

  4. École Polytechnique de Montréal, Centre de Recherche du Centre Hospitalier de l’Université de Montréal, Montréal, Canada

    Samuel Kadoury

Authors
  1. Liset Vázquez Romaguera
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tal Mezheritsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rihab Mansour
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. William Tanguay
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Samuel Kadoury
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Samuel Kadoury.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interests.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 5788 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Romaguera, L.V., Mezheritsky, T., Mansour, R. et al. Predictive online 3D target tracking with population-based generative networks for image-guided radiotherapy. Int J CARS 16, 1213–1225 (2021). https://doi.org/10.1007/s11548-021-02425-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-021-02425-x

Keywords

Search

Navigation