Skip to main content
SpringerLink
Log in

Inpainting truncated areas of CT images based on generative adversarial networks with gated convolution for radiotherapy

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

This study aimed to inpaint the truncated areas of CT images by using generative adversarial networks with gated convolution (GatedConv) and apply these images to dose calculations in radiotherapy. CT images were collected from 100 patients with esophageal cancer under thermoplastic membrane placement, and 85 cases were used for training based on randomly generated circle masks. In the prediction stage, 15 cases of data were used to evaluate the accuracy of the inpainted CT in anatomy and dosimetry based on the mask with a truncated volume covering 40% of the arm volume, and they were compared with the inpainted CT synthesized by U-Net, pix2pix, and PConv with partial convolution. The results showed that GatedConv could directly and effectively inpaint incomplete CT images in the image domain. For the results of U-Net, pix2pix, PConv, and GatedConv, the mean absolute errors for the truncated tissue were 195.54, 196.20, 190.40, and 158.45 HU, respectively. The mean dose of the planning target volume, heart, and lung in the truncated CT was statistically different (p < 0.05) from those of the ground truth CT (\({\mathrm{CT}}_{\mathrm{gt}}\)). The differences in dose distribution between the inpainted CT obtained by the four models and \({\mathrm{CT}}_{\mathrm{gt}}\) were minimal. The inpainting effect of clinical truncated CT images based on GatedConv showed better stability compared with the other models. GatedConv can effectively inpaint the truncated areas with high image quality, and it is closer to \({\mathrm{CT}}_{\mathrm{gt}}\) in terms of image visualization and dosimetry than other inpainting models.

Graphical Abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Chen HH, Kuo MT (2017) Improving radiotherapy in cancer treatment: promises and challenges. Oncotarget 8(37):62742–62758

    Article  PubMed  PubMed Central  Google Scholar 

  2. Molinelli S, Russo S, Magro G, Maestri D, Mairani A, Mastella E, Mirandola A, Vai A, Vischioni B, Valvo F (2019) Impact of TPS calculation algorithms on dose delivered to the patient in proton therapy treatments. Phys Med Biol 64(7):075016

    Article  CAS  PubMed  Google Scholar 

  3. van der Heyden B, Wohlfahrt P, Eekers DB, Richter C, Terhaag K, Troost EG, Verhaegen F (2019) Dual-energy CT for automatic organs-at-risk segmentation in brain-tumor patients using a multi-atlas and deep-learning approach. Sci Rep 9(1):4126

    Article  PubMed  PubMed Central  Google Scholar 

  4. Swinnen AC, Öllers MC, Loon Ong C, Verhaegen F (2020) The potential of an optical surface tracking system in non-coplanar single isocenter treatments of multiple brain metastases. J Appl Clin Med Phys 21(6):63–72

    Article  PubMed  PubMed Central  Google Scholar 

  5. Podesta M, Nijsten SMJJG, Persoon LCGG, Scheib SG, Baltes C, Verhaegen F (2014) Time dependent pre-treatment EPID dosimetry for standard and FFF VMAT. Phys Med Biol 59(16):4749

    Article  PubMed  Google Scholar 

  6. Fonseca GP, Baer-Beck M, Fournie E, Hofmann C, Rinaldi I, Ollers MC, van Elmpt WJ, Verhaegen F (2021) Evaluation of novel AI-based extended field-of-view CT reconstructions. Med Phys 48(7):3583–3594

    Article  PubMed  Google Scholar 

  7. Wu V, Podgorsak MB, Tran TA, Malhotra HK, Wang IZ (2011) Dosimetric impact of image artifact from a wide-bore CT scanner in radiotherapy treatment planning. Med Phys 38(7):4451–4463

    Article  PubMed  Google Scholar 

  8. Murrell DH, Karnas SJ, Corkum MT, Hipwell S, Louie AV (2019) Radical radiotherapy for locally advanced non-small cell lung cancer—what’s up with arm positioning? J Thorac Dis 11(5):2099–2104

    Article  PubMed  PubMed Central  Google Scholar 

  9. Hui Z, Wang Q, Han W, Sun S, Wang M, Dai J, Wang L (2014) Comparation of set-up errors between two different body positions in precision radiotherapy for esophageal cancer. Chin J Radiat Oncol 23(4):336–339

    Google Scholar 

  10. Cheung JP, Shugard E, Mistry N, Pouliot J, Chen J (2019) Evaluating the impact of extended field-of-view CT reconstructions on CT values and dosimetric accuracy for radiation therapy. Med Phys 46(2):892–901

    Article  PubMed  Google Scholar 

  11. Hsieh J, Chao E, Thibault J, Grekowicz B, Horst A, Mcolash S, Myers TJ (2004) A novel reconstruction algorithm to extend the CT scan field-of-view. Med Phys 31(9):2385–2391

    Article  CAS  PubMed  Google Scholar 

  12. Sourbelle K, Kachelrieß M, Kalender WA (2005) Reconstruction from truncated projections in CT using adaptive detruncation. Eur Radiol 15(5):1008–1014

    Article  CAS  PubMed  Google Scholar 

  13. Xia Y, Berger M, Bauer S, Hu S, Aichert A, Maier A (2017) An improved extrapolation scheme for truncated CT data using 2D Fourier-based Helgason-Ludwig consistency conditions. Int J Biomed Imaging 2017:1867025

    Article  PubMed  PubMed Central  Google Scholar 

  14. Du Y, Zhu L, Xi X, Han Y, Li L, Yan B (2020) CT local reconstruction method based on truncated data extrapolation network. Appl Opt Photon China (AOPC 2020) 11565:221–226

    Google Scholar 

  15. Li B, Deng J, Lonn AH, Jiang H (2012) An enhanced reconstruction algorithm to extend CT scan field-of-view with z-axis consistency constraint. Med Phys 39(10):6028–6034

    Article  PubMed  Google Scholar 

  16. Bruder H, Sedlmair MU, Stierstorfer K, Flohr TG (2011) High definition extended field of view (HD FOV) reconstruction in CT. In: Radiological Society of North America 2011 Scientific Assembly and Annual Meeting. Chicago, IL, p 1. http://archive.rsna.org/2011/11015308.html. Accessed 8 Mar 2023

  17. Bhosale YH, Patnaik KS (2022) Application of deep learning techniques in diagnosis of covid-19 (coronavirus): a systematic review. Neural Process Lett 1–53. https://doi.org/10.1007/s11063-022-11023-0

  18. Bhosale YH, Patnaik KS (2023) PulDi-COVID: Chronic obstructive pulmonary (lung) diseases with COVID-19 classification using ensemble deep convolutional neural network from chest X-ray images to minimize severity and mortality rates. Biomed Sig Process Control 81:104445

    Article  Google Scholar 

  19. Fournié É, Baer-Beck M, Stierstorfer K (2019) CT field of view extension using combined channels extension and deep learning methods, arXiv preprint arXiv:1908.09529. https://doi.org/10.48550/arXiv.1908.09529

  20. Huang Y, Gao L, Preuhs A, Maier A (2020) Field of view extension in computed tomography using deep learning prior. In: Tolxdorff T, Deserno T, Handels H, Maier A, Maier-Hein K, Palm C (eds) Bildverarbeitung für die Medizin 2020. Informatik aktuell. Springer Vieweg, Wiesbaden, pp 186–191. https://doi.org/10.1007/978-3-658-29267-6_40

  21. Bertalmio M, Sapiro G, Caselles V, Ballester C (2000) Image inpainting. In: Proceedings of the 27th annual conference on computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co, New York, NY, pp 417–424

  22. Pathak D, Krahenbuhl P, Donahue J, Darrell T, Efros A A (2016) Context encoders: feature learning by inpainting. In: Proceeding of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA, pp 2536–2544. https://doi.org/10.1109/CVPR.2016.278

  23. Goodfellow IJ, Pougetabadie J, Mirza M, Xu B, Wardefarley D, Ozair S, Courville AC, Bengio Y (2014) Generative adversarial nets. Neural Inf Process Syst 2:2672–2680

    Google Scholar 

  24. Iizuka S, Simo-Serra E, Ishikawa H (2017) Globally and locally consistent image completion. ACM Trans Graph (ToG) 36(4):107.101–107.114

    Article  Google Scholar 

  25. Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative image inpainting with contextual attention. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA, pp 5505–5514. https://doi.org/10.1109/CVPR.2018.00577

  26. Liu G, Reda FA, Shih KJ, Wang T-C, Tao A, Catanzaro B (2018) Image inpainting for irregular holes using partial convolutions. In: Ferrari V, Hebert M, Sminchisescu C, Weiss Y (eds) Computer Vision – ECCV 2018. ECCV 2018. Lecture Notes in Computer Science, vol 11215. Springer, Cham, pp 85–100. https://doi.org/10.1007/978-3-030-01252-6_6

  27. Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2019) Free-form image inpainting with gated convolution. In: Proceedings of 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Seoul, Korea, pp 4471–4480

  28. Suvorov R, Logacheva E, Mashikhin A, Remizova A, Ashukha A, Silvestrov A, Kong N, Goka H, Park K, Lempitsky V (2022) Resolution-robust large mask inpainting with fourier convolutions. In: Proceedings of 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, pp 2149–2159. https://doi.org/10.48550/arXiv.2109.07161

  29. Yu Y, Zhan F, Wu R, Pan J, Cui K, Lu S, Ma F, Xie X, Miao C (2021) Diverse image inpainting with bidirectional and autoregressive transformers. In: Proceedings of the 29th ACM International Conference on Multimedia. Association for Computing Machinery, New York, NY, pp 69–78

  30. Li W, Yu X, Zhou K, Song Y, Lin Z, Jia J (2022) SDM: spatial diffusion model for large hole image inpainting. arXiv preprint arXiv:2212.02963. https://doi.org/10.48550/arXiv.2212.02963

  31. Armanious K, Mecky Y, Gatidis S, Yang B (2019) Adversarial inpainting of medical image modalities. In: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, Brighton, UK, pp 3267–3271

  32. Armanious K, Kumar V, Abdulatif S, Hepp T, Yang B (2020) ipA-MedGAN: inpainting of arbitrarily regions in medical modalities. In: 2020 IEEE International Conference on Image Processing (ICIP). IEEE, Abu Dhabi, United Arab Emirates, pp 3005–3009

  33. Wang Q, Chen Y, Zhang N, Gu Y (2021) Medical image inpainting with edge and structure priors. Measurement 185:110027

    Article  Google Scholar 

  34. Tran M-T, Kim S-H, Yang H-J, Lee G-S (2021) Multi-task learning for medical image inpainting based on organ boundary awareness. Appl Sci 11(9):4247

    Article  CAS  Google Scholar 

  35. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells W, Frangi A (eds) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015. MICCAI 2015. Lecture Notes in Computer Science, vol 9351. Springer, Cham, pp 234–241. https://doi.org/10.1007/978-3-319-24574-4_28

  36. Isola P, Zhu J, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA, pp 5967–5976

  37. Miyato T, Kataoka T, Koyama M, Yoshida Y (2018) Spectral normalization for generative adversarial networks. In: International Conference on Learning Representations. ICLR, pp 1–26. https://openreview.net/forum?id=B1QRgziT-

  38. Hore A, Ziou D (2010) Image quality metrics: PSNR vs. SSIM. In: 2010 20th International Conference on Pattern Recognition. IEEE, Istanbul, Turkey, pp 2366–2369

  39. Sara U, Akter M, Uddin MS (2019) Image quality assessment through FSIM, SSIM, MSE and PSNR—a comparative study. J Comput Commun 7(3):8–18

    Article  Google Scholar 

  40. Dice LR (1945) Measures of the amount of ecologic association between species. Ecology 26(3):297–302

    Article  Google Scholar 

  41. Unkelbach J, Bortfeld T, Craft D, Alber M, Bangert M, Bokrantz R, Chen D, Li R, Xing L, Men C (2015) Optimization approaches to volumetric modulated arc therapy planning. Med Phys 42(3):1367–1377

    Article  PubMed  PubMed Central  Google Scholar 

  42. Fiandra C, Filippi AR, Catuzzo P, Botticella A, Ciammella P, Franco P, Borca VC, Ragona R, Tofani S, Ricardi U (2012) Different IMRT solutions vs. 3D-conformal radiotherapy in early stage Hodgkin’s Lymphoma: dosimetric comparison and clinical considerations. Radiat Oncol 7(1):1–9

    Article  Google Scholar 

  43. Ma P, Wang X, Xu Y, Dai J, Wang L (2014) Applying the technique of volume-modulated arc radiotherapy to upper esophageal carcinoma. J Appl Clin Med Phys 15(3):221–228

    Article  PubMed Central  Google Scholar 

  44. Zhang W-Z, Zhai T-T, Lu J-Y, Chen J-Z, Chen Z-J, Li D-R, Chen C-Z (2015) Volumetric modulated arc therapy vs. c-IMRT for the treatment of upper thoracic esophageal cancer. PLoS ONE 10(3):e0121385

    Article  PubMed  PubMed Central  Google Scholar 

  45. Paddick I (2000) A simple scoring ratio to index the conformity of radiosurgical treatment plans. J Neurosurg 93(supplement_3):219–222

    Article  PubMed  Google Scholar 

  46. Semenenko VA, Reitz B, Day E, Qi XS, Miften M, Li XA (2008) Evaluation of a commercial biologically based IMRT treatment planning system. Med Phys 35(12):5851–5860

    Article  PubMed  Google Scholar 

  47. Aziz NAA, Mubin M, Ibrahim Z, Nawawi SW (2015) Statistical analysis for swarm intelligence—simplified. Int J Future Comput Commun 4(3):193–197

    Article  Google Scholar 

  48. Huang Y, Preuhs A, Manhart M, Lauritsch G, Maier A (2021) Data extrapolation from learned prior images for truncation correction in computed tomography. IEEE Trans Med Imaging 99:1–12

    Google Scholar 

  49. Liu Y, Le Y, Wang T, Fu Y, Tang X, Curran W, Liu T, Patel P, Yang X (2020) CBCT-based synthetic CT generation using deep-attention cycleGAN for pancreatic adaptive radiotherapy. Med Phys 47(6):2472–2483

    Article  PubMed  Google Scholar 

Download references

Funding

This work is supported by the Changzhou Sci&Tech Program (Nos. CJ20210128 and CJ20200099), Jiangsu Provincial Key Research and Development Program Social Development Project (No. BE2022720), General Program of Jiangsu Provincial Health Commission (No. M2020006), Changzhou Key Laboratory of Medical Physics (No. CM20193005), Young Talent Development Plan of Changzhou Health Commission (Nos. CZQM2020075 and CZQM2020067), and the Science and Technology Programs for Young Talents of Changzhou Health Commission (No. QN201932).

Author information

Author notes

  1. Kai Xie and Liugang Gao have equal contribution.

Authors and Affiliations

  1. Radiotherapy Department, Second People’s Hospital of Changzhou, Nanjing Medical University, Changzhou, 213000, China

    Kai Xie, Liugang Gao, Jiawei Sun, Tao Lin, Jianfeng Sui & Xinye Ni

  2. Jiangsu Province Engineering Research Center of Medical Physics, Changzhou, 213000, China

    Kai Xie, Liugang Gao, Jiawei Sun, Tao Lin, Jianfeng Sui & Xinye Ni

  3. Center for Medical Physics, Nanjing Medical University, Changzhou, 213003, China

    Heng Zhang, Sai Zhang, Qianyi Xi, Fan Zhang & Xinye Ni

  4. Key Laboratory of Medical Physics, Changzhou, 213000, China

    Heng Zhang, Sai Zhang, Qianyi Xi, Fan Zhang & Xinye Ni

Authors
  1. Kai Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liugang Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Heng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sai Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qianyi Xi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Fan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jiawei Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Tao Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jianfeng Sui
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Xinye Ni
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Kai Xie and Liugang Gao contributed equally to this work. Kai Xie, Liugang Gao, and Xinye Ni collected the data and wrote the manuscript draft. Kai Xie, Liugang Gao, and Heng Zhang performed the training of different models. Jiawei Sun, Tao Lin, and Jianfeng Sui designed the VMAT plans. Sai Zhang, Qianyi Xi, and Fan analyzed the results. All authors discussed the results and have given approval to the final version of the manuscript.

Corresponding author

Correspondence to Xinye Ni.

Ethics declarations

Ethics approval and consent to participate

The retrospective study was approved by the Clinical Ethics Committee of Second People’s Hospital of Changzhou of Nanjing Medical University (#2020KY154-01). Written informed consent to participate in this study was not required in accordance with national and institutional guidelines.

Competing interests

The authors declare no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xie, K., Gao, L., Zhang, H. et al. Inpainting truncated areas of CT images based on generative adversarial networks with gated convolution for radiotherapy. Med Biol Eng Comput 61, 1757–1772 (2023). https://doi.org/10.1007/s11517-023-02809-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-023-02809-y

Keywords

Search

Navigation