Skip to main content
SpringerLink
Log in

Multi-national CT image-label pairs synthesis for COVID-19 diagnosis via few-shot generative adversarial networks adaptation

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Coronavirus disease 2019 (COVID-19) is an ongoing global pandemic that has threatened health worldwide for years, necessitating accurate and rapid diagnostic techniques given its high pathogenicity and transmissibility. Numerous deep learning models have been developed to assist radiologists in chest computed tomography (CT)-based COVID-19 diagnosis. However, existing COVID-19 CT datasets is often characterized by significant geographic and class imbalances, which impede the model’s ability to generalize effectively across varying patient cohorts. With the advancements in generative adversarial networks (GAN), one potential solution to the problem is to synthesize data for the target datasets by leveraging a large-scale source dataset for pre-training (i.e., few-shot GAN adaptation). To calibrate the target GAN during adaptation, we incorporate contrastive learning coupled with LeCam regularization, ensuring that the diversity inherent in the source dataset is preserved. Additionally, overfitting is mitigated through the use of consistency regularization and differentiable augmentation techniques. Also, we incorporate off-the-shelf vision models into the discriminator ensemble, predicated on the linear separability between real and fake samples in the feature space, thereby encouraging the generator to match the real distribution in different, complementary feature spaces. We demonstrate the effectiveness of our approach in improving COVID-19 diagnostic performance by generating realistic and diverse CT image-label pairs for the target datasets and show that it consistently outperforms the state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Acar E, Şahin E, Yılmaz İ (2021) Improving effectiveness of different deep learning-based models for detecting Covid-19 from computed tomography (CT) images. Neural Comput Appl 33:17589–17609

    Article  PubMed  PubMed Central  Google Scholar 

  2. Afshar P, Heidarian S, Enshaei N, Naderkhani F, Rafiee MJ, Oikonomou A, Fard FB, Samimi K, Plataniotis KN, Mohammadi A (2021) Covid-CT-md, Covid-19 computed tomography scan dataset applicable in machine learning and deep learning. Sci Data 8:1–8

    Article  CAS  Google Scholar 

  3. Arjovsky M, Chintala S, Bottou L (2017) Wasserstein generative adversarial networks. In: International conference on machine learning PMLR. pp. 214–223

  4. Bai HX, Hsieh B, Xiong Z, Halsey K, Choi JW, Tran TML, Pan I, Shi LB, Wang DC, Mei J et al (2020) Performance of radiologists in differentiating Covid-19 from non-Covid-19 viral pneumonia at chest CT. Radiology 296:E46–E54

    Article  PubMed  Google Scholar 

  5. Brock A, Donahue J, Simonyan K (2018) Large scale GAN training for high fidelity natural image synthesis. In: International conference on learning representations

  6. Chen J, Li Y, Guo L, Zhou X, Zhu Y, He Q, Han H, Feng Q (2022) Machine learning techniques for CT imaging diagnosis of novel Coronavirus pneumonia: a review. Neural Comput Appl 1–19

  7. Fang Y, Zhang H, Xie J, Lin M, Ying L, Pang P, Ji W (2020) Sensitivity of chest CT for Covid-19: comparison to RT-PCR. Radiology 296:e115

    Article  PubMed  Google Scholar 

  8. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Adv Neural Inform Process Syst pp. 2672–2680

  9. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2020) Generative adversarial networks. Commun ACM 63:139–144

    Article  Google Scholar 

  10. Gu Z, Li W, Huo J, Wang L, Gao Y (2021) Lofgan: fusing local representations for few-shot image generation. In: Proceedings of the IEEE/CVF international conference on computer vision pp. 8463–8471

  11. Harmon SA, Sanford TH, Xu S, Turkbey EB, Roth H, Xu Z, Yang D, Myronenko A, Anderson V, Amalou A et al (2020) Artificial intelligence for the detection of Covid-19 pneumonia on chest CT using multinational datasets. Nat Commun 11:1–7

    Article  Google Scholar 

  12. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition pp. 770–778

  13. He X, Wang S, Chu X, Shi S, Tang J, Liu X, Yan C, Zhang J, Ding G (2021a) Automated model design and benchmarking of deep learning models for Covid-19 detection with chest CT scans. In: Proceedings of the AAAI conference on artificial intelligence pp. 4821–4829

  14. He X, Wang S, Ying G, Zhang J, Chu X (2021b) Efficient multi-objective evolutionary 3d neural architecture search for Covid-19 detection with chest CT scans. arXiv preprint arXiv:2101.10667

  15. Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in neural information processing systems, pp. 6626–6637

  16. Hong Y, Niu L, Zhang J, Zhang L (2022) Deltagan: Towards diverse few-shot image generation with sample-specific delta. In: ECCV

  17. Hou L, Cao Q, Shen H, Pan S, Li X, Cheng X (2022) Conditional GANS with auxiliary discriminative classifier. In: International conference on machine learning PMLR. pp. 8888–8902

  18. Jiang Y, Chen H, Loew M, Ko H (2020) Covid-19 CT image synthesis with a conditional generative adversarial network. IEEE J Biomed Health Inform 25:441–452

    Article  Google Scholar 

  19. Kang M, Shim W, Cho M, Park J (2021) Rebooting acgan: auxiliary classifier GANS with stable training. Adv Neural Inf Process Syst 34:23505–23518

    Google Scholar 

  20. Karras T, Aittala M, Hellsten J, Laine S, Lehtinen J, Aila T (2020) Training generative adversarial networks with limited data. Adv Neural Inf Process Syst 33:12104–12114

    Google Scholar 

  21. Karras T, Aittala M, Laine S, Härkönen E, Hellsten J, Lehtinen J, Aila T (2021) Alias-free generative adversarial networks. Adv Neural Inf Process Syst 34:852–863

    Google Scholar 

  22. Karras T, Laine S, Aittala M, Hellsten J, Lehtinen J, Aila T (2020b) Analyzing and improving the image quality of stylegan. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition pp. 8110–8119

  23. Khurana Y, Soni U (2022) Leveraging deep learning for Covid-19 diagnosis through chest imaging. Neural Comput Appl 34:14003–14012

    Article  PubMed  PubMed Central  Google Scholar 

  24. Kumari N, Zhang R, Shechtman E, Zhu JY (2022) Ensembling off-the-shelf models for gan training. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10651–10662

  25. Li Y, Wu CY, Fan H, Mangalam K, Xiong B, Malik J, Feichtenhofer C (2022) Mvitv2: improved multiscale vision transformers for classification and detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition pp. 4804–4814

  26. Li Y, Zhang R, Lu JC, Shechtman E (2020) Few-shot image generation with elastic weight consolidation. Adv Neural Inf Process Syst 33:15885–15896

    Google Scholar 

  27. Liu MY, Huang X, Mallya A, Karras T, Aila T, Lehtinen J, Kautz J (2019) Few-shot unsupervised image-to-image translation. In: Proceedings of the IEEE/CVF international conference on computer vision pp. 10551–10560

  28. Liu Z, Hu H, Lin Y, Yao Z, Xie Z, Wei Y, Ning J, Cao Y, Zhang Z, Dong L et al. (2022a) Swin transformer v2: scaling up capacity and resolution. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition pp. 12009–12019

  29. Liu Z, Lin Y, Cao Y, Hu H, Wei, Y., Zhang, Z., Lin, S., Guo, B., 2021. Swin transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012–10022

  30. Liu Z, Lin Y, Cao Y, Hu H, Wei Y, Zhang Z, Lin S, Guo B (2021) Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision pp. 10012–10022

  31. Meng L, Dong D, Li L, Niu M, Bai Y, Wang M, Qiu X, Zha Y, Tian J (2020) A deep learning prognosis model help alert for Covid-19 patients at high-risk of death: a multi-center study. IEEE J Biomed Health Inform 24:3576–3584

    Article  PubMed  Google Scholar 

  32. Mirza M, Osindero S (2014) Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784

  33. Miyato T, Kataoka T, Koyama M, Yoshida Y (2018) Spectral normalization for generative adversarial networks. In: International conference on learning representations

  34. Miyato T, Koyama M (2018) CGANS with projection discriminator. In: International conference on learning representation

  35. Mo S, Cho M, Shin J (2020) Freeze the discriminator: a simple baseline for fine-tuning GANS. In: CVPR AI for content creation workshop

  36. Ning W, Lei S, Yang J, Cao Y, Jiang P, Yang Q, Zhang J, Wang X, Chen F, Geng Z et al (2020) Open resource of clinical data from patients with pneumonia for the prediction of Covid-19 outcomes via deep learning. Nat Biomed Eng 4:1197–1207

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  37. Noguchi A, Harada T (2019) Image generation from small datasets via batch statistics adaptation. In: Proceedings of the IEEE/CVF international conference on computer vision pp. 2750–2758

  38. Ojha U, Li Y, Lu J, Efros A A, Lee YJ, Shechtman E, Zhang R (2021) Few-shot image generation via cross-domain correspondence. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition pp. 10743–10752

  39. Parmar G, Zhang R, Zhu JY (2022) On buggy resizing libraries and surprising subtleties in fid calculation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

  40. Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434

  41. Rahimzadeh M, Attar A, Sakhaei SM (2021) A fully automated deep learning-based network for detecting Covid-19 from a new and large lung CT scan dataset. Biomed Signal Process Control 68:102588

    Article  PubMed  PubMed Central  Google Scholar 

  42. Rao Y, Zhao W, Tang Y, Zhou J, Lim SN, Lu J (2022) Hornet: efficient high-order spatial interactions with recursive gated convolutions. Adv Neural Inf Process Syst 35:10353–10366

    Google Scholar 

  43. Revel MP, Boussouar S, de Margerie-Mellon C, Saab I, Lapotre T, Mompoint D, Chassagnon G, Milon A, Lederlin M, Bennani S et al. (2021) Study of thoracic CT in Covid-19: the stoic project. Radiology

  44. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510–4520

  45. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision pp. 618–626

  46. Tan M, Le Q (2019) Efficientnet: rethinking model scaling for convolutional neural networks. In: International conference on machine learning, PMLR. pp. 6105–6114

  47. Tseng HY, Jiang L, Liu C, Yang MH, Yang W (2021) Regularizing generative dversarial networks under limited data. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7921–7931

  48. Tuinstra T, Gunraj H, Wong A (2022) Covidx ct-3: A large-scale, multinational, open-source benchmark dataset for computer-aided Covid-19 screening from chest CT images. arXiv preprint arXiv:2206.03043

  49. Wang Y, Wu C, Herranz L, van de Weijer J, Gonzalez-Garcia A, Raducanu B (2018) Transferring GANS: generating images from limited data. In: Proceedings of the European conference on computer vision (ECCV), pp. 218–234

  50. Xiao J, Li L, Wang C, Zha ZJ, Huang Q (2022) Few shot generative model adaption via relaxed spatial structural alignment. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition pp. 11204–11213

  51. Xie X, Zhong Z, Zhao W, Zheng C, Wang F, Liu J (2020) Chest CT for typical 2019-NCOV pneumonia: relationship to negative RT-PCR testing. Radiology

  52. Zhang K, Liu X, Shen J, Li Z, Sang Y, Wu X, Zha Y, Liang W, Wang C et al (2020) Clinically applicable AI system for accurate diagnosis, quantitative measurements, and prognosis of Covid-19 pneumonia using computed tomography. Cell 181:1423–1433

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  53. Zhao S, Liu Z, Lin J, Zhu JY, Han S (2020) Differentiable augmentation for data-efficient GAN training. Adv Neural Inf Process Syst 33:7559–7570

    Google Scholar 

  54. Zhao Y, Ding H, Huang H, Cheung NM (2022) A closer look at few-shot image generation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition pp. 9140–9150

  55. Zhao Z, Singh S, Lee H, Zhang Z, Odena A, Zhang H (2021) Improved consistency regularization for GANS. In: Proceedings of the AAAI conference on artificial intelligence, pp 11033–11041

  56. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision, pp 2223–2232

  57. Zhang J, Xie Y, Sun D, Huang R, Wang T, Lei B, Chen K (2023) Multi-national COVID-19 CT image-label pairs synthesis via few-shot GANs adaptation. In: 2023 IEEE 20th International Symposium on Biomedical Imaging, pp 1–4

Download references

Acknowledgements

This work was supported partly by Science and Technology Plan Project of Guizhou Province (Qiankehe Support [2020] No. 4Y179), National Natural Science Foundation of China (No. 82260341), Medical Science and Technology Research Fund Project of Guangdong Province (B2022144), Science and Technology Plan Fund of Guizhou Provincial (Qiankehe Foundation-ZK [2022] General 634), and Shenzhen University-Lingnan University Joint Research Programme (SZU-LU006/2122).

Author information

Author notes

  1. Jing Zhang and Yingpeng Xie contributed equally to this work.

Authors and Affiliations

  1. Department of Radiology, The Fifth Affiliated Hospital of Zunyi Medical University, Zhuhai, 519000, Guangdong, China

    Jing Zhang, Dandan Sun, Ruidong Huang & Kuntao Chen

  2. National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, Guangdong Key Laboratory for Biomedical Measurements and Ultrasound Imaging, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, 518055, Guangdong, China

    Yingpeng Xie, Tianfu Wang & Baiying Lei

Authors
  1. Jing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yingpeng Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dandan Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ruidong Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tianfu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Baiying Lei
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Kuntao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kuntao Chen.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest to this work.

Data availability statements

All data analyzed during this study are included in this published article [48].

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Xie, Y., Sun, D. et al. Multi-national CT image-label pairs synthesis for COVID-19 diagnosis via few-shot generative adversarial networks adaptation. Neural Comput & Applic 36, 5007–5019 (2024). https://doi.org/10.1007/s00521-023-09317-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-09317-y

Keywords

Search

Navigation