Skip to main content

Advertisement

Springer Nature Link
Log in

MDVA-GAN: multi-domain visual attribution generative adversarial networks

  • S.I. : TAM-LHR
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Some pixels of an input image have thick information and give insights about a particular category during classification decisions. Visualization of these pixels is a well-studied problem in computer vision, called visual attribution (VA), which helps radiologists to recognize abnormalities and identify a particular disease in the medical image. In recent years, several classification-based techniques for domain-specific attribute visualization have been proposed, but these techniques can only highlight a small subset of most discriminative features. Therefore, their generated VA maps are inadequate to visualize all effects in an input image. Due to recent advancements in generative models, generative model-based VA techniques are introduced which generate efficient VA maps and visualize all affected regions. To deal the issue, generative adversarial network-based VA techniques are recently proposed, where the researchers leverage the advances in domain adaption techniques to learn a map for abnormal-to-normal medical image translation. As these approaches rely on a two-domain translation model, it would require training as many models as number of diseases in a medical dataset, which is a tedious and compute-intensive task. In this work, we introduce a unified multi-domain VA model that generates a VA map of more than one disease at a time. The proposed unified model gets images from a particular domain and its domain label as input to generate VA map and visualize all the affected regions by that particular disease. Experiments on the CheXpert dataset, which is a publicly available multi-disease chest radiograph dataset, and the TBX11K dataset show that the proposed model generates identical results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 2921–2929)

  2. Feng X, Yang J, Laine AF, Angelini ED (2017, September) Discriminative localization in CNNs for weakly-supervised segmentation of pulmonary nodules. In: International conference on medical image computing and computer-assisted intervention (pp 568–576). Springer, Cham

  3. Fong RC, Vedaldi A (2017) Interpretable explanations of black boxes by meaningful perturbation. In: Proceedings of the IEEE International Conference on Computer Vision (pp 3429–3437)

  4. Ge Z, Demyanov S, Chakravorty R, Bowling A, Garnavi R (2017, September) Skin disease recognition using deep saliency features and multimodal learning of dermoscopy and clinical images. In: International conference on medical image computing and computer-assisted intervention (pp 250–258). Springer, Cham. (year)

  5. Baumgartner CF, Kamnitsas K, Matthew J, Fletcher TP, Smith S, Koch LM, Rueckert D (2017) SonoNet: real-time detection and localisation of fetal standard scan planes in freehand ultrasound. IEEE Trans Med Imaging 36(11):2204–2215

    Article  Google Scholar 

  6. Sundararajan M, Taly A, Yan Q (2017, July) Axiomatic attribution for deep networks. In International Conference on Machine Learning (pp. 3319-3328). PMLR

  7. Shwartz-Ziv R, Tishby N (2017) Opening the black box of deep neural networks via information. arXiv:1703.00810

  8. Sun L, Wang J, Huang Y, Ding X, Greenspan H, Paisley J (2020) An adversarial learning approach to medical image synthesis for lesion detection. IEEE J Biomed Health Informatics 24(8):2303–2314

    Article  Google Scholar 

  9. Baumgartner CF, Koch LM, Tezcan KC, Ang JX, Konukoglu E (2018) Visual feature attribution using wasserstein gans. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 8309–8319)

  10. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Bengio Y (2014) Generative adversarial networks. arXiv:1406.2661

  11. Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision (pp 2223–2232)

  12. Arjovsky M, Chintala S, Bottou L (2017, July) Wasserstein generative adversarial networks. In: International conference on machine learning (pp 214–223). PMLR

  13. Choi Y, Choi M, Kim M, Ha JW, Kim S, Choo J (2018) Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 8789–8797)

  14. Choi Y, Uh Y, Yoo J, Ha JW (2020) Stargan v2: Diverse image synthesis for multiple domains. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp 8188–8197)

  15. He Z, Zuo W, Kan M, Shan S, Chen X (2019) Attgan: facial attribute editing by only changing what you want. IEEE Trans Image Process 28(11):5464–5478

    Article  MathSciNet  MATH  Google Scholar 

  16. Ping Q, Wu B, Ding W, Yuan J (2019) Fashion-AttGAN: Attribute-aware fashion editing with multi-objective GAN. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops (pp 0-0)

  17. Liu M, Ding Y, Xia M, Liu X, Ding E, Zuo W, Wen S (2019) STGAN: A unified selective transfer network for arbitrary image attribute editing. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 3673–3682)

  18. Huang X, Li Y, Poursaeed O, Hopcroft J, Belongie S (2017) Stacked generative adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 5077–5086)

  19. Karras T, Aila T, Laine S, Lehtinen J (2017) Progressive growing of gans for improved quality, stability, and variation. arXiv:1710.10196

  20. Isola P, Zhu JY, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 1125–1134)

  21. Kim T, Cha M, Kim H, Lee JK, Kim J (2017, July) Learning to discover cross-domain relations with generative adversarial networks. In: International conference on machine learning (pp 1857–1865). PMLR

  22. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Shi W (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 4681–4690)

  23. Kim T, Kim B, Cha M, Kim J (2017) Unsupervised visual attribute transfer with reconfigurable generative adversarial networks. arXiv:1707.09798

  24. Shen W, Liu R (2017) Learning residual images for face attribute manipulation. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 4030–4038)

  25. Nie D, Trullo R, Lian J, Petitjean C, Ruan S, Wang Q, Shen D (2017, September) Medical image synthesis with context-aware generative adversarial networks. In: International conference on medical image computing and computer-assisted intervention (pp 417–425). Springer, Cham

  26. Zhang Z, Yang L, Zheng Y (2018) Translating and segmenting multimodal medical volumes with cycle-and shape-consistency generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern Recognition (pp 9242–9251)

  27. Chartsias A, Joyce T, Giuffrida MV, Tsaftaris SA (2017) Multimodal MR synthesis via modality-invariant latent representation. IEEE Trans Med Imaging 37(3):803–814

    Article  Google Scholar 

  28. Cao B, Zhang H, Wang N, Gao X, Shen D (2020, April) Auto-GAN: self-supervised collaborative learning for medical image synthesis. In: Proceedings of the AAAI conference on artificial intelligence (Vol 34, No 07, pp 10486-10493)

  29. Zhang Y, Yang L, Chen J, Fredericksen M, Hughes DP, Chen DZ (2017, September) Deep adversarial networks for biomedical image segmentation utilizing unannotated images. In: International conference on medical image computing and computer-assisted intervention (pp 408–416). Springer, Cham

  30. Yang D, Xu D, Zhou SK, Georgescu B, Chen M, Grbic S, Comaniciu D (2017, September) Automatic liver segmentation using an adversarial image-to-image network. In: International conference on medical image computing and computer-assisted intervention (pp 507–515). Springer, Cham

  31. Yu K, Wang Y, Cai Y, Xiao C, Zhao E, Glass L, Sun J (2019) Rare disease detection by sequence modeling with generative adversarial networks. arXiv:1907.01022

  32. Park KB, Choi SH, Lee JY (2020) M-gan: Retinal blood vessel segmentation by balancing losses through stacked deep fully convolutional networks. IEEE Access 8:146308–146322

    Article  Google Scholar 

  33. Oquab M, Bottou L, Laptev I, Sivic J (2015) Is object localization for free?-weakly-supervised learning with convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 685–694)

  34. Gondal WM, Köhler JM, Grzeszick R, Fink GA, Hirsch M (2017, September) Weakly-supervised localization of diabetic retinopathy lesions in retinal fundus images. In: 2017 IEEE international conference on image processing (ICIP) (pp 2069–2073). IEEE

  35. Kim HE, Hwang S (2016) Deconvolutional feature stacking for weakly-supervised semantic segmentation. arXiv:1602.04984

  36. Zhang J, Bargal SA, Lin Z, Brandt J, Shen X, Sclaroff S (2018) Top-down neural attention by excitation backprop. Int J Comput Vis 126(10):1084–1102

    Article  Google Scholar 

  37. Gao Y, Noble JA (2017, September) Detection and characterization of the fetal heartbeat in free-hand ultrasound sweeps with weakly-supervised two-streams convolutional networks. In: International conference on medical image computing and computer-assisted intervention (pp 305–313). Springer, Cham

  38. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp 770–778)

  39. Ulyanov D, Vedaldi A, Lempitsky V (2016) Instance normalization: The missing ingredient for fast stylization. arXiv:1607.08022. Jul 27

  40. Irvin J, Rajpurkar P, Ko M, Yu Y, Ciurea-Ilcus S, Chute C, Ng AY (2019, July) Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI conference on artificial intelligence (Vol 33, No 01, pp 590–597)

  41. Liu Y, Wu YH, Ban Y, Wang H, Cheng MM (2020) Rethinking computer-aided tuberculosis diagnosis. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp 2646–2655)

  42. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  43. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision (pp 618–626)

  44. Rudin C (2019) Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Mach Intell 1(5):206–215

    Article  Google Scholar 

  45. Zintgraf LM, Cohen TS, Adel T, Welling M (2017) Visualizing deep neural network decisions: Prediction difference analysis. arXiv:1702.04595

Download references

Author information

Authors and Affiliations

  1. Medical Imaging and Diagnostic Lab, National Center of Artificial Intelligence, COMSATS University Islamabad, Islamabad, Pakistan

    Muhammad Nawaz

  2. College of Technological Innovation, Zayed University, Abu Dhabi, United Arab Emirates

    Feras Al-Obeidat & Abdallah Tubaishat

  3. Department of Computer Science, COMSATS University Islamabad, Islamabad, Pakistan

    Tehseen Zia

  4. Department of Computer Science, University of Sargodha, Sargodha, Pakistan

    Fahad Maqbool

  5. University of Lisbon, ISEG, Rua do Quelhas, No 6, 1200-781, Lisboa, Portugal

    Alvaro Rocha

Authors
  1. Muhammad Nawaz
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Feras Al-Obeidat
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Abdallah Tubaishat
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Tehseen Zia
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Fahad Maqbool
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Alvaro Rocha
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Tehseen Zia.

Ethics declarations

Conflict of Interest

Muhammad Nawaz, Feras Al-Obeidat, Abdallah Tubaishat, Tehseen Zia, Fahad Maqbool, and Alvaro Rocha declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nawaz, M., Al-Obeidat, F., Tubaishat, A. et al. MDVA-GAN: multi-domain visual attribution generative adversarial networks. Neural Comput & Applic 35, 8035–8050 (2023). https://doi.org/10.1007/s00521-022-06969-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-06969-0

Keywords