Skip to main content

Advertisement

SpringerLink
Log in

HFANet: hierarchical feature fusion attention network for classification of glomerular immunofluorescence images

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The chronic kidney disease (CKD) accompanied by permanent kidney damage, has become a heavy burden for worldwide public health. Clinically, glomerular immunofluorescence (IF) images are widely-used to reveal the occurrence probability and type of CKD. In histopathological assessment for glomerular IF image, multiple descriptive indicators are used to characterize deposits from different aspects, which suggest associated kidney lesions. In this paper, we design a hierarchical feature fusion attention network (HFANet) to classify two main descriptive indicators, namely fluorescence intensity and distribution pattern. Through the hierarchical feature fusion attention (HFA) module, HFANet supplements deep semantic features using shallow texture features to maximize its feature extraction capability and efficiency of information fusion. Different from directly adding or concatenating, HFANet weighted concatenates the feature maps from different hierarchies to highlight more discriminative regions. Further, by integrating HFANet with the proposed intensity equalization (IE) algorithm, U-Net++, and Grad-CAM, a computer-aided diagnostic system for glomerular IF images is constructed. With this system, the classification accuracy of the fluorescence intensity and distribution pattern reaches 90.48% and 90.87%, respectively. Extensive comparative experiments and ablation studies demonstrate that HFANet outperforms other universal backbones with the help of HFA module, and the classification performance of the devised system is comparable to senior pathologists. The heatmap given by the system, which is similar to the classification evidence used by the clinicians, can be used as diagnostic reference and training material for pathologists. The systematic demonstration video is available in the supplementary material.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Zhang L et al (2012) Prevalence of chronic kidney disease in china: a cross-sectional survey. lancet 379(9818):815–822

    Article  Google Scholar 

  2. Johansen KL et al (2021) Us renal data system 2020 annual data report: Epidemiology of kidney disease in the united states. Am J Kidney Dis 77(4 Supplement 1):7–8

    Article  Google Scholar 

  3. Goolsby MJ (2010) National kidney foundation guidelines for chronic kidney disease: evaluation, classification, and stratification. J Am Assoc Nurse Pract 14(6):238–242

    Article  Google Scholar 

  4. Eckardt K-U, Coresh J, Devuyst O, Johnson RJ, Köttgen A, Levey AS, Levin A (2013) Evolving importance of kidney disease: from subspecialty to global health burden. Lancet 382(9887):158–169

    Article  Google Scholar 

  5. Coresh J, Astor BC, Greene T, Eknoyan G, Levey AS (2003) Prevalence of chronic kidney disease and decreased kidney function in the adult us population: third national health and nutrition examination survey. Am J Kidney Dis 41(1):1–12

    Article  Google Scholar 

  6. Brown WW, Peters RM, Ohmit SE, Keane WF, Collins A, Chen S-C, King K, Klag MJ, Molony DA, Flack JM (2003) Early detection of kidney disease in community settings: the kidney early evaluation program (keep). Am J Kidney Dis 42(1):22–35

    Article  Google Scholar 

  7. Anothaisintawee T, Rattanasiri S, Ingsathit A, Attia J, Thakkinstian A (2009) Prevalence of chronic kidney disease: a systematic review and meta-analysis. Clin Nephrol 71(3):244–254

    Article  Google Scholar 

  8. Sedor JR (2009) Tissue proteomics: a new investigative tool for renal biopsy analysis. Kidney Int 75(9):876–879

    Article  Google Scholar 

  9. Fogo AB, Cohen AH, Colvin RB, Jennette JC, Alpers CE (2007) Fundamentals of Renal Pathology.

  10. Walker PD, Cavallo T, Bonsib SM (2004) Practice guidelines for the renal biopsy. Mod Pathol 17(12):1555–1563

    Article  Google Scholar 

  11. Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak JAWM, van Ginneken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88

    Article  Google Scholar 

  12. Nogales A, García-Tejedor Á, Monge D, Vara JS, Antón C (2021) A survey of deep learning models in medical therapeutic areas. Artificial Intelligence in Medicine, 102020

  13. Andre E, Chou K, Yeung S, Nikhil N, Madani A, Mottaghi A, Liu Y, Topol E, Dean J, Socher R (2021) Deep learning-enabled medical computer vision. NPJ Digit Med 4(1):1–9

    Google Scholar 

  14. Zeng C et al (2020) Identification of glomerular lesions and intrinsic glomerular cell types in kidney diseases via deep learning. J Pathol 252(1):53–64

    Article  Google Scholar 

  15. Ginley B et al (2019) Computational segmentation and classification of diabetic glomerulosclerosis. J Am Soc Nephrol 30(10):1953–1967

    Article  Google Scholar 

  16. Hermsen M et al (2019) Deep learning-based histopathologic assessment of kidney tissue. J Am Soc Nephrol 30(10):1968–1979

    Article  Google Scholar 

  17. Jayapandian CP et al (2021) Development and evaluation of deep learning-based segmentation of histologic structures in the kidney cortex with multiple histologic stains. Kidney Int 99(1):86–101

    Article  Google Scholar 

  18. Bueno G, Fernandez-Carrobles MM, Gonzalez-Lopez L, Deniz O (2020) Glomerulosclerosis identification in whole slide images using semantic segmentation. Comput Methods Programs Biomed 184:105273

    Article  Google Scholar 

  19. Ligabue G et al (2020) Evaluation of the classification accuracy of the kidney biopsy direct immunofluorescence through convolutional neural networks. Clin J Am Soc Nephrol 15(10):1445–1454

    Article  Google Scholar 

  20. D’Agati VD, Mengel M (2013) The rise of renal pathology in nephrology: structure illuminates function. Am J Kidney Dis 61(6):1016–1025

    Article  Google Scholar 

  21. Chang A, Gibson IW, Cohen AH, Weening JW, Jennette JC, Fogo AB (2012) A position paper on standardizing the nonneoplastic kidney biopsy report. Hum Pathol 43(8):1192–1196

    Article  Google Scholar 

  22. Mise K et al (2014) Clinical implications of linear immunofluorescent staining for immunoglobulin g in patients with diabetic nephropathy. Diabetes Res Clin Pract 106(3):522–530

    Article  Google Scholar 

  23. Zhao K, Tang YJJ, Zhang T, Carvajal J, Smith DF, Wiliem A, Hobson P, Jennings A, Lovell BC (2018) Dgdi: A dataset for detecting glomeruli on renal direct immunofluorescence. In: 2018 Digital Image Computing: Techniques and Applications (DICTA), pp 1–7. IEEE

  24. Kitamura S, Takahashi K, Sang Y, Fukushima K, Tsuji K, Wada J (2020) Deep learning could diagnose diabetic nephropathy with renal pathological immunofluorescent images. Diagnostics 10(7):466

    Article  Google Scholar 

  25. Wang F, Tax DM (2016) Survey on the attention based rnn model and its applications in computer vision. arXiv preprint arXiv:1601.06823

  26. Bahdanau D, Cho K, Bengio Y (2014) Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473

  27. Chaudhari S, Mithal V, Polatkan G, Ramanath R (2019) An attentive survey of attention models. arXiv preprint arXiv:1904.02874

  28. Fu J, Zheng H, Mei T (2017) Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.4438–4446

  29. Wang F, Jiang M, Qian C, Yang S, Li C, Zhang H, Wang X, Tang X (2017) Residual attention network for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3156–3164

  30. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7132–7141

  31. Woo S, Park J, Lee JY, Kweon IS (2018) Cbam: convolutional block attention module. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 3–19

  32. He A, Li T, Li N, Wang K, Fu H (2020) Cabnet: category attention block for imbalanced diabetic retinopathy grading. IEEE Trans Med Imaging 40(1):143–153

    Article  Google Scholar 

  33. Gu R, Wang G, Song T, Huang R, Aertsen M, Deprest J, Ourselin S, Vercauteren T, Zhang S (2020) Ca-net: comprehensive attention convolutional neural networks for explainable medical image segmentation. IEEE Trans Med Imaging 40(2):699–711

    Article  Google Scholar 

  34. Zhang Q, Wu YN, Zhu SC (2018) Interpretable convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 8827–8836

  35. Kong T, Yao A, Chen Y, Sun F (2016) Hypernet: Towards accurate region proposal generation and joint object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 845–853

  36. Bell S, Zitnick CL, Bala K, Girshick R (2016) Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2874–2883

  37. Lin TY, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2117–2125

  38. Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J (2019) Unet++: redesigning skip connections to exploit multiscale features in image segmentation. IEEE Trans Med Imaging 39(6):1856–1867

    Article  Google Scholar 

  39. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  40. Lin M, Chen Q, Yan S (2013) Network in network. arXiv preprint arXiv:1312.4400

  41. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. biometrics, pp 159–174

  42. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3431–3440

  43. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation, 234–241. Springer

  44. Zhou Z, Siddiquee MMR, Tajbakhsh N, Liang J (2019) Unet++: redesigning skip connections to exploit multiscale features in image segmentation. IEEE Trans Med Imaging 39(6):1856–1867

    Article  Google Scholar 

  45. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4510–4520

  46. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 770–778

  47. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2017) Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4700–4708

  48. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp 618–626

  49. Menck PJ, Heitzig J, Kurths J, Schellnhuber HJ (2014) How dead ends undermine power grid stability. Nat Commun 5(1):1–8

    Article  Google Scholar 

Download references

Funding

This work was supported in part by the National Natural Science Foundation of China under Grant Nos. 61971111, 62027803, 61601096, 61801089, and 61701095; and in part by the Department of Science and Technology of Sichuan Province, China under Grant Nos. 2020YFG0044, 2021YFG0200, 2020YFG0046, 2021YJ0144, 2019YFS0538 and 2018YSKY0017-12; and in part by the Medico-Engineering Cooperation Funds from University of Electronic Science and Technology of China under Grant No. ZYGX2021YGLH223.

Author information

Authors and Affiliations

  1. School of Automation Engineering, University of Electronic Science and Technology of China, No.2006, Xiyuan Ave, West Hi-Tech Zone, Chengdu, 611731, Sichuan, China

    Haoran Liu, Yongle Xie, Xifeng Li & Dongjie Bi

  2. Department of Nephrology, Sichuan Provincial People’s Hospital, University of Electronic Science and Technology of China, No.2006, Xiyuan Ave, West Hi-Tech Zone, Chengdu, 611731, Sichuan, China

    Ping Zhang, Yurong Zou, Lei Peng & Guisen Li

Authors
  1. Haoran Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ping Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yongle Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xifeng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dongjie Bi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yurong Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Lei Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Guisen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yongle Xie or Lei Peng.

Ethics declarations

Conflict of interest

The authors have no conflict of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (m4v 3485 KB)

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, H., Zhang, P., Xie, Y. et al. HFANet: hierarchical feature fusion attention network for classification of glomerular immunofluorescence images. Neural Comput & Applic 34, 22565–22581 (2022). https://doi.org/10.1007/s00521-022-07676-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07676-6

Keywords

Search

Navigation