Skip to main content
SpringerLink
Log in

A Novel original feature fusion network for joint diabetic retinopathy and diabetic Macular edema grading

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Diabetic retinopathy (DR) and its complication diabetic macular edema (DME) are the leading cause of permanent blindness in the working-age population worldwide. Automated grading of DR and DME enables ophthalmologists to carry out tailored treatments to patients in early stages of their diseases. However, most of the current works only focus on the grading of a single disease, ignoring the relationship between DR and DME, and the traditional convolutional architectures face the problem that they can not capture long-distance dependencies despite of the effectiveness of extracting image features. To this end, we propose an original feature fusion network (OFFNet) for joint DR and DME grading based on the idea of key-value query, which consists of a specific feature extraction module (SFEM) based on self-attention and an original feature fusion module (OFFM) based on cross-attention. The proposed OFFNet enjoys several merits. First, to the best of our knowledge, this is the first joint grading effort based on the idea of key-value query. Second, OFFNet only needs image-level supervision, which can facilitate the acquisition of training data, rather than patch-level or pixel-level supervision. Third, OFFNet has obvious advantages in capturing long-distance dependencies. Extensive experiments on two public datasets Messidor and 2018 IDRiD challenge show that our method outperforms other joint grading methods on joint grading accuracy and the ability of capturing long-distance dependencies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availibility

Data will be made available on reasonable request.

References

  1. Mathews MR, Anzar S (2021) A comprehensive review on automated systems for severity grading of diabetic retinopathy and macular edema. Int J Imaging Syst Technol 31(4):2093–2122

    Article  Google Scholar 

  2. Yang Y, Shang F, Wu B, Yang D, Wang L, Xu Y, Zhang W, Zhang T (2021) Robust collaborative learning of patch-level and image-level annotations for diabetic retinopathy grading from fundus image. IEEE Trans Cybern 155(11):11407–11417

    Article  Google Scholar 

  3. Wilkinson CP, Iii FLF, Klein RE, Lee PP, Agardh CD, Davis M, Dills D, Kampik A, Pararajasegaram R, Verdaguer JT (2003) Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology 110(9):1677–1682

    Article  Google Scholar 

  4. Li X, Hu X, Yu L, Zhu L, Fu C-W, Heng P-A (2019) Canet: cross-disease attention network for joint diabetic retinopathy and diabetic macular edema grading. IEEE Trans Med Imaging 39(5):1483–1493

    Article  Google Scholar 

  5. The Eye M.D. Association, American Academy of Ophthalmology (2002) International clinical diabetic retinopathy disease severity scale detailed table. American Academy of Ophthalmology, San Francisco, CA, USA, Technical Report

  6. Sun R, Li Y, Zhang T, Mao Z, Wu F, Zhang Y (2021) Lesion-aware transformers for diabetic retinopathy grading. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10 938–10 947

  7. Das A, McGuire PG, Rangasamy S (2015) Diabetic macular edema: pathophysiology and novel therapeutic targets. Ophthalmology 122(7):1375–1394

    Article  Google Scholar 

  8. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  9. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778

  10. Howard A, Sandler M, Chu G, Chen L-C, Chen B, Tan M, Wang W, Zhu Y, Pang R, Vasudevan V et al. (2019) Searching for mobilenetv3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324

  11. Wang J, Sun K, Cheng T, Jiang B, Deng C, Zhao Y, Liu D, Mu Y, Tan M, Wang X et al (2020) Deep high-resolution representation learning for visual recognition. IEEE Trans Pattern Anal Mach Intell 43(10):3349–3364

    Article  Google Scholar 

  12. Lei B, Liang E, Yang M, Yang P, Zhou F, Tan EL, Lei Y, Liu CM, Wang T, Xiao X (2022) Predicting clinical scores for alzheimer’s disease based on joint and deep learning. Expert Syst Appl 187:115966

    Article  Google Scholar 

  13. Abràmoff MD, Lou Y, Erginay A, Clarida W, Amelon R, Folk JC, Niemeijer M (2016) Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Investig Ophthalmol Visual Sci 57(13):5200–5206

    Article  Google Scholar 

  14. Yang Y, Li T, Li W, Wu H, Fan W, Zhang W (2017) Lesion detection and grading of diabetic retinopathy via two-stages deep convolutional neural networks. In: International conference on medical image computing and computer-assisted intervention. Springer, pp. 533–540

  15. Gondal WM, Köhler JM, Grzeszick R, Fink GA, Hirsch M (2017) Weakly-supervised localization of diabetic retinopathy lesions in retinal fundus images. In: 2017 IEEE international conference on image processing (ICIP).IEEE, pp. 2069–2073

  16. Foo A, Hsu W, Lee ML, Lim G, Wong TY (2020) Multi-task learning for diabetic retinopathy grading and lesion segmentation. In: Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 08, pp. 13 267–13 272

  17. Mo J, Zhang L, Feng Y (2018) Exudate-based diabetic macular edema recognition in retinal images using cascaded deep residual networks. Neurocomputing 290:161–171

    Article  Google Scholar 

  18. He X, Zhou Y, Wang B, Cui S, Shao L (2019) Dme-net: Diabetic macular edema grading by auxiliary task learning. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 788–796

  19. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, Venugopalan S, Widner K, Madams T, Cuadros J et al (2016) Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316(22):2402–2410

    Article  Google Scholar 

  20. Krause J, Gulshan V, Rahimy E, Karth P, Widner K, Corrado GS, Peng L, Webster DR (2018) Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 125(8):1264–1272

    Article  Google Scholar 

  21. Tu Z, Gao S, Zhou K, Chen X, Fu H, Gu Z, Cheng J, Yu Z, Liu J (2020) Sunet: a lesion regularized model for simultaneous diabetic retinopathy and diabetic macular edema grading. In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI).IEEE, pp. 1378–1382

  22. Silberman N, Ahrlich K, Fergus R, Subramanian L (2010) Case for automated detection of diabetic retinopathy. In: 2010 AAAI spring symposium series

  23. Sopharak A, Uyyanonvara B, Barman S (2009) Automatic exudate detection from non-dilated diabetic retinopathy retinal images using fuzzy c-means clustering. Sensors 9(3):2148–2161

    Article  Google Scholar 

  24. Roychowdhury S, Koozekanani DD, Parhi KK (2013) Dream: diabetic retinopathy analysis using machine learning. IEEE J Biomed Health Inform 18(5):1717–1728

    Article  Google Scholar 

  25. Acharya UR, Lim CM, Ng EYK, Chee C, Tamura T (2009) Computer-based detection of diabetes retinopathy stages using digital fundus images. Proc Inst Mech Eng [H] 223(5):545–553

    Article  Google Scholar 

  26. Akram MU, Khalid S, Tariq A, Khan SA, Azam F (2014) Detection and classification of retinal lesions for grading of diabetic retinopathy. Comput Biol Med 45:161–171

    Article  Google Scholar 

  27. Antal B, Hajdu A (2014) An ensemble-based system for automatic screening of diabetic retinopathy. Knowl-Based Syst 60:20–27

    Article  Google Scholar 

  28. Seoud L, Hurtut T, Chelbi J, Cheriet F, Langlois JP (2015) Red lesion detection using dynamic shape features for diabetic retinopathy screening. IEEE Trans Med Imaging 35(4):1116–1126

    Article  Google Scholar 

  29. Kumar N, Rajwade AV, Chandran S, Awate SP (2017) “Kernel generalized-gaussian mixture model for robust abnormality detection. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 21–29

  30. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, vol. 25,

  31. Gargeya R, Leng T (2017) Automated identification of diabetic retinopathy using deep learning. Ophthalmology 124(7):962–969

    Article  Google Scholar 

  32. Wang Z, Yin Y, Shi J, Fang W, Li H, Wang X (2017) Zoom-in-net: Deep mining lesions for diabetic retinopathy detection. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 267–275

  33. Li X, Shen L, Shen M, Tan F, Qiu CS (2019) Deep learning based early stage diabetic retinopathy detection using optical coherence tomography. Neurocomputing 369:134–144

    Article  Google Scholar 

  34. Wang S, Wang X, Hu Y, Shen Y, Lei B (2020) Diabetic retinopathy diagnosis using multichannel generative adversarial network with semisupervision’’. IEEE Trans Autom Sci Eng 99:1–12

    Google Scholar 

  35. Lin Z, Guo R, Wang Y, Wu B, Chen T, Wang W, Chen DZ, Wu J (2018) A framework for identifying diabetic retinopathy based on anti-noise detection and attention-based fusion. In: International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer, pp. 74–82

  36. Zhou Y, He X, Huang L, Liu L, Zhu F, Cui S, Shao L (2019) Collaborative learning of semi-supervised segmentation and classification for medical images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2079–2088

  37. Wang X, Xu M, Zhang J, Jiang L, Li L (2021) Deep multi-task learning for diabetic retinopathy grading in fundus images. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1–5

  38. Akram MU, Tariq A, Khan SA, Javed MY (2014) Automated detection of exudates and macula for grading of diabetic macular edema. Comput Methods Programs Biomed 114(2):141–152

    Article  Google Scholar 

  39. Acharya UR, Mookiah MRK, Koh JE, Tan JH, Bhandary SV, Rao AK, Hagiwara Y, Chua CK, Laude A (2017) Automated diabetic macular edema (dme) grading system using dwt, dct features and maculopathy index. Comput Biol Med 84:59–68

    Article  Google Scholar 

  40. Ren F, Cao P, Zhao D, Wan C (2018) Diabetic macular edema grading in retinal images using vector quantization and semi-supervised learning. Technol Health Care 26(S1):389–397

    Article  Google Scholar 

  41. Syed AM, Akram MU, Akram T, Muzammal M, Khalid S, Khan MA (2018) Fundus images-based detection and grading of macular edema using robust macula localization. IEEE Access 6:58784–58793

    Article  Google Scholar 

  42. Al-Bander B, Al-Nuaimy W, Al-Taee MA, Williams BM, Zheng Y (2016) Diabetic macular edema grading based on deep neural networks. In: Proceedings of the Ophthalmic Medical Image Analysis International Workshop

  43. Woo S, Park J, Lee J-Y, Kweon IS (2018) Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp. 3–19

  44. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Advances in neural information processing systems, vol 30, pp 1–11

  45. Hendrycks D, Gimpel K (2016) “Gaussian error linear units (gelus),” arXiv preprint arXiv:1606.08415

  46. Ba JL, Kiros JR, Hinton GE (2016) “Layer normalization,” arXiv preprint arXiv:1607.06450

  47. Decencière E, Zhang X, Cazuguel G, Lay B, Cochener B, Trone C, Gain P, Ordonez R, Massin P, Erginay A et al (2014) Feedback on a publicly distributed image database: the messidor database. Image Anal Stereol 33(3):231–234

    Article  MATH  Google Scholar 

  48. Porwal P, Pachade S, Kamble R, Kokare M, Deshmukh G, Sahasrabuddhe V, Meriaudeau F (2018) Indian diabetic retinopathy image dataset (idrid): a database for diabetic retinopathy screening research. Data 3(3):25

    Article  Google Scholar 

  49. Sánchez CI, Niemeijer M, Dumitrescu AV, Suttorp-Schulten MS, Abramoff MD, van Ginneken B (2011) Evaluation of a computer-aided diagnosis system for diabetic retinopathy screening on public data. Investig Ophthalmol Visual Sci 52(7):4866–4871

    Article  Google Scholar 

  50. Li J, Hassani A, Walton S, Shi H (2021) “Convmlp: Hierarchical convolutional mlps for vision,”arXiv preprint arXiv:2109.04454

  51. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

  52. Loshchilov I, Hutter F (2016) Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983

  53. Pires R, Avila S, Jelinek HF, Wainer J, Valle E, Rocha A (2015) Beyond lesion-based diabetic retinopathy: a direct approach for referral. IEEE J Biomed Health Inform 21(1):193–200

    Article  Google Scholar 

  54. Vo HH, Verma A (2016) New deep neural nets for fine-grained diabetic retinopathy recognition on hybrid color space. In: IEEE International Symposium on Multimedia (ISM). IEEE 2016:209–215

  55. He A, Li T, Li N, Wang K, Fu H (2020) Cabnet: category attention block for imbalanced diabetic retinopathy grading. IEEE Trans Med Imaging 40(1):143–153

    Article  Google Scholar 

  56. Porwal P, Pachade S, Kokare M, Deshmukh G, Son J, Bae W, Liu L, Wang J, Liu X, Gao L et al (2020) Idrid: diabetic retinopathy-segmentation and grading challenge. Med Image Anal 59:101561

    Article  Google Scholar 

  57. Chen Q, Peng Y, Keenan T, Dharssi S, Agro E et al (2019) A multi-task deep learning model for the classification of age-related macular degeneration. AMIA Summits Trans Sci Proc 2019:505

    Google Scholar 

  58. Tan C, Zhao L, Yan Z, Li K, Metaxas D, Zhan Y (2018) Deep multi-task and task-specific feature learning network for robust shape preserved organ segmentation. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). IEEE, pp. 1221–1224

Download references

Acknowledgements

This work was supported by national natural science foundation of China under Grant 82071995, key research and development program of Jilin Province, China under Grant 20220201141GX and natural science foundation of Jilin Province, China under Grant 20200201292JC.

Author information

Authors and Affiliations

  1. Key Laboratory of Symbol Computation and Knowledge Engineering of Ministry of Education, College of Computer Science and Technology, Jilin University, Changchun, 130012, China

    Jia Zhang, Xiaoxin Guo, Qifeng Lin & Haoren Wang

  2. Ophthalmology Department, Bethune First Hospital of Jilin University, Changchun, 130021, China

    Xiaoying Hu

  3. Ophthalmology Department, Bethune Second Hospital of Jilin University, Changchun, 130041, China

    Songtian Che

Authors
  1. Jia Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoxin Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qifeng Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haoren Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaoying Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Songtian Che
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoxin Guo.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Guo, X., Lin, Q. et al. A Novel original feature fusion network for joint diabetic retinopathy and diabetic Macular edema grading. Neural Comput & Applic 35, 6699–6712 (2023). https://doi.org/10.1007/s00521-022-08038-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-08038-y

Keywords

Search

Navigation