Abstract
Diabetic retinopathy (DR) and its complication diabetic macular edema (DME) are the leading cause of permanent blindness in the working-age population worldwide. Automated grading of DR and DME enables ophthalmologists to carry out tailored treatments to patients in early stages of their diseases. However, most of the current works only focus on the grading of a single disease, ignoring the relationship between DR and DME, and the traditional convolutional architectures face the problem that they can not capture long-distance dependencies despite of the effectiveness of extracting image features. To this end, we propose an original feature fusion network (OFFNet) for joint DR and DME grading based on the idea of key-value query, which consists of a specific feature extraction module (SFEM) based on self-attention and an original feature fusion module (OFFM) based on cross-attention. The proposed OFFNet enjoys several merits. First, to the best of our knowledge, this is the first joint grading effort based on the idea of key-value query. Second, OFFNet only needs image-level supervision, which can facilitate the acquisition of training data, rather than patch-level or pixel-level supervision. Third, OFFNet has obvious advantages in capturing long-distance dependencies. Extensive experiments on two public datasets Messidor and 2018 IDRiD challenge show that our method outperforms other joint grading methods on joint grading accuracy and the ability of capturing long-distance dependencies.







Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data availibility
Data will be made available on reasonable request.
References
Mathews MR, Anzar S (2021) A comprehensive review on automated systems for severity grading of diabetic retinopathy and macular edema. Int J Imaging Syst Technol 31(4):2093–2122
Yang Y, Shang F, Wu B, Yang D, Wang L, Xu Y, Zhang W, Zhang T (2021) Robust collaborative learning of patch-level and image-level annotations for diabetic retinopathy grading from fundus image. IEEE Trans Cybern 155(11):11407–11417
Wilkinson CP, Iii FLF, Klein RE, Lee PP, Agardh CD, Davis M, Dills D, Kampik A, Pararajasegaram R, Verdaguer JT (2003) Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology 110(9):1677–1682
Li X, Hu X, Yu L, Zhu L, Fu C-W, Heng P-A (2019) Canet: cross-disease attention network for joint diabetic retinopathy and diabetic macular edema grading. IEEE Trans Med Imaging 39(5):1483–1493
The Eye M.D. Association, American Academy of Ophthalmology (2002) International clinical diabetic retinopathy disease severity scale detailed table. American Academy of Ophthalmology, San Francisco, CA, USA, Technical Report
Sun R, Li Y, Zhang T, Mao Z, Wu F, Zhang Y (2021) Lesion-aware transformers for diabetic retinopathy grading. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10 938–10 947
Das A, McGuire PG, Rangasamy S (2015) Diabetic macular edema: pathophysiology and novel therapeutic targets. Ophthalmology 122(7):1375–1394
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778
Howard A, Sandler M, Chu G, Chen L-C, Chen B, Tan M, Wang W, Zhu Y, Pang R, Vasudevan V et al. (2019) Searching for mobilenetv3. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1314–1324
Wang J, Sun K, Cheng T, Jiang B, Deng C, Zhao Y, Liu D, Mu Y, Tan M, Wang X et al (2020) Deep high-resolution representation learning for visual recognition. IEEE Trans Pattern Anal Mach Intell 43(10):3349–3364
Lei B, Liang E, Yang M, Yang P, Zhou F, Tan EL, Lei Y, Liu CM, Wang T, Xiao X (2022) Predicting clinical scores for alzheimer’s disease based on joint and deep learning. Expert Syst Appl 187:115966
Abràmoff MD, Lou Y, Erginay A, Clarida W, Amelon R, Folk JC, Niemeijer M (2016) Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Investig Ophthalmol Visual Sci 57(13):5200–5206
Yang Y, Li T, Li W, Wu H, Fan W, Zhang W (2017) Lesion detection and grading of diabetic retinopathy via two-stages deep convolutional neural networks. In: International conference on medical image computing and computer-assisted intervention. Springer, pp. 533–540
Gondal WM, Köhler JM, Grzeszick R, Fink GA, Hirsch M (2017) Weakly-supervised localization of diabetic retinopathy lesions in retinal fundus images. In: 2017 IEEE international conference on image processing (ICIP).IEEE, pp. 2069–2073
Foo A, Hsu W, Lee ML, Lim G, Wong TY (2020) Multi-task learning for diabetic retinopathy grading and lesion segmentation. In: Proceedings of the AAAI conference on artificial intelligence, vol. 34, no. 08, pp. 13 267–13 272
Mo J, Zhang L, Feng Y (2018) Exudate-based diabetic macular edema recognition in retinal images using cascaded deep residual networks. Neurocomputing 290:161–171
He X, Zhou Y, Wang B, Cui S, Shao L (2019) Dme-net: Diabetic macular edema grading by auxiliary task learning. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 788–796
Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, Venugopalan S, Widner K, Madams T, Cuadros J et al (2016) Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316(22):2402–2410
Krause J, Gulshan V, Rahimy E, Karth P, Widner K, Corrado GS, Peng L, Webster DR (2018) Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology 125(8):1264–1272
Tu Z, Gao S, Zhou K, Chen X, Fu H, Gu Z, Cheng J, Yu Z, Liu J (2020) Sunet: a lesion regularized model for simultaneous diabetic retinopathy and diabetic macular edema grading. In: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI).IEEE, pp. 1378–1382
Silberman N, Ahrlich K, Fergus R, Subramanian L (2010) Case for automated detection of diabetic retinopathy. In: 2010 AAAI spring symposium series
Sopharak A, Uyyanonvara B, Barman S (2009) Automatic exudate detection from non-dilated diabetic retinopathy retinal images using fuzzy c-means clustering. Sensors 9(3):2148–2161
Roychowdhury S, Koozekanani DD, Parhi KK (2013) Dream: diabetic retinopathy analysis using machine learning. IEEE J Biomed Health Inform 18(5):1717–1728
Acharya UR, Lim CM, Ng EYK, Chee C, Tamura T (2009) Computer-based detection of diabetes retinopathy stages using digital fundus images. Proc Inst Mech Eng [H] 223(5):545–553
Akram MU, Khalid S, Tariq A, Khan SA, Azam F (2014) Detection and classification of retinal lesions for grading of diabetic retinopathy. Comput Biol Med 45:161–171
Antal B, Hajdu A (2014) An ensemble-based system for automatic screening of diabetic retinopathy. Knowl-Based Syst 60:20–27
Seoud L, Hurtut T, Chelbi J, Cheriet F, Langlois JP (2015) Red lesion detection using dynamic shape features for diabetic retinopathy screening. IEEE Trans Med Imaging 35(4):1116–1126
Kumar N, Rajwade AV, Chandran S, Awate SP (2017) “Kernel generalized-gaussian mixture model for robust abnormality detection. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 21–29
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, vol. 25,
Gargeya R, Leng T (2017) Automated identification of diabetic retinopathy using deep learning. Ophthalmology 124(7):962–969
Wang Z, Yin Y, Shi J, Fang W, Li H, Wang X (2017) Zoom-in-net: Deep mining lesions for diabetic retinopathy detection. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 267–275
Li X, Shen L, Shen M, Tan F, Qiu CS (2019) Deep learning based early stage diabetic retinopathy detection using optical coherence tomography. Neurocomputing 369:134–144
Wang S, Wang X, Hu Y, Shen Y, Lei B (2020) Diabetic retinopathy diagnosis using multichannel generative adversarial network with semisupervision’’. IEEE Trans Autom Sci Eng 99:1–12
Lin Z, Guo R, Wang Y, Wu B, Chen T, Wang W, Chen DZ, Wu J (2018) A framework for identifying diabetic retinopathy based on anti-noise detection and attention-based fusion. In: International Conference on Medical Image Computing and Computer-Assisted Intervention.Springer, pp. 74–82
Zhou Y, He X, Huang L, Liu L, Zhu F, Cui S, Shao L (2019) Collaborative learning of semi-supervised segmentation and classification for medical images. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2079–2088
Wang X, Xu M, Zhang J, Jiang L, Li L (2021) Deep multi-task learning for diabetic retinopathy grading in fundus images. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1–5
Akram MU, Tariq A, Khan SA, Javed MY (2014) Automated detection of exudates and macula for grading of diabetic macular edema. Comput Methods Programs Biomed 114(2):141–152
Acharya UR, Mookiah MRK, Koh JE, Tan JH, Bhandary SV, Rao AK, Hagiwara Y, Chua CK, Laude A (2017) Automated diabetic macular edema (dme) grading system using dwt, dct features and maculopathy index. Comput Biol Med 84:59–68
Ren F, Cao P, Zhao D, Wan C (2018) Diabetic macular edema grading in retinal images using vector quantization and semi-supervised learning. Technol Health Care 26(S1):389–397
Syed AM, Akram MU, Akram T, Muzammal M, Khalid S, Khan MA (2018) Fundus images-based detection and grading of macular edema using robust macula localization. IEEE Access 6:58784–58793
Al-Bander B, Al-Nuaimy W, Al-Taee MA, Williams BM, Zheng Y (2016) Diabetic macular edema grading based on deep neural networks. In: Proceedings of the Ophthalmic Medical Image Analysis International Workshop
Woo S, Park J, Lee J-Y, Kweon IS (2018) Cbam: Convolutional block attention module. In: Proceedings of the European conference on computer vision (ECCV), pp. 3–19
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Advances in neural information processing systems, vol 30, pp 1–11
Hendrycks D, Gimpel K (2016) “Gaussian error linear units (gelus),” arXiv preprint arXiv:1606.08415
Ba JL, Kiros JR, Hinton GE (2016) “Layer normalization,” arXiv preprint arXiv:1607.06450
Decencière E, Zhang X, Cazuguel G, Lay B, Cochener B, Trone C, Gain P, Ordonez R, Massin P, Erginay A et al (2014) Feedback on a publicly distributed image database: the messidor database. Image Anal Stereol 33(3):231–234
Porwal P, Pachade S, Kamble R, Kokare M, Deshmukh G, Sahasrabuddhe V, Meriaudeau F (2018) Indian diabetic retinopathy image dataset (idrid): a database for diabetic retinopathy screening research. Data 3(3):25
Sánchez CI, Niemeijer M, Dumitrescu AV, Suttorp-Schulten MS, Abramoff MD, van Ginneken B (2011) Evaluation of a computer-aided diagnosis system for diabetic retinopathy screening on public data. Investig Ophthalmol Visual Sci 52(7):4866–4871
Li J, Hassani A, Walton S, Shi H (2021) “Convmlp: Hierarchical convolutional mlps for vision,”arXiv preprint arXiv:2109.04454
Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980
Loshchilov I, Hutter F (2016) Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983
Pires R, Avila S, Jelinek HF, Wainer J, Valle E, Rocha A (2015) Beyond lesion-based diabetic retinopathy: a direct approach for referral. IEEE J Biomed Health Inform 21(1):193–200
Vo HH, Verma A (2016) New deep neural nets for fine-grained diabetic retinopathy recognition on hybrid color space. In: IEEE International Symposium on Multimedia (ISM). IEEE 2016:209–215
He A, Li T, Li N, Wang K, Fu H (2020) Cabnet: category attention block for imbalanced diabetic retinopathy grading. IEEE Trans Med Imaging 40(1):143–153
Porwal P, Pachade S, Kokare M, Deshmukh G, Son J, Bae W, Liu L, Wang J, Liu X, Gao L et al (2020) Idrid: diabetic retinopathy-segmentation and grading challenge. Med Image Anal 59:101561
Chen Q, Peng Y, Keenan T, Dharssi S, Agro E et al (2019) A multi-task deep learning model for the classification of age-related macular degeneration. AMIA Summits Trans Sci Proc 2019:505
Tan C, Zhao L, Yan Z, Li K, Metaxas D, Zhan Y (2018) Deep multi-task and task-specific feature learning network for robust shape preserved organ segmentation. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018). IEEE, pp. 1221–1224
Acknowledgements
This work was supported by national natural science foundation of China under Grant 82071995, key research and development program of Jilin Province, China under Grant 20220201141GX and natural science foundation of Jilin Province, China under Grant 20200201292JC.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhang, J., Guo, X., Lin, Q. et al. A Novel original feature fusion network for joint diabetic retinopathy and diabetic Macular edema grading. Neural Comput & Applic 35, 6699–6712 (2023). https://doi.org/10.1007/s00521-022-08038-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-022-08038-y