Abstract
Longitudinal medical image analysis is crucial for identifying the unobvious emergence and evolution of early lesions, towards earlier and better patient-specific pathology management. However, traditional computer-aided diagnosis (CAD) systems for diabetic retinopathy (DR) rarely make use of longitudinal information to improve DR analysis. In this work, we present a deep information fusion framework that exploits two consecutive longitudinal studies for the assessment of early DR severity changes. In particular, three fusion schemes are investigated: (1) early fusion of inputs, (2) intermediate fusion of feature vectors incorporating Spatial Transformer Networks (STN) and (3) late fusion of feature vectors. Exhaustive experiments compared with respect to no-fusion baselines validate that incorporating prior DR studies can improve the referable DR severity classification performance through the late fusion scheme whose AUC reaches 0.9296. Advantages and limitations of the different fusion methods are discussed in depth. We also propose different pre-training strategies which are employed to bring considerable performance gains for DR severity grade change detection purposes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adal, K.M., Van Etten, P.G., Martinez, J.P., Rouwen, K.W., Vermeer, K.A., van Vliet, L.J.: An automated system for the detection and classification of retinal changes due to red lesions in longitudinal fundus images. IEEE Trans. Biomed. Eng. 65(6), 1382–1390 (2017)
Bernardes, R., et al.: Computer-assisted microaneurysm turnover in the early stages of diabetic retinopathy. Ophthalmologica 223(5), 284–291 (2009)
Geras, K.J., et al.: High-resolution breast cancer screening with multi-view deep convolutional neural networks. arXiv preprint arXiv:1703.07047 (2017)
Gulshan, V., et al.: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA 316(22), 2402–2410 (2016)
Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial transformer networks. arXiv preprint arXiv:1506.02025 (2015)
Liu, H., Yue, K., Cheng, S., Pan, C., Sun, J., Li, W.: Hybrid model structure for diabetic retinopathy classification. J. Healthcare Eng. 2020, Article id: 8840174 (2020)
Massin, P., et al.: OPHDIAT: a telemedical network screening system for diabetic retinopathy in the île-de-france. Diab. Meta. 34(3), 227–234 (2008)
Narasimha-Iyer, H., Can, A., Roysam, B., Tanenbaum, H.L., Majerovics, A.: Integrated analysis of vascular and nonvascular changes from color retinal fundus image sequences. IEEE Trans. Biomed. Eng. 54(8), 1436–1445 (2007)
Ogurtsova, K., et al.: IDF diabetes atlas: global estimates for the prevalence of diabetes for 2015 and 2040. Diab. Res. Clin. Pract. 128, 40–50 (2017)
Perek, S., Hazan, A., Barkan, E., Akselrod-Ballin, A.: Siamese network for dual-view mammography mass matching. In: Image Analysis for Moving Organ, Breast, and Thoracic Images, pp. 55–63 (2018)
Perek, S., Ness, L., Amit, M., Barkan, E., Amit, G.: Learning from longitudinal mammography studies. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 712–720 (2019)
Quellec, G., Charrière, K., Boudi, Y., Cochener, B., Lamard, M.: Deep image mining for diabetic retinopathy screening. Med. Image Anal. 39, 178–193 (2017)
Quellec, G., et al.: Instant automatic diagnosis of diabetic retinopathy. arXiv preprint arXiv:1906.11875 (2019)
Robin, X., et al.: pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinf. 12(1), 1–8 (2011)
Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)
Saha, S.K., Xiao, D., Bhuiyan, A., Wong, T.Y., Kanagasingam, Y.: Color fundus image registration techniques and applications for automated analysis of diabetic retinopathy progression: a review. Biomed. Signal Process. Control 47, 288–302 (2019)
Santeramo, R., Withey, S., Montana, G.: Longitudinal detection of radiological abnormalities with time-modulated LSTM. In: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 326–333 (2018)
Shankar, K., Sait, A.R.W., Gupta, D., Lakshmanaprabu, S., Khanna, A., Pandey, H.M.: Automated detection and classification of fundus diabetic retinopathy images using synergic deep learning model. Pattern Recogn. Lett. 133, 210–216 (2020)
Sikder, N., Masud, M., Bairagi, A.K., Arif, A.S.M., Nahid, A.A., Alhumyani, H.A.: Severity classification of diabetic retinopathy using an ensemble learning algorithm through analyzing retinal images. Symmetry 13(4), 670 (2021)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.: Inception-v4, Inception-ResNet and the impact of residual connections on learning. In: AAAI Conference on Artificial Intelligence, vol. 31 (2017)
Thomas, R., Halim, S., Gurudas, S., Sivaprasad, S., Owens, D.: IDF diabetes atlas: a review of studies utilising retinal photography on the global prevalence of diabetes related retinopathy between 2015 and 2018. Diab. Res. Clin. Pract. 157, 107840 (2019)
Wilkinson, C., et al.: Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology 110(9), 1677–1682 (2003)
Yan, Y., Conze, P.H., Lamard, M., Quellec, G., Cochener, B., Coatrieux, G.: Multi-tasking siamese networks for breast mass detection using dual-view mammogram matching. In: International Workshop on Machine Learning in Medical Imaging, pp. 312–321 (2020)
Yan, Y., Conze, P.H., Lamard, M., Quellec, G., Cochener, B., Coatrieux, G.: Towards improved breast mass detection using dual-view mammogram matching. Med. Image Anal. 71, 102083 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Yan, Y. et al. (2021). Longitudinal Detection of Diabetic Retinopathy Early Severity Grade Changes Using Deep Learning. In: Fu, H., Garvin, M.K., MacGillivray, T., Xu, Y., Zheng, Y. (eds) Ophthalmic Medical Image Analysis. OMIA 2021. Lecture Notes in Computer Science(), vol 12970. Springer, Cham. https://doi.org/10.1007/978-3-030-87000-3_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-87000-3_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86999-1
Online ISBN: 978-3-030-87000-3
eBook Packages: Computer ScienceComputer Science (R0)
