Skip to main content
SpringerLink
Log in

Retinal artery/vein classification by multi-channel multi-scale fusion network

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The automatic artery/vein (A/V) classification in retinal fundus images plays a significant role in detecting vascular abnormalities and could speed up the diagnosis of various systemic diseases. Deep-learning methods have been extensively employed in this task. However, due to the lack of annotated data and the serious data imbalance, the performance of the existing methods is constricted. To address these limitations, we propose a novel multi-channel multi-scale fusion network (MMF-Net) that employs the enhancement of vessel structural information to constrain the A/V classification. First, the newly designed multi-channel (MM) module could extract the vessel structure from the original fundus image by the frequency filters, increasing the proportion of blood vessel pixels and reducing the influence caused by the background pixels. Second, the MMF-Net introduces a multi-scale transformation (MT) module, which could efficiently extract the information from the multi-channel feature representations. Third, the MMF-Net utilizes a multi-feature fusion (MF) module to improve the robustness of A/V classification by splitting and reorganizing the pixel feature from different scales. We validate our results on several public benchmark datasets. The experimental results show that the proposed method could achieve the best result compared with the existing state-of-the-art methods, which demonstrate the superior performance of the MMF-Net. The highly optimized Python implementations of our method is released at: https://github.com/chenchouyu/MMF_Net.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Algorithm 1
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Availability of data and materials

The datasets generated during and/or analysed during the current study are available in the https://medicine.uiowa.edu/eye/rite-dataset, https://www5.cs.fau.de/research/data/fundus-images/, and https://www.idiap.ch/software/bob/docs/bob/bob.db.iostar/stable/ repository

Code Availability

Available

References

  1. Wong TY, Klein R, Sharrett AR, Duncan BB, Couper DJ, Klein BE, Hubbard LD, Nieto FJ (2004) Retinal arteriolar diameter and risk for hypertension. Ann Intern Med 140(4):248–255

    Article  Google Scholar 

  2. Pellegrini E, Robertson G, MacGillivray T, Hemert J, Houston G, Trucco E (2017) A graph cut approach to artery/vein classification in ultra-widefield scanning laser ophthalmoscopy. IEEE Trans Med Imaging 37(2):516–526

    Article  Google Scholar 

  3. Huang F, Dashtbozorg B, Tan T, Haar Romeny BM (2018) Retinal artery/vein classification using genetic-search feature selection. Comput Methods Programs Biomed 161:197–207

    Article  Google Scholar 

  4. Li L, Verma M, Nakashima Y, Kawasaki R, Nagahara H (2020) Joint learning of vessel segmentation and artery/vein classification with post-processing. Proceedings of Machine Learning Research 1:14

    Google Scholar 

  5. Hu J, Wang H, Cao Z, Wu G, Jonas JB, Wang YX, Zhang J (2021) Automatic artery/vein classification using a vessel-constraint network for multicenter fundus images. Frontiers in Cell and Developmental Biology 9

  6. Zhang C, Bi J, Xu S, Ramentol E, Fan G, Qiao B, Fujita H (2019) Multi-imbalance: An open-source software for multi-class imbalance learning. Knowl-Based Syst 174:137–143

    Article  Google Scholar 

  7. Zhang S, Li Z, Yan S, He X, Sun J (2021) Distribution alignment: A unified framework for long-tail visual recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 2361–2370

  8. Xu X, Wang R, Lv P, Gao B, Li C, Tian Z, Tan T, Xu F (2018) Simultaneous arteriole and venule segmentation with domain-specific loss function on a new public database. Biomedical Optics Express 9(7):3153–3166

    Article  Google Scholar 

  9. Girard F, Kavalec C, Cheriet F (2019) Joint segmentation and classification of retinal arteries/veins from fundus images. Artif Intell Med 94:96–109

    Article  Google Scholar 

  10. Karlsson RA, Hardarson SH (2022) Artery vein classification in fundus images using serially connected u-nets. Comput Methods Programs Biomed 216:106650

    Article  Google Scholar 

  11. Wang Z, Lin J, Wang R, Zheng W (2019) Retinal artery/vein classification via rotation augmentation and deeply supervised u-net segmentation. ICBIP 2019:71–76

    Article  Google Scholar 

  12. Ma W, Yu S, Ma K, Wang J, Ding X, Zheng Y (2019) Multi-task neural networks with spatial activation for retinal vessel segmentation and artery/vein classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp 769–778

  13. Wu Y, Xia Y, Zhang Y (2018a) Deep classification and segmentation model for vessel extraction in retinal images. In: Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pp 250–258

  14. Wu Y, Xia Y, Song Y, Zhang Y, Cai W (2018b) Multiscale network followed network model for retinal vessel segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp 119–126

  15. Wang B, Wang S, Qiu S, Wei W, Wang H, He H (2020) Csu-net: a context spatial u-net for accurate blood vessel segmentation in fundus images. IEEE Journal of Biomedical and Health Informatics 25(4):1128–1138

    Article  Google Scholar 

  16. Fu H, Cheng J, Xu Y, Wong DWK, Liu J, Cao X (2018) Joint optic disc and cup segmentation based on multi-label deep network and polar transformation. IEEE Trans Med Imaging 1597–1605

  17. Mishra S, Wang YX, Wei CC, Chen DZ, Hu XS (2021) Vtg-net: a cnn based vessel topology graph network for retinal artery/vein classification. Frontiers in Medicine 2124

  18. Tan Y, Yang K-F, Zhao S-X, Li Y-J (2022) Retinal vessel segmentation with skeletal prior and contrastive loss. IEEE Trans Med Imaging 41(9):2238–2251

    Article  Google Scholar 

  19. Guo C, Szemenyei M, Yi Y, Wang W, Chen B, Fan C (2021) Sa-unet: Spatial attention u-net for retinal vessel segmentation. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp 1236–1242

  20. Tong H, Fang Z, Wei Z, Cai Q, Gao Y (2021) Sat-net: a side attention network for retinal image segmentation. Appl Intell 51:5146–5156

    Article  Google Scholar 

  21. Gu Z, Cheng J, Fu H, Zhou K, Hao H, Zhao Y, Zhang T, Gao S, Liu J (2019) Ce-net: Context encoder network for 2d medical image segmentation. IEEE Trans Med Imaging 38(10):2281–2292

    Article  Google Scholar 

  22. Zhang F, Yan Z, Wu Y, Tan X (2019) Attention guided network for retinal image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention 797–805

  23. Yuan Y, Zhang L, Wang L, Huang H (2021) Multi-level attention network for retinal vessel segmentation. IEEE Journal of Biomedical and Health Informatics 26(1):312–323

    Article  Google Scholar 

  24. Lin A, Chen B, Xu J, Zhang Z, Lu G, Zhang D (2022) Ds-transunet: Dual swin transformer u-net for medical image segmentation. IEEE Trans Instrum Meas 71:1–15

    Google Scholar 

  25. Li X, Jiang Y, Li M, Yin S (2020) Lightweight attention convolutional neural network for retinal vessel image segmentation. IEEE Transactions on Industrial Informatics 17(3):1958–1967

    Article  Google Scholar 

  26. Tan X, Chen X, Meng Q, Shi F, Xiang D, Chen Z, Pan L, Zhu W (2023) Oct2former: A retinal oct-angiography vessel segmentation transformer. Comput Methods Programs Biomed 233:107454

    Article  Google Scholar 

  27. Shen X, Xu J, Jia H, Fan P, Dong F, Yu B, Ren S (2022) Self-attentional microvessel segmentation via squeeze-excitation transformer unet. Comput Med Imaging Graph 97:102055

    Article  Google Scholar 

  28. Huang X, Deng Z, Li D, Yuan X, Fu Y (2022) Missformer: An effective transformer for 2d medical image segmentation. IEEE Trans Med Imaging

  29. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp 234–241

  30. Soares JV, Leandro JJ, Cesar RM, Jelinek HF, Cree MJ (2006) Retinal vessel segmentation using the 2-d gabor wavelet and supervised classification. IEEE Trans Med Imaging 25(9):1214–1222

    Article  Google Scholar 

  31. Gao S, Cheng M-M, Zhao K, Zhang X-Y, Yang M-H, Torr PH (2019) Res2net: A new multi-scale backbone architecture. IEEE Transactions on Pattern Analysis and Machine Intelligence 43(2):652–662

    Article  Google Scholar 

  32. Alom MZ, Yakopcic C, Hasan M, Taha TM, Asari VK (2019) Recurrent residual u-net for medical image segmentation. Journal of Medical Imaging 6(1):014006

    Article  Google Scholar 

  33. He K, Xiangyu Zhang SR, Sun J (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778

  34. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Advances in Neural Information Processing Systems 30

  35. Kang H, Gao Y, Guo S, Xu X, Li T, Wang K (2020) Avnet: A retinal artery/vein classification network with category-attention weighted fusion. Comput Methods Prog Biomed 195:105629

  36. Hu Q, Abràmoff MD, Garvin MK (2013) Automated separation of binary overlapping trees in low-contrast color retinal images. In: International Conference on Medical Image Computing and Computer-assisted Intervention, pp 436–443

  37. Hu Q, Abràmoff MD, Garvin MK (2015) Automated construction of arterial and venous trees in retinal images. Journal of Medical Imaging 2(4):044001

    Article  Google Scholar 

  38. Odstrcilik J, Kolar R, Budai A, Hornegger J, Jan J, Gazarek J, Kubena T, Cernosek P, Svoboda O, Angelopoulou E (2013) Retinal vessel segmentation by improved matched filtering: evaluation on a new high-resolution fundus image database. IET Image Process 7(4):373–383

    Article  MathSciNet  Google Scholar 

  39. Zhang J, Dashtbozorg B, Bekkers E, Pluim JPW, Duits R, ter Haar Romeny BM (2016) Robust retinal vessel segmentation via locally adaptive derivative frames in orientation scores. IEEE Trans Med Imaging 35(12):2631–2644

    Article  Google Scholar 

  40. Galdran A, Meyer M, Costa P, Campilho A, et al. (2019) Uncertainty-aware artery/vein classification on retinal images. In: ISBI 2019, pp 556–560 . IEEE

  41. Noh KJ, Park SJ, Lee S (2020) Combining fundus images and fluorescein angiography for artery/vein classification using the hierarchical vessel graph network. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp 595–605

  42. Ye Y, Pan C, Wu Y, Wang S, Xia Y (2022) Mfi-net: Multiscale feature interaction network for retinal vessel segmentation. IEEE Journal of Biomedical and Health Informatics

  43. Yuan Y, Zhang L, Wang L, Huang H (2021) Multi-level attention network for retinal vessel segmentation. IEEE Journal of Biomedical and Health Informatics 26(1):312–323

    Article  Google Scholar 

  44. Xu R, Liu T, Ye X, Liu F, Lin L, Li L, Tanaka S, Chen Y-W (2020) Joint extraction of retinal vessels and centerlines based on deep semantics and multi-scaled cross-task aggregation. IEEE Journal of Biomedical and Health Informatics 25(7):2722–2732

    Article  Google Scholar 

  45. Samuel PM, Veeramalai T (2019) Multilevel and multiscale deep neural network for retinal blood vessel segmentation. Symmetry 11(7):946

    Article  Google Scholar 

  46. Odstrcilik J, Kolar R, Budai A, Hornegger J, Jan J, Gazarek J, Kubena T, Cernosek P, Svoboda O, Angelopoulou E (2013) Retinal vessel segmentation by improved matched filtering: evaluation on a new high-resolution fundus image database. IET Image Process 7(4):373–383

    Article  MathSciNet  Google Scholar 

Download references

Funding

This research was funded by the Science and Technology Project of Beijing Municipal Commission of Education, grant number KM202010016011, the National Natural Science Foundation of China, grant number 61871020, 62031003, Scientific Research Foundation of Beijing University of Civil Engineering and Architecture, grant number 00331613002, the Fundamental Research Funds for Beijing University of Civil Engineering and Architecture, grant number X18064. The computer resources were provided by Public Computing Cloud Platform of Renmin University of China

Author information

Authors and Affiliations

  1. Department of Computer Science and Technology, Beijing University of Civil Engineering and Architecture, Beijing, China

    Junyan Yi & Chouyu Chen

  2. School of Information, Renmin University of China, Beijing, China

    Gang Yang

Authors
  1. Junyan Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chouyu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization: Junyan Yi, Chouyu Chen, Gang Yang; Methodology: Junyan Yi, Gang Yang; Formal analysis and investigation: Junyan Yi, Chouyu Chen, Gang Yang; Software: Chouyu Chen; Writing - original draft preparation: Junyan Yi; Writing - review and editing: Junyan Yi, Chouyu Chen, Gang Yang; Supervision: Gang Yang

Corresponding author

Correspondence to Gang Yang.

Ethics declarations

Ethics approval

Not applicable

Consent to participate

We confirm that the manuscript has been read and approved by all named authors and that there are no other persons who satisfied the criteria for authorship but are not listed. We further confirm that the order of authors listed in the manuscript has been approved by all of us

Consent for publication

This manuscript is the authors’ original work and has not been published nor has it been submitted simultaneously elsewhere. All authors have checked the manuscript and have agreed to the submission

Conflict of interest/Competing interests

All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yi, J., Chen, C. & Yang, G. Retinal artery/vein classification by multi-channel multi-scale fusion network. Appl Intell 53, 26400–26417 (2023). https://doi.org/10.1007/s10489-023-04939-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04939-0

Keywords

Search

Navigation