Skip to main content
SpringerLink
Log in
SpringerLink
Log in

Inter-observer Reliability and Agreement Study on Early Diagnosis of Diabetic Retinopathy and Diabetic Macular Edema Risk

  • Conference paper
  • First Online:
Bioinformatics and Biomedical Engineering (IWBBIO 2016)

Part of the book series: Lecture Notes in Computer Science ((LNBI,volume 9656))

Included in the following conference series:

Abstract

The degree of inter-observer agreement on early diagnosis of diabetic retinopathy (DR) and diabetic macular edema (DME) risk has been assessed in this paper. Three sets of DR and DME risk ratings on 529 diabetic patients were independently built by ophthalmologists of the Andalusian (Spain) Health Service through observation of two macula-centered retinographies from these patients (one image per eye, 1058 images). DR was graded on a 0–3 scale from DR-unrelated to severe DR, while DME risk was graded on a 0–2 scale from no risk to moderate-severe risk. Inter-rater reliability (IRR) assessment was performed by the intra-class correlation (ICC) and two kappa-like statistical variants —Light’s kappa and Fleiss’ kappa. ICC-computed IRR showed excellent agreement between our three coders: values were 0.844 (95 % CI, 0.822–0.865) and 0.833 (95 % CI, 0.805–0.853) for DR and DME ratings, respectively. Kappa index-quantified assessment resulted in substantial agreement, as both kappa indexes rendered values around 0.60 for DR and 0.75 for DME ratings. All computed IRR metrics proved high inter-observer agreement and consistency among DR degree and DME risk diagnoses. Reliable diagnosis provided by human experts supports the generation of reference standards that can be used in the development of automatic DR diagnosis systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The collaboration of medical experts was formally developed through the Project “Expert System for Early Automated Detection of Diabetic Retinopathy by Analysis of Digital Retinal Images”, supported and funded by the Health Ministry of the Andalusian Regional Government (Spain).

References

  1. Guariguata, L., Whiting, D.R., Hambleton, I., Beagley, J., Linnenkamp, U., Shaw, J.E.: Global estimates of diabetes prevalence for 2013 and projections for 2035. Diabetes Res. Clin. Pract. 103(2), 137–149 (2014)

    Article  Google Scholar 

  2. Klein, B.E.K.: Overview of epidemiologic studies of diabetic retinopathy. Ophthalmic Epidemiol. 14(4), 179–183 (2007)

    Article  Google Scholar 

  3. Boyd, S., Advani, A., Altomare, F., Stockl, F.: Clinical practice guidelines for the prevention and management of diabetes in Canada: Retinopathy. Can. J. Diab. 37(Suppl. 1), S137–S141 (2013)

    Article  Google Scholar 

  4. Vila, L., Viguera, J., Aleman, R.: Diabetic retinopathy and blindness in Spain: epidemiology and prevention. Endocrinol. Nutr. 55(10), 459–475 (2008)

    Article  Google Scholar 

  5. Gibelalde, A., et al.: Prevalence of diabetic retinopathy using non-mydriaticretinography. An. SistSanit. Navar. 33(3), 271–276 (2010)

    Google Scholar 

  6. Sender, M.J., Bagur, S.M., Badia, X., Maseras, M., de la Puente, M.L., Foz, M.: Cámara de retina no midríatica: estudio de coste-efectividad en la detección temprana de la retinopatía diabética. Med. Clín. 121(12), 446–452 (2003)

    Article  Google Scholar 

  7. Aptel, F., Denis, P., Rouberol, F., Thivolet, C.: Screening of diabetic retinopathy: effect of field number and mydriasis on sensitivity and specificity of digital fundus photography. Diab. Metab. 34(3), 290–293 (2008)

    Article  Google Scholar 

  8. Patton, N., Aslam, T.M., MacGillivray, T., Deary, I.J., Dhillon, B., Eikelboom, R.H., Yogesan, K., Constable, I.J.: Retinal image analysis: conc epts, applications and potential. Prog. Retin. Eye Res. 25, 99–127 (2006)

    Article  Google Scholar 

  9. Singalavanija, A., Supokavej, J., Bamroongsuk, P., Sinthanayothin, C., Phoojaruenchanachai, S., Kongbunkiat, V.: Feasibility study on computer- aided screening for diabetic retinopathy. Jpn. J. Ophthalmol. 50, 361–366 (2006)

    Article  Google Scholar 

  10. American Academy of Ophthalmology. Diabetic retinopathy. Preferred practice pattern guidelines (2008). http://www.aao.org/ppp

  11. Lairson, D.R., Pugh, J.A., Kapadia, A.S., Lorimor, R.J., Jacobson, J., Velez, R.: Cost effectiveness of alternative methods for diabetic retinopathy screening. Diab. Care 15, 1369–1377 (1992)

    Article  Google Scholar 

  12. Aquino, A., Gegúndez-Arias, M.E., Marín, D.: Detecting the optic disc boundary in digital fundus images using morphological, edge detection, and feature extraction techniques. IEEE Trans. Med. Imag. 29(11), 1860–1869 (2010)

    Article  Google Scholar 

  13. Marín, D., Aquino, A., Gegúndez-Arias, M.E., Bravo, J.M.: A new supervised method for blood vessel segmentation in retinal images by using gray-level and moment invariants-based features. IEEE Trans. Med. Imag. 30(1), 146–158 (2011)

    Article  Google Scholar 

  14. Gegúndez-Arias, M.E., Marin, D., Bravo, J.M., Suero, A.: Locating the fovea center position in digital fundus images using thresholding and feature extraction techniques. Comput. Med. Imaging Graph. 37, 386–393 (2013)

    Article  Google Scholar 

  15. Akram, M.U., Tariq, A., Anjum, M.A., Javed, M.Y.: Automated detection of exudates in colored retinal images for diagnosis of diabetic retinopathy. Appl. Opt. 51(20), 4858–4866 (2010)

    Article  Google Scholar 

  16. Quellec, G., Lamard, M., Josselin, P.M., Cazuguel, G., Cochener, B., Roux, C.: Optimal wavelet transform for the detection of microaneurysms in retina photographs. IEEE Trans. Med. Imag. 27(9), 1230–1241 (2008)

    Article  Google Scholar 

  17. Acharya, U.R., Lim, C.M., Ng, E.Y.K., Chee, C., Tamura, T.: Computer-based detection of diabetes retinopathy stages using digital fundus images. Proc. Inst. Mech. Eng. H 223(5), 545–553 (2009)

    Article  Google Scholar 

  18. Niemeijer, M., Abràmoff, M.D., van Ginneken, B.: Information fusion for diabetic retinopathy CAD in digital color fundus photographs. IEEE Trans. Med. Imag. 28(5), 775–785 (2009)

    Article  Google Scholar 

  19. Philip, S., Fleming, A.D., Goatman, K.A., Fonseca, S., Mcnamee, P., Scotland, G.S., Prescott, G.J., Sharp, P.F., Olson, J.A.: The efficacy of automated ‘‘disease/no disease’’ grading for diabetic retinopathy in a systematic screening programme. Br. J. Ophthalmol. 91, 1512–1517 (2007)

    Article  Google Scholar 

  20. Abràmoff, M.D., Folk, J.C., Han, D.P., Walker, J.D., Williams, D.F., Russell, S.R., Massin, P., Cochener, B., Gain, P., Tang, L., Lamard, M., Moga, D.C., Quellec, G., Niemeijer, M.: Automated analysis of retinal images for detection of referable diabetic retinopathy. JAMA Ophthalmol. 131(3), 351–357 (2013)

    Article  Google Scholar 

  21. MESSIDOR TECHNO-VISION Project, France, MESSIDOR: Digital Retinal Images (Download images section). http://messidor.crihan.fr/download-en.php

  22. MESSIDOR TECHNO-VISION Project, France, Methods to evaluate segmentation and indexing techniques in the field of retinal ophthalmology. http://messidor.crihan.fr/index-en.php

  23. Massin, P., Angioi-Duprez, K., Bacin, F., Cathelineau, B., Cathelineau, G., Chaine, G., Coscas, G., Flament, J., Sahel, J., Turut, P., Guillausseau, P.J., Gaudric, A.: Recommandations de l’ALFEDIAM pour le d´epistage, et la surveillance de la r´etinopathiediab´etique. Diab. Metab. 22, 203–209 (1996)

    Google Scholar 

  24. Massin, P., Angioi-Duprez, K., Bacin, F., Cathelineau, B., Cathelineau, G., Chaine, G., Coscas, G., Flament, J., Sahel, J., Turut, P., Guillausseau, P.J., Gaudric, A.: Recommandations de lALFEDIAMpour le d´epistage et la surveillance de la r´etinopathiediab´etique. J. Fr. Ophtalmol. 20, 302–310 (1997)

    Google Scholar 

  25. Early Treatment Diabetic Retinopathy Study Research Group: Grading diabetic retinopathy from stereoscopic color fundus photographs an extension of the Modified Airlie House classification: ETDRS report number 10”. Ophthalmol. 98, 786–806 (1991)

    Article  Google Scholar 

  26. Diabetic Retinopathy Screening Services in Scotland. Diabetic retinopathy screening: Annex E. Scottish diabetic retinopathy grading scheme. The Scottish Government Publications. http://www.scotland.gov.uk/Publications/2003/07/17638/23088

  27. Shrout, P.E., Fleiss, J.L.: Intraclass correlations: uses in assessing rater reliability. Psychol. Bull. 86(2), 420–428 (1979)

    Article  Google Scholar 

  28. Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Measur. 20(1), 37–46 (1960)

    Article  Google Scholar 

  29. Light, R.J.: Measures of response agreement for qualitative data: Some generalizations and alternatives. Psychol. Bull. 76(5), 365–377 (1971)

    Article  Google Scholar 

  30. Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychol. Bull. 76(5), 378–382 (1971)

    Article  Google Scholar 

  31. Scott, W.A.: Reliability of content analysis: The case of nominal scale coding. Public Opin. Q. 19(3), 321–325 (1955)

    Article  Google Scholar 

  32. Gamer, M., Lemon, J., Fellows, I., Singh, P.: Various coefficients of interrater reliability and agreement. R package version 0.83 (2010). http://CRAN.R-project.org/package=irr

  33. Cicchetti, D.V.: Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol. Assess. 6(4), 284–290 (1994)

    Article  Google Scholar 

  34. Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the Messidor program partners for facilitating their database.

This work was carried out as part of the Project “Expert System for Early Automated Detection of Diabetic Retinopathy by Analysis of Digital Retinal Images”, supported and funded by the Health Ministry of the Andalusian Regional Government (Spain).

Author information

Authors and Affiliations

  1. Department of Mathematics, University of Huelva, Huelva, Spain

    Manuel Emilio Gegundez-Arias

  2. North Area of Sanitary Management of Cordoba, Andalusian Health Service, Seville, Spain

    Carlos Ortega & Javier Garrido

  3. Virgen de Macarena Universitary Hospital of Seville, Andalusian Health Service, Seville, Spain

    Beatriz Ponte

  4. Juan Ramon Jimenez Hospital of Huelva, Andalusian Health Service, Seville, Spain

    Fatima Alvarez

  5. Department of Electronic, Computer Science and Automatic Engineering, University of Huelva, Huelva, Spain

    Diego Marin

Authors
  1. Manuel Emilio Gegundez-Arias
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carlos Ortega
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Javier Garrido
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Beatriz Ponte
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Fatima Alvarez
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Diego Marin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Diego Marin .

Editor information

Editors and Affiliations

  1. Universidad de Granada, Granada, Spain

    Francisco Ortuño

  2. Universidad de Granada, Granada, Spain

    Ignacio Rojas

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Gegundez-Arias, M.E., Ortega, C., Garrido, J., Ponte, B., Alvarez, F., Marin, D. (2016). Inter-observer Reliability and Agreement Study on Early Diagnosis of Diabetic Retinopathy and Diabetic Macular Edema Risk. In: Ortuño, F., Rojas, I. (eds) Bioinformatics and Biomedical Engineering. IWBBIO 2016. Lecture Notes in Computer Science(), vol 9656. Springer, Cham. https://doi.org/10.1007/978-3-319-31744-1_33

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-31744-1_33

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-31743-4

  • Online ISBN: 978-3-319-31744-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation