Skip to main content
SpringerLink
Log in

Inter-observer Variability Analysis of Automatic Lung Delineation in Normal and Disease Patients

  • Education & Training
  • Published:
Journal of Medical Systems Aims and scope Submit manuscript

Abstract

Human interaction has become almost mandatory for an automated medical system wishing to be accepted by clinical regulatory agencies such as Food and Drug Administration. Since this interaction causes variability in the gathered data, the inter-observer and intra-observer variability must be analyzed in order to validate the accuracy of the system. This study focuses on the variability from different observers that interact with an automated lung delineation system that relies on human interaction in the form of delineation of the lung borders. The database consists of High Resolution Computed Tomography (HRCT): 15 normal and 81 diseased patients’ images taken retrospectively at five levels per patient. Three observers manually delineated the lungs borders independently and using software called ImgTracer™ (AtheroPoint™, Roseville, CA, USA) to delineate the lung boundaries in all five levels of 3-D lung volume. The three observers consisted of Observer-1: lesser experienced novice tracer who is a resident in radiology under the guidance of radiologist, whereas Observer-2 and Observer-3 are lung image scientists trained by lung radiologist and biomedical imaging scientist and experts. The inter-observer variability can be shown by comparing each observer’s tracings to the automated delineation and also by comparing each manual tracing of the observers with one another. The normality of the tracings was tested using D’Agostino-Pearson test and all observers tracings showed a normal P-value higher than 0.05. The analysis of variance (ANOVA) test between three observers and automated showed a P-value higher than 0.89 and 0.81 for the right lung (RL) and left lung (LL), respectively. The performance of the automated system was evaluated using Dice Similarity Coefficient (DSC), Jaccard Index (JI) and Hausdorff (HD) Distance measures. Although, Observer-1 has lesser experience compared to Obsever-2 and Obsever-3, the Observer Deterioration Factor (ODF) shows that Observer-1 has less than 10 % difference compared to the other two, which is under acceptable range as per our analysis. To compare between observers, this study used regression plots, Bland-Altman plots, two tailed T-test, Mann-Whiney, Chi-Squared tests which showed the following P-values for RL and LL: (i) Observer-1 and Observer-3 were: 0.55, 0.48, 0.29 for RL and 0.55, 0.59, 0.29 for LL; (ii) Observer-1 and Observer-2 were: 0.57, 0.50, 0.29 for RL and 0.54, 0.59, 0.29 for LL; (iii) Observer-2 and Observer-3 were: 0.98, 0.99, 0.29 for RL and 0.99, 0.99, 0.29 for LL. Further, CC and R-squared coefficients were computed between observers which came out to be 0.9 for RL and LL. All three observers however manage to show the feature that diseased lungs are smaller than normal lungs in terms of area.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Abdullah, N., Mesurolle, B., El-Khoury, M., and Kao, E., Breast imaging reporting and data system lexicon for US: interobserver agreement for assessment of breast masses. Radiology 252:665–672, 2009.

    Article  PubMed  Google Scholar 

  2. Acharya, U. R., Sree, S. V., Krishnan, M. M. R., Krishnananda, N., Ranjan, S., Umesh, P., and Suri, J. S., Automated classification of patients with coronary artery disease using grayscale features from left ventricle echocardiographic images. Comput. Methods Programs Biomed. 112:624–632, 2013.

    Article  PubMed  Google Scholar 

  3. Acharya, U. R., Sree, S. V., Krishnan, M. M. R., Molinari, F., Saba, L., Ho, S. Y. S., Ahuja, A. T., Ho, S. C., Nicolaides, A., and Suri, J. S., Atherosclerotic risk stratification strategy for carotid arteries using texture-based features. Ultrasound Med. Biol. 38:899–915, 2012.

    Article  PubMed  Google Scholar 

  4. Alberola-López, C., Martín-Fernández, M., and Ruiz-Alzola, J., Comments on: a methodology for evaluation of boundary detection algorithms on medical images. IEEE Trans. Med. Imaging 23:658–60, 2004.

    Article  PubMed  Google Scholar 

  5. Centers for Disease Control and Prevention, Deaths: final data for 2005. Natl. Cent. Heal. Stat. Reports. 56. 2008.

  6. Centers for Disease Control and Prevention, Deaths: final data for 2004. Natl. Vital Stat. Reports. 55. 2007.

  7. Churg, A., Thurlbeck’s pathology of the lung. Thieme New York, NY, 2005.

    Google Scholar 

  8. Dehmeshki, J., Amin, H., and Valdivieso, M., X. Ye Segmentation of pulmonary nodules in thoracic CT scans: a region growing approach. IEEE Trans. Med. Imaging 27:467–480, 2008.

    Article  CAS  PubMed  Google Scholar 

  9. Devan, L., Santosham, R., and Hariharan, R., Automated texture-based characterization of fibrosis and carcinoma using low-dose lung CT images. Int. J. Imaging. Syst. Technol. 24:39–44, 2014.

    Article  Google Scholar 

  10. Doi, K., Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput. Med. Imaging Graph. 31:198–211, 2007.

    Article  PubMed  PubMed Central  Google Scholar 

  11. El-Baz, A., Suri, J. S., Lung imaging and computer aided diagnosis. CRC Press. 2011.

  12. Erasmus, J. J., Gladish, G. W., Broemeling, L., Sabloff, B. S., Truong, M. T., Herbst, R. S., and Munden, R. F., Interobserver and intraobserver variability in measurement of non-small-cell carcinoma lung lesions: implications for assessment of tumor response. J. Clin. Oncol. 21:2574–2582, 2003.

    Article  PubMed  Google Scholar 

  13. Gupta, A., Kesavabhotla, K., Baradaran, H., Kamel, H., Pandya, A., Giambrone, A. E., Wright, D., Pain, K. J., Mtui, E. E., Suri, J. S., and Sanelli, P. C., Plaque echolucency and stroke risk in asymptomatic carotid stenosis a systematic review and meta-analysis. Stroke 46:91–97, 2015.

    Article  PubMed  Google Scholar 

  14. Farag, A., Suri, J. S., Deformable models: theory and biomaterial applications. Springer Sci. Bus. Media. 2007.

  15. Hollander, M., Wolfe, D. A., Nonparametric statistical methods, John Wiley & Sons. 1999.

  16. Hopper, K. D., Kasales, C. J., Van Slyke, M. A., Schwartz, T. A., TenHave, T. R., Jozefiak, J. A., and Van Slyke, M. A., Analysis of interobserver and intraobserver variability in CT tumor measurements. AJR Am. J. Roentgenol. 167:851–854, 1996.

    Article  CAS  PubMed  Google Scholar 

  17. Hu, S., Hoffman, E. A., and Reinhardt, J. M., Automatic lung segmentation for accurate quantitation of volumetric X-ray CT images. IEEE Trans. Med. Imaging 20:490–498, 2001.

    Article  CAS  PubMed  Google Scholar 

  18. Huttenlocher, P., Klanderman, G. A., and Rucklidge, W. J., Comparing images using the Hausdorff distance. IEEE Trans. Pattern Anal. Mach. Intell. 15:850–863, 1993. doi:10.1109/34.232073.

    Article  Google Scholar 

  19. Jackson, S., Research methods and statistics: a critical thinking approach. Cengage Learni. 2011.

  20. Ko, J. P., Rusinek, H., Jacobs, E. L., Babb, J. S., Betke, M., Naidich, D. P., McGuinness, G., and Naidich, D. P., Small pulmonary nodules: volume measurement at chest CT—phantom study 1. Radiology 228:864–870, 2003.

    Article  PubMed  Google Scholar 

  21. Korfiatis, P., Kalogeropoulou, C., Karahaliou, A., Kazantzi, A., Skiadopoulos, S., and Costaridou, L., Texture classification-based segmentation of lung affected by interstitial pneumonia in high-resolution CT. Med. Phys. 35:5290–5302, 2008.

    Article  PubMed  Google Scholar 

  22. Kuhnigk, J. J. M., Hahn, H., Hindennach, M., Dicken, V., Krass, S., and Peitgen, H. O., Lung lobe segmentation by anatomy-guided 3D watershed transform. Med. Imaging 2003:1482–1490, 2003.

    Google Scholar 

  23. Martin Bland, J., and Altman, D., Statistical methods for assessing agreement between two methods of clinical measurement. Lancet 327:307–310, 1986.

    Article  Google Scholar 

  24. Molinari, F., Mantovani, A., Deandrea, M., Limone, P., Garberoglio, R., and Suri, J. S., Characterization of single thyroid nodules by contrast-enhanced 3-D ultrasound. Ultrasound Med. Biol. 36:1616–1625, 2010.

    Article  PubMed  Google Scholar 

  25. Molinari, F., Meiburger, K. M., Zeng, G., Nicolaides, A., and Suri, J. S., CAUDLES-EF: carotid automated ultrasound double line extraction system using edge flow. Ultrasound Imaging. 129–162. 2012.

  26. Nagaraj, S., Rao, G. N., and Koteswararao, K., The role of pattern recognition in computer-aided diagnosis and computer-aided detection in medical imaging: a clinical validation. Int. J. Comput. Appl. 8:18–22, 2010.

    Google Scholar 

  27. Nandy, K., Interactive segmentation and tracking in optical microscopic images. Cytometry Part A 81:357–359, 2012.

    Article  Google Scholar 

  28. Nery, F., Silva, J. S., Ferreira, N. C., Caramelo, F. J., and Faustino, R., Automated identification of the lung contours in positron emission tomography. .J Instrum. 8:C03018, 2013.

    Article  Google Scholar 

  29. Nery, F., Silvestre, J., Ferreira, N. C., Caramelo, F., Silva, J. S., and Faustino, R., An algorithm for the pulmonary border extraction in PET Images. Procedia Technol. 5:876–884, 2012.

    Article  Google Scholar 

  30. Noor, N. M., Than, J. C. M., Rijal, O. M., Kassim, R. M., Yunus, A., Zeki, A. A., Anzidei, M., Saba, L., and Suri, S. J., Automatic lung segmentation using control feedback system: morphology and texture paradigm. J. Med. Syst. 39:1–18, 2015.

    Article  Google Scholar 

  31. O’Dwyer, D. N., Armstrong, M. E., Cooke, G., Dodd, J. D., Veale, D. J., and Donnelly, S. C., Rheumatoid arthritis (RA) associated interstitial lung disease (ILD). Eur. J. Intern. Med. 24:597–603, 2013.

    Article  PubMed  Google Scholar 

  32. Osareh, A., and Shadgar, B., A segmentation method of lung cavities using region aided geometric snakes. J. Med. Syst. 34:419–433, 2010.

    Article  PubMed  Google Scholar 

  33. Otsu, N., A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man. Cybern. 9:62–66, 1975.

    Google Scholar 

  34. Peroš-Golubičić, T., Sharma, O., Clinical atlas of interstitial lung disease. Springer. 2006.

  35. Pope, A., Reproducibility: intraobserver and interobserver variability, Biostat. Radiol. Springer. 125–140. 2009.

  36. van Rikxoort, E. M., and van Ginneken, B., Automated segmentation of pulmonary structures in thoracic computed tomography scans: a review. Phys. Med. Biol. 58:R187, 2013.

    Article  PubMed  Google Scholar 

  37. van Rikxoort, E. M., de Hoop, B., van de Vorst, S., Prokop, M., and van Ginneken, B., Automatic segmentation of pulmonary segments from volumetric chest CT scans. IEEE Trans. Med. Imaging 28:621–30, 2009.

    Article  PubMed  Google Scholar 

  38. Saba, L., Molinari, F., Meiburger, K. M., Acharya, U. R., Nicolaides, A., and Suri, J. S., Inter-and intra-observer variability analysis of completely automated cIMT measurement software (AtheroEdgeTM) and its benchmarking against commercial ultrasound scanner and expert Readers. Comput. Biol. Med. 43:1261–1272, 2013.

    Article  PubMed  Google Scholar 

  39. Santos, B. S., Ferreira, C., Sousa Santos, B., Silva, J. S., Silva, A., and Teixeira, L., Quantitative evaluation of a pulmonary contour segmentation algorithm in x-ray computed tomography images. Acad. Radiol. 11:868–878, 2004.

    Article  Google Scholar 

  40. Schwarz, M. I., Matthay, R. A., Sahn, S. A., Stanford, R. M., Larry, B., and Scheinhorn, D. J., Interstitial lung disease in polymyositis and dermatomyositis: analysis of six cases and review of the literature. Medicine 55:89–104, 1976.

    Article  CAS  PubMed  Google Scholar 

  41. Sharma, A. M., Gupta, A., Kumar, P. K., Rajan, J., Saba, L., Nobutaka, I., Laird, J. R., Nicolades, A., and Suri, J. S., A review on carotid ultrasound atherosclerotic tissue characterization and stroke risk stratification in machine learning framework. Curr. Atheroscler. Rep. 17:1–13, 2015.

    Article  Google Scholar 

  42. Sharman, P., and Wood-Baker, R., Interstitial lung disease due to fumes from heat-cutting polymer rope. Occup. Med. 63:451–453, 2013.

    Article  CAS  Google Scholar 

  43. Sheehan, F. H., Stewart, D. K., Dodge, H. T., Mitten, S., Bolson, E. L., and Brown, B. G., Variability in the measurement of regional left ventricular wall motion from contrast angiograms. Circulation 68:550–559, 1983.

    Article  CAS  PubMed  Google Scholar 

  44. Shrivastava, V. K., Londhe, N. D., Sonawane, R. S., and Suri, J. S., Reliable and accurate psoriasis disease classification in dermatology images using comprehensive feature space in machine learning paradigm. Expert Syst. Appl. 42:6184–6195, 2015.

    Article  Google Scholar 

  45. Shrivastava, V. K., Londhe, N. D., Sonawane, R. S., and Suri, J. S., Computer-aided diagnosis of psoriasis skin images with HOS, texture and color features: a first comparative study of its kind. Comput. Methods Programs Biomed. 126:98–108, 2016.

    Article  PubMed  Google Scholar 

  46. Singh, S., Maxwell, J., Baker, J. A., Nicholas, J. L., and Lo, J. Y., Computer-aided classification of breast masses: performance and interobserver variability of expert radiologists versus residents 1. Radiology 258:73–80, 2011.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Trivedi, R., Saba, L., Suri, J. S., 3D imaging technologies in atherosclerosis. Springer. 2015.

  48. Wang, A., and Yan, H., Delineating low-count defective-contour SPECT lung scans for PE diagnosis using adaptive dual exponential thresholding and active contours. Int. J. Imaging Syst. Technol. 20:149–154, 2010.

    Article  Google Scholar 

  49. Watadani, T., Sakai, F., Johkoh, T., and Noma, S., Interobserver variability in the CT assessment of honeycombing in the lungs. Radiology 266:936–944, 2013.

    Article  PubMed  Google Scholar 

  50. World Health Organization, World health statistics 2011. Geneva WHO. 2011.

  51. Wormanns, D., Diederich, S., and Lentschig, M., Spiral CT of pulmonary nodules: interobserver variation in assessment of lesion size. Eur. Radiol. 10:710–713, 2000.

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgments

This research was supported Research University Grant: GUP QK130000.2540.06H35, Universiti Teknologi Malaysia and Ministry of Higher Education Malaysia. We would like to thank Dr. Amir Zeki, Pulmonary and Critical Care Medicine Division, UC Davis School of Medicine, California, Sacramento, USA, as well as all the clinicians and radiologists who contributed and made this study a success. We are grateful to AtheroPoint™ LLC, Roseville, CA, USA for gracefully letting us use ImgTracer™ 1.0 software for tracing the manual borders of the lung.

Author information

Authors and Affiliations

  1. Azienda Ospedaliero Universitaria (A.O.U.) di Cagliari – Polo di Monserrato, Università di Cagliari, s.s. 554 Monserrato, Cagliari, 09045, Italy

    Luca Saba

  2. UTM Razak School of Engineering and Advanced Technology, Universiti Teknologi Malaysia, Johor Bahru, Malaysia

    Joel C. M. Than & Chue R. Ng

  3. Department of Engineering, UTM Razak School of Engineering and Advanced Technology, Universiti Teknologi Malaysia, Johor Bahru, Malaysia

    Norliza M. Noor

  4. Institute of Mathematical Sciences, Faculty of Science, University of Malaya, Kuala Lumpur, Malaysia

    Omar M. Rijal

  5. Department of Diagnostic Imaging, Kuala Lumpur Hospital, Kuala Lumpur, Malaysia

    Rosminah M. Kassim

  6. Institute of Respiratory Medicine, Kuala Lumpur, Malaysia

    Ashari Yunus

  7. Global Biomedical Technologies, Inc., Roseville, CA, USA

    Jasjit S. Suri

  8. AtheroPoint™ LLC, Roseville, CA, USA

    Jasjit S. Suri

  9. Department of Electrical Engineering (Affl.), Idaho State University, Pocatello, ID, USA

    Jasjit S. Suri

Authors
  1. Luca Saba
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joel C. M. Than
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Norliza M. Noor
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Omar M. Rijal
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rosminah M. Kassim
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ashari Yunus
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Chue R. Ng
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jasjit S. Suri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jasjit S. Suri.

Additional information

This article is part of the Topical Collection on Education & Training

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saba, L., Than, J.C.M., Noor, N.M. et al. Inter-observer Variability Analysis of Automatic Lung Delineation in Normal and Disease Patients. J Med Syst 40, 142 (2016). https://doi.org/10.1007/s10916-016-0504-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10916-016-0504-7

Keywords

Search

Navigation