Abstract
This paper investigates what quality of ground truth might be obtained when crowdsourcing specialist medical imaging ground truth from non-experts. Following basic tuition, 34 volunteer participants independently delineated regions belonging to 7 pathological patterns in 20 scans according to expert-provided pattern labels. Participants’ annotations were compared to a set of reference annotations using Dice similarity coefficient (DSC), and found to range between 0.41 and 0.77. The reference repeatability was 0.81. Analysis of prior imaging experience, annotation behaviour, scan ordering and time spent showed that only the last was correlated with annotation quality. Multiple observers combined by voxelwise majority vote outperformed a single observer, matching the reference repeatability for 5 of 7 patterns. In conclusion, crowdsourcing from non-experts yields acceptable quality ground truth, given sufficient expert task supervision and a sufficient number of observers per scan.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Albarqouni, S., Matl, S., Baust, M., Navab, N., Demirci, S.: Playsourcing: a novel concept for knowledge creation in biomedical research. In: Carneiro, G., et al. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 269–277. Springer, Cham (2016). doi:10.1007/978-3-319-46976-8_28
Anthimopoulos, M., Christodoulidis, S., Ebner, L., Christe, A., Mougiakakou, S.: Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans. Med. Imaging 35(5), 1207–1216 (2016)
Cheplygina, V., Perez-Rovira, A., Kuo, W., Tiddens, H.A.W.M., de Bruijne, M.: Early experiences with crowdsourcing airway annotations in chest CT. In: Carneiro, G., et al. (eds.) LABELS/DLMIA -2016. LNCS, vol. 10008, pp. 209–218. Springer, Cham (2016). doi:10.1007/978-3-319-46976-8_22
Depeursinge, A., Vargas, A., Platon, A., Geissbuhler, A., Poletti, P.A., Müller, H.: Building a reference multimedia database for interstitial lung diseases. Comput. Med. Imaging Graph. 36(3), 227–238 (2012)
Hansell, D.M., Bankier, A.A., MacMahon, H., McLoud, T.C., Müller, N.L., Remy, J.: Fleischner society: glossary of terms for thoracic imaging. Radiology 246(3), 697–722 (2008)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: Neural Information Processing Systems (2014)
Hossain, M., Kauranen, I.: Crowdsourcing: a comprehensive literature review. Strateg. Outsourcing Int. J. 8(1), 1753–8297 (2015)
Humphries, S.M., Yagihashi, K., Huckleberry, J., Rho, B.H., Schroeder, J.D., Strand, M., Schwarz, M.I., Flaherty, K.R., Kazerooni, E.A., van Beek, E.J.R., Lynch, D.A.: Idiopathic pulmonary fibrosis: data-driven textural analysis of extent of fibrosis at baseline and 15-month follow-up. Radiology 5, 161177 (2017)
Langerak, T.R., van der Heide, U.A., Kotte, A.N., Viergever, M.A., van Vulpen, M., Pluim, J.P.: Label fusion in atlas-based segmentation using a selective and iterative method for performance level estimation (SIMPLE). IEEE Trans. Med. Imaging 29(12), 2000–2008 (2010)
Van Leemput, K., Sabuncu, M.R.: A cautionary analysis of STAPLE using direct inference of segmentation truth. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8673, pp. 398–406. Springer, Cham (2014). doi:10.1007/978-3-319-10404-1_50
Luengo-Oroz, M.A., Arranz, A., Frean, J.: Crowdsourcing malaria parasite quantification: an online game for analyzing images of infected thick blood smears. J. Med. Internet Res. 14(6), e167 (2012)
Piciucchi, S., Tomassetti, S., Ravaglia, C., Gurioli, C., Gurioli, C., Dubini, A., Carloni, A., Chilosi, M., Colby, T.V., Poletti, V.: From traction bronchiectasis to honeycombing in idiopathic pulmonary fibrosis: a spectrum of bronchiolar remodeling also in radiology? BMC Pulm. Med. 16(1), 87 (2016)
Salisbury, M.L., Lynch, D.A., van Beek, E.J.R., Kazerooni, E.A., Guo, J., Xia, M., Murray, S., Anstrom, K.A., Yow, E., Martinez, F.J., Hoffman, E.A., Flaherty, K.R.: Idiopathic pulmonary fibrosis: the association between the adaptive multiple features method and fibrosis outcomes. Am. J. Respir. Crit. Care Med. 195(7), 921–929 (2017)
Schlesinger, D., Jug, F., Myers, G., Rother, C., Kainmuller, D.: Crowdsourcing image segmentation with aSTAPLE. arXiv (2017)
Warfield, S.K., Zhou, K.H., Wells, W.M.: Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Trans. Med. Imaging 23(7), 903–921 (2004)
Acknowledgements
Many thanks to Phil Tolland who developed the software for the ground truth collection tool, and to all of the employees at Toshiba Medical Visualization Systems who took part in this study: Allan Barklie, Erin Beveridge, Antony Brown, Gerald Chau, Alasdair Corbett, Ross Davies, Matt Daykin, Ben Docherty, Venkatesh Gaddam, Keith Goatman, Marta Guarisco, Joseph Henry, Corné Hoogendoorn, Pia Kullik, Aneta Lisowska, Steve Magness, Craig Matear, James Matthews, Chris McGough, Haritha Miryala, Brian Mohr, Costas Plakas, Ian Poole, Marco Razeto, Faye Riley, Matt Shepherd, Simeon Skopalik, Andy Smout, Ken Sutherland, Paul Thomson, Phil Tolland, John Tough, Aidan Wellington and Gavin Wheeler.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
O’Neil, A.Q., Murchison, J.T., van Beek, E.J.R., Goatman, K.A. (2017). Crowdsourcing Labels for Pathological Patterns in CT Lung Scans: Can Non-experts Contribute Expert-Quality Ground Truth?. In: Cardoso, M., et al. Intravascular Imaging and Computer Assisted Stenting, and Large-Scale Annotation of Biomedical Data and Expert Label Synthesis. LABELS STENT CVII 2017 2017 2017. Lecture Notes in Computer Science(), vol 10552. Springer, Cham. https://doi.org/10.1007/978-3-319-67534-3_11
Download citation
DOI: https://doi.org/10.1007/978-3-319-67534-3_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-67533-6
Online ISBN: 978-3-319-67534-3
eBook Packages: Computer ScienceComputer Science (R0)