Skip to main content

Advertisement

SpringerLink
Log in

Crowdsourcing for error detection in cortical surface delineations

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

    We’re sorry, something doesn't seem to be working properly.

    Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Abstract

Purpose

With the recent trend toward big data analysis, neuroimaging datasets have grown substantially in the past years. While larger datasets potentially offer important insights for medical research, one major bottleneck is the requirement for resources of medical experts needed to validate automatic processing results. To address this issue, the goal of this paper was to assess whether anonymous nonexperts from an online community can perform quality control of MR-based cortical surface delineations derived by an automatic algorithm.

Methods

So-called knowledge workers from an online crowdsourcing platform were asked to annotate errors in automatic cortical surface delineations on 100 central, coronal slices of MR images.

Results

On average, annotations for 100 images were obtained in less than an hour. When using expert annotations as reference, the crowd on average achieves a sensitivity of 82 % and a precision of 42 %. Merging multiple annotations per image significantly improves the sensitivity of the crowd (up to 95 %), but leads to a decrease in precision (as low as 22 %).

Conclusion

Our experiments show that the detection of errors in automatic cortical surface delineations generated by anonymous untrained workers is feasible. Future work will focus on increasing the sensitivity of our method further, such that the error detection tasks can be handled exclusively by the crowd and expert resources can be focused on error correction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. The instructions for the KWs can be seen here: http://melanie.clausundmelanie.de/projects/crowd.

  2. http://freesurfer.net.

References

  1. Ashburner J (2012) SPM: a history. Neuroimage 62(2):791–800

    Article  PubMed  Google Scholar 

  2. Brabham DC (2008) Crowdsourcing as a model for problem solving an introduction and cases. Converg Int J Res New Media Technol 14(1):75–90

    Article  Google Scholar 

  3. Brady CJ, Villanti AC, Pearson JL, Kirchner TR, Gupta OP, Shah CP (2014) Rapid grading of fundus photographs for diabetic retinopathy using crowdsourcing. J Med Internet Res 16(10):e233

  4. Chen C, White L, Kowalewski T, Aggarwal R, Lintott C, Comstock B, Kuksenok K, Aragon C, Holst D, Lendvay T (2014) Crowd-sourced assessment of technical skills: a novel method to evaluate surgical performance. J Surg Res 187(1):65–71

    Article  PubMed  Google Scholar 

  5. Dale AM, Fischl B, Sereno MI (1999) Cortical surface-based analysis: I. Segmentation and surface reconstruction. Neuroimage 9(2):179–194

    Article  CAS  PubMed  Google Scholar 

  6. Fischl B (2012) FreeSurfer. Neuroimage 62(2):774–781

    Article  PubMed  PubMed Central  Google Scholar 

  7. Fischl B, Sereno MI, Dale AM (1999) Cortical surface-based analysis: II. Inflation, flattening, and a surface-based coordinate system. Neuroimage 9(2):195–207

    Article  CAS  PubMed  Google Scholar 

  8. Foncubierta Rodríguez A, Müller H (2012) Ground truth generation in medical imaging: a crowdsourcing-based iterative approach. In: Proceedings of the ACM multimedia workshop on crowdsourcing for multimedia. ACM, pp 9–14

  9. Fritzsche KH, Neher PF, Reicht I, van Bruggen T, Goch C, Reisert M, Nolden M, Zelzer S, Meinzer HP, Stieltjes B (2012) MITK diffusion imaging. Methods Inf Med 51(5):441

    Article  CAS  PubMed  Google Scholar 

  10. Haahr M, Hansen D, Fisher P, Svarer C, Stenbæk D, Madsen K, Madsen J, Holst JJ, Baaré W, Hojgaard L, Almdal T, Knudsen G (2015) Central 5-HT neurotransmission modulates weight loss following gastric bypass surgery in obese individuals. J Neurosci 35(14):5884–5889

  11. de Herrera AGS, Foncubierta-Rodríguez A, Markonis D, Schaer R, Müller H (2014) Crowdsourcing for medical image classification. Swiss Med Inform 30

  12. Jannin P, Grova C, Maurer CR Jr (2006) Model for defining and reporting reference-based validation protocols in medical image processing. Int J Comput Assist Radiol Surg 1(2):63–73

    Article  Google Scholar 

  13. Jenkinson M, Beckmann CF, Behrens TE, Woolrich MW, Smith SM (2012) FSL. Neuroimage 62(2):782–790

    Article  PubMed  Google Scholar 

  14. Maier-Hein L, Kondermann D, Roß T, Mersmann S, Heim E, Bodenstedt S, Kenngott HG, Sanchez A, Wagner M, Preukschas A, Wekerle AL, Helfert S, März K, Mehrabi A, Speidel S, Stock C (2015) Crowdtruth validation: a new paradigm for validating algorithms that rely on image correspondences. Int J Comput Assist Radiol Surg 10(8):1201–1212

  15. Maier-Hein L, Mersmann S, Kondermann D, Bodenstedt S, Sanchez A, Stock C, Kenngott HG, Eisenmann M, Speidel S (2014) Can masses of non-experts train highly accurate image classifiers? In: Medical image computing and computer-assisted intervention (MICCAI 2014). Springer, New York, pp 438–445

  16. Maier-Hein L, Mersmann S, Kondermann D, Stock C, Kenngott HG, Sanchez A, Wagner M, Preukschas A, Wekerle AL, Helfert S, Bodenstedt S, Speidel S (2014) Crowdsourcing for reference correspondence generation in endoscopic images. In: Medical image computing and computer-assisted intervention (MICCAI 2014). Springer, New York, pp 349–356

  17. Mavandadi S, Dimitrov S, Feng S, Yu F, Sikora U, Yaglidere O, Padmanabhan S, Nielsen K, Ozcan A (2012) Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study. PLoS One 7(5):37245

    Article  Google Scholar 

  18. Nguyen TB, Wang S, Anugu V, Rose N, McKenna M, Petrick N, Burns JE, Summers RM (2012) Distributed human intelligence for colonic polyp classification in computer-aided detection for CT colonography. Radiology 262(3):824–833

    Article  PubMed  PubMed Central  Google Scholar 

  19. Ranard BL, Ha YP, Meisel ZF, Asch DA, Hill SS, Becker LB, Seymour AK, Merchant RM (2014) Crowdsourcing harnessing the masses to advance health and medicine, a systematic review. J Gen Intern Med 29(1):187–203

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

Collection of data that were included in the study was supported by the Lundbeck Foundation Center of Excellence Cimbi (R90-A7722). Melanie Ganz’ research was supported by the Carlsberg Foundation (2013-01-0502) and National Institutes of Health (NIH) (5R21EB018964-02). The work was further conducted within the setting of SFB TRR 125: cognition-guided surgery funded by the German Research Foundation (DFG) (Project A02).

Author information

Authors and Affiliations

  1. Neurobiology Research Unit, Center for Integrated Molecular Brain Imaging, Rigshospitalet, Juliane Maries Vej 28, 2100, Copenhagen, Denmark

    Melanie Ganz & Gitte Moos Knudsen

  2. Pallas Ludens GmbH, Im Bosseldorn 29, 69126, Heidelberg, Germany

    Daniel Kondermann & Jonas Andrulis

  3. Computer-assisted Interventions, German Cancer Research Center (DKFZ), Im Neuenheimer Feld 280, 69120, Heidelberg, Germany

    Lena Maier-Hein

Authors
  1. Melanie Ganz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Kondermann
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jonas Andrulis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gitte Moos Knudsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lena Maier-Hein
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Melanie Ganz.

Ethics declarations

Conflict of interest

Melanie Ganz, Daniel Kondermann, Jonas Andrulis and Lena Maier-Hein declare that they have no conflict of interest. Gitte Moos Knudsen has been an invited lecturer at Pfizer A/S, worked as a consultant and received grants from H. Lundbeck A/S and is a stockholder of Novo Nordisk/Novozymes. Furthermore, she is on the board of directors of the BrainPrize and Elsass Foundation and the Advisory Board of the Kristian Jebsen Foundation and has authored for FADL and served as editor for Elsevier (IJNP).

Ethical standards

All procedures followed were in accordance with the ethical standards of the responsible committees on human experimentation (institutional and national) and with the Declaration of Helsinki 1975, as revised in 2008. Informed consent was obtained from all patients for being included in the study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ganz, M., Kondermann, D., Andrulis, J. et al. Crowdsourcing for error detection in cortical surface delineations. Int J CARS 12, 161–166 (2017). https://doi.org/10.1007/s11548-016-1445-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-016-1445-9

Keywords

Search

Navigation