Skip to main content

Advertisement

SpringerLink
Log in

Crowdtruth validation: a new paradigm for validating algorithms that rely on image correspondences

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Feature tracking and 3D surface reconstruction are key enabling techniques to computer-assisted minimally invasive surgery. One of the major bottlenecks related to training and validation of new algorithms is the lack of large amounts of annotated images that fully capture the wide range of anatomical/scene variance in clinical practice. To address this issue, we propose a novel approach to obtaining large numbers of high-quality reference image annotations at low cost in an extremely short period of time.

Methods

The concept is based on outsourcing the correspondence search to a crowd of anonymous users from an online community (crowdsourcing) and comprises four stages: (1) feature detection, (2) correspondence search via crowdsourcing, (3) merging multiple annotations per feature by fitting Gaussian finite mixture models, (4) outlier removal using the result of the clustering as input for a second annotation task.

Results

On average, 10,000 annotations were obtained within 24 h at a cost of $100. The annotation of the crowd after clustering and before outlier removal was of expert quality with a median distance of about 1 pixel to a publically available reference annotation. The threshold for the outlier removal task directly determines the maximum annotation error, but also the number of points removed.

Conclusions

Our concept is a novel and effective method for fast, low-cost and highly accurate correspondence generation that could be adapted to various other applications related to large-scale data annotation in medical image computing and computer-assisted interventions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. http://archive.wired.com/wired/archive/14.06/crowds.html.

  2. eyewire.org.

  3. In this manuscript, the term data set will generally refer to a set of endoscopic image pairs with annotations.

  4. http://turkopticon.ucsd.edu.

References

  1. Maier-Hein L, Groch A, Bartoli A, Bodenstedt S, Boissonnat G, Chang PL, Clancy NT, Elson DS, Haase S, Heim E, Hornegger J, Jannin P, Kenngott H, Kilgus T, Muller-Stich B, Oladokun D, Rohl S, Dos Santos TR, Schlemmer HP, Seitel A, Speidel S, Wagner M, Stoyanov D (2014) Comparative validation of single-shot optical techniques for laparoscopic 3-D surface reconstruction. IEEE Trans Med Imaging 33:1913–1930

    Article  CAS  PubMed  Google Scholar 

  2. Von Ahn L, Dabbish L (2004) Labeling images with a computer game. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 319–326

  3. Estellés-Arolas E, González-Ladrón-de Guevara F (2012) Towards an integrated crowdsourcing definition. J Inf Sci 38:189–200

    Article  Google Scholar 

  4. Chen JJ, Menezes NJ, Bradley AD, North T (2011) Opportunities for crowdsourcing research on amazon mechanical turk. Interfaces 5. Jg., No. 3

  5. Russell BC, Torralba A, Murphy KP, Freeman WT (2008) LabelMe: a database and web-based tool for image annotation. Int J Comput Vis 77(1–3):157–173

    Article  Google Scholar 

  6. Khatib F, DiMaio F, Cooper S, Kazmierczyk M, Gilski M, Krzywda S, Zabranska H, Pichova I, Thompson J, Popovic Z, Jaskolski M, Baker D (2011) Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nat Struct Mol Biol 18:1175–1177

  7. Ranard B, Ha Y, Meisel Z, Asch D, Hill S, Becker L, Seymour A, Merchant R (2014) Crowdsourcing—harnessing the masses to advance health and medicine, a systematic review. J Gen Intern Med 29:187–203

    Article  PubMed Central  PubMed  Google Scholar 

  8. Mavandadi S, Dimitrov S, Feng S, Yu F, Sikora U, Yaglidere O, Padmanabhan S, Nielsen K, Ozcan A (2012) Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study. PLoS ONE 7:e37245

    Article  CAS  PubMed Central  PubMed  Google Scholar 

  9. Nguyen TB, Wang S, Anugu V, Rose N, McKenna M, Petrick N, Burns JE, Summers RM (2012) Distributed human intelligence for colonic polyp classification in computer-aided detection for CT colonography. Radiology 262:824–833

    Article  PubMed Central  PubMed  Google Scholar 

  10. Foncubierta Rodríguez A, Müller H (2012) Ground truth generation in medical imaging: a crowdsourcing-based iterative approach. In: Proceedings of the ACM multimedia 2012 workshop on crowdsourcing for multimedia. CrowdMM’12. ACM, New York, NY, USA, pp 9–14

  11. Chen C, White L, Kowalewski T, Aggarwal R, Lintott C, Comstock B, Kuksenok K, Aragon C, Holst D, Lendvay T (2014) Crowd-sourced assessment of technical skills: a novel method to evaluate surgical performance. J Surg Res 187:65–71

    Article  PubMed  Google Scholar 

  12. Maier-Hein L, Mersmann S, Kondermann D, Bodenstedt S, Sanchez A, Stock C, Kenngott HG, Eisenmann M, Speidel S (2014) Can masses of non-experts train highly accurate image classifiers? In: Golland P, Hata N, Barillot C, Hornegger J, Howe R (eds) Medical image computing and computer-assisted intervention—MICCAI 2014, vol 8674. Lecture notes in computer science. Springer, pp 438–445

  13. Maier-Hein L, Mersmann S, Kondermann D, Stock C, Kenngott HG, Sanchez A, Wagner M, Preukschas A, Wekerle AL, Helfert S, Bodenstedt S, Speidel S (2014) Crowdsourcing for reference correspondence generation in endoscopic images. In: Golland P, Hata N, Barillot C, Hornegger J, Howe R (eds) Medical image computing and computer-assisted intervention—MICCAI 2014, vol 8674. Lecture notes in computer science. Springer, pp 349–356

  14. Bay H, Tuytelaars T, Van Gool L (2006) SURF: speeded up robust features. In: European conference on computer vision (ECCV), vol 3951. Lecture notes in computer science, pp 404–417

  15. Morris RR, Dontcheva M, Gerber EM (2012) Priming for better performance in microtask crowdsourcing environments. IEEE Internet Comput 16:13–19

    Article  Google Scholar 

  16. Fraley C, Raftery AE (2002) Model-based clustering, discriminant analysis and density estimation. J Am Stat Assoc 97:611–631

    Article  Google Scholar 

  17. Fraley C, Raftery AE, Murphy TB, Scrucca L (2012) MCLUST version 4 for R: normal mixture modeling for model-based clustering, classification, and density estimation. Technical report, Technical report no. 597, Department of Statistics, University of Washington

  18. Puerto G, Mariottini GL (2012) A comparative study of correspondence-search algorithms in MIS images. In: International conference on medical image computing and computer-assisted intervention (MICCAI), pp 625–633

  19. R Core Team (2014) R: A language and environment for statistical computing. R Core Team, Vienna

    Google Scholar 

  20. Pinheiro J, Bates D, DebRoy S, Sarkar D (2014) R Core Team: NLME: linear and nonlinear mixed effects models. R package version 3.1-118

Download references

Acknowledgments

This work was conducted within the setting of SFB TRR 125: Cognition-guided surgery funded by the German Research Foundation (DFG) (Projects A02 and A01). It was further sponsored by the European Social Fund of the State of Baden-Württemberg and the Klaus Tschira Foundation.

Author information

Authors and Affiliations

  1. Junior Group Computer-Assisted Interventions, German Cancer Research Center (DKFZ), Heidelberg, Germany

    Lena Maier-Hein, Tobias Roß, Sven Mersmann, Eric Heim & Keno März

  2. Heidelberg Collaboratory for Image Processing, University of Heidelberg, Heidelberg, Germany

    Daniel Kondermann & Alexandro Sanchez

  3. Institute for Anthropomatics, Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany

    Sebastian Bodenstedt & Stefanie Speidel

  4. Department of General, Visceral and Transplant Surgery, University of Heidelberg, Heidelberg, Germany

    Hannes Götz Kenngott, Martin Wagner, Anas Preukschas, Anna-Laura Wekerle, Stefanie Helfert & Arianeb Mehrabi

  5. Institute of Medical Biometry and Informatics, University of Heidelberg, Heidelberg, Germany

    Christian Stock

Authors
  1. Lena Maier-Hein
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Kondermann
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tobias Roß
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sven Mersmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Eric Heim
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sebastian Bodenstedt
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Hannes Götz Kenngott
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Alexandro Sanchez
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Martin Wagner
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Anas Preukschas
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Anna-Laura Wekerle
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Stefanie Helfert
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Keno März
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Arianeb Mehrabi
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Stefanie Speidel
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Christian Stock
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lena Maier-Hein.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Maier-Hein, L., Kondermann, D., Roß, T. et al. Crowdtruth validation: a new paradigm for validating algorithms that rely on image correspondences. Int J CARS 10, 1201–1212 (2015). https://doi.org/10.1007/s11548-015-1168-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-015-1168-3

Keywords

Search

Navigation