Evaluation of Image Annotation Using Amazon Mechanical Turk in ImageCLEF

Abstract

With the increasing amount of digital information in the Web and on personal computers, the need for systems that are capable of automated indexing, searching, and organizing multimedia documents is incessantly growing. Automated systems have to retrieve information with high performance in order to be accepted by industry and end users. Multimedia retrieval systems are often evaluated on different test collections with different performance measures, which makes the comparison of retrieval performance impossible and limits the benefits of the approaches. Benchmarking campaigns counteract these tendencies and establish an objective comparison among the performance of different approaches by posing challenging tasks and by pushing the availability of test collections, topics, and performance measures. As part of the THESEUS research program, Fraunhofer IDMT organized the “Visual Concept Detection and Annotation Task” (VCDT) of the international benchmark ImageCLEF, with the goal of enabling the comparison of technologies developed within THESEUS CTC to international developments. While the test collection in 2009 was assessed with expert knowledge, the relevance assessments for the task have been acquired in a crowdsourcing approach since 2010 by using the platform of Amazon Mechanical Turk (MTurk). In this article the evaluation of THESEUS core technologies within ImageCLEF is explained in detail. A special focus lies on the acquisition of ground truth data using MTurk. Advantages and disadvantages of this approach are discussed and best practices are shared.