CAPTCHA for crowdsourced image annotation: directions and efficiency analysis

Image annotation plays an important role in image retrieval process, especially when it comes to content-based image retrieval. In order to compensate the intrinsic weakness of machines in performing cognitive task of (human-like) image annotation, leveraging humans’ knowledge and abilities in the form of crowdsourcing-based annotation have gained momentum. Among various approaches for this purpose, an innovative one is integrating the annotation process into the CAPTCHA workflow. In this paper, the current state of the research works in the field and experimental efficiency analysis of this approach are investigated.

At first, and with the aim of presenting a current state report of research studies in the field, a comprehensive literature review is provided. Then, several experiments and statistical analyses are conducted to investigate how CAPTCHA-based image annotation is reliable, accurate and efficient.

In addition to study of current trends and best practices for CAPTCHA-based image annotation, the experimental results demonstrated that despite some intrinsic limitations on leveraging the CAPTCHA as a crowdsourcing platform, when the challenge, i.e. annotation task, is selected and designed appropriately, the efficiency of CAPTCHA-based image annotation can outperform traditional approaches. Nonetheless, there are several design considerations that should be taken into account when the CAPTCHA is used as an image annotation platform.

To the best of the authors’ knowledge, this is the first study to analyze different aspects of the titular topic through exploration of the literature and experimental investigation. Therefore, it is anticipated that the outcomes of this study can draw a roadmap for not only CAPTCHA-based image annotation but also CAPTCHA-mediated crowdsourcing and even image annotation.