Effects of Increasing Working Opportunity on Result Quality in Labor-Intensive Crowdsourcing

Abstract

When selecting workers in microtask crowdsourcing platforms, a common practice of requesters is to select qualified workers by looking at the evaluation results for the tasks in the past or by conducting qualifying tests for the tasks. This sometimes misses workers who may be able to complete some of the tasks. Increasing working opportunities for such workers has advantages not only for the workers but also for requesters because they obtain labor resources for faster completion of tasks. However, in general, an increase in the working opportunity and obtaining high-quality task results is a trade-off; if they choose workers whose skill levels are above a lower threshold to increase the number of workers, the quality of the task will be undermined. In this paper, we address the problem of improving the trade-off in labor-intensive crowdsourcing by exploring different task assignment strategies. For that purpose, we apply Item Response Theory to evaluate the skills of workers and the difficulty of tasks and devise an algorithm for assigning tasks in such a way that the variance in the number of tasks assigned among workers is minimized trying to take advantage of the potential parallelism of crowdsourcing. Second, we address the problem that the difficulty of the tasks is unknown in advance. We explore an approach that uses ML outputs for difficulty estimation. This paper reports on our experimental result, which shows the potential of this approach, and discusses when this approach is effective.