Crowdsourcing label quality: a theoretical analysis

Abstract

Crowdsourcing has been an effective and efficient paradigm for providing labels for large-scale unlabeled data. In the past few years, many methods have been developed for inferring labels from the crowd, but few theoretical analyses have been presented to support this popular human-machine interaction process. In this paper, we theoretically study the quality of labels inferred from crowd workers by majority voting and provide an analysis of label quality that shows that the label error rate decreases exponentially with the number of workers selected for each task. We also study the problem of eliminating low-quality workers from the crowd, and provide a conservative condition for eliminating low-quality workers without eliminating any non-low-quality worker with high probability. We also provide an aggressive condition for eliminating all low-quality workers with high probability.

摘要

创新点

众包目前已成为一种给大数据提供标记的有效范式。在过去的几年中, 出现了很多依据众包提供的伪标记来推理数据真实标记的算法, 但很少有分析工作出现来为这种流行的人机交互过程提供理论支撑。本文从理论上研究众包中利用多数投票策略推理标记的质量, 理论分析表明标记的错误率随着为同一任务分配雇员数的增加呈指数级降低。此外, 本文还从理论上分析如何去除低质量的雇员, 提出一种能够以较高概率去除低质量雇员却不去除任何非低质量雇员的保守条件和一种能够以较高概率去除所有低质量雇员的激进条件。