A Study on the Influence of the Number of MTurkers on the Quality of the Aggregate Output

Abstract

Recent years have seen an increased interest in crowdsourcing as a way of obtaining information from a large group of workers at a reduced cost. In general, there are arguments for and against using multiple workers to perform a task. On the positive side, multiple workers bring different perspectives to the process, which may result in a more accurate aggregate output since biases of individual judgments might offset each other. On the other hand, a larger population of workers is more likely to have a higher concentration of poor workers, which might bring down the quality of the aggregate output.

In this paper, we empirically investigate how the number of workers on the crowdsourcing platform Amazon Mechanical Turk influences the quality of the aggregate output in a content-analysis task. We find that both the expected error in the aggregate output as well as the risk of a poor combination of workers decrease as the number of workers increases.

Moreover, our results show that restricting the population of workers to up to the overall top 40 % workers is likely to produce more accurate aggregate outputs, whereas removing up to the overall worst 40 % workers can actually make the aggregate output less accurate. We find that this result holds due to top-performing workers being consistent across multiple tasks, whereas worst-performing workers tend to be inconsistent. Our results thus contribute to a better understanding of, and provide valuable insights into, how to design more effective crowdsourcing processes.

Appendices

A Description of the Texts

We describe in this appendix the texts used in our experiments as well as the gold-standard scores.

1.1 Text 1

An excerpt from the “Sonnet XVII” by Neruda [9]. Intentionally misspelled words are highlighted in bold.

“I do not love you as if you was salt-rose, or topaz

or the arrown of carnations that spread fire:

I love you as certain dark things are loved,

secretly, between the shadown and the soul”

The gold-standard scores for the criteria grammar, clarity, and relevance are, respectively, 1, 2, and 2.

1.2 Text 2

An excerpt from “The Cow” by Taylor et al. [12]. Intentionally misspelled words are highlighted in bold.

“THANK you, prety cow, that made

Plesant milk to soak my bread,

Every day and every night,

Warm, and fresh, and sweet, and white.”

The gold-standard scores for the criteria grammar, clarity, and relevance are, respectively, 1, 2, and 1.

1.3 Text 3

Words randomly generated in a semi-structured way. Each line starts with a noun followed by a verb in a wrong verb form. In order to mimic a poetic writing style, all the words in the same line start with a similar letter.

“Baby bet binary boundaries bubbles

Carlos cease CIA conditionally curve

Daniel deny disease domino dumb

Faust fest fierce forced furbished”

The gold-standard scores for the criteria grammar, clarity, and relevance are, respectively, 0, 0, and 0.

B Experimental Results

Table 1 shows the numerical results from all the analysis performed in this paper.

Table 1. The average error, the standard deviation of the errors, and the maximum error per text for different populations of agents. All the values are rounded to 3 decimal places.