Djellel Difallah
Panos Ipeirotis
ABSTRACT
We present an analysis of the population dynamics and demographics of Amazon Mechanical Turk workers based on the results of the survey that we conducted over a period of 28 months, with more than 85K responses from 40K unique participants. The demographics survey is ongoing (as of November 2017), and the results are available at http://demographics.mturk-tracker.com: we provide an API for researchers to download the survey data. We use techniques from the field of ecology, in particular, the capture-recapture technique, to understand the size and dynamics of the underlying population. We also demonstrate how to model and account for the inherent selection biases in such surveys. Our results indicate that there are more than 100K workers available in Amazon»s crowdsourcing platform, the participation of the workers in the platform follows a heavy-tailed distribution, and at any given time there are more than 2K active workers. We also show that the half-life of a worker on the platform is around 12-18 months and that the rate of arrival of new workers balances the rate of departures, keeping the overall worker population relatively stable. Finally, we demonstrate how we can estimate the biases of different demographics to participate in the survey tasks, and show how to correct such biases. Our methodology is generic and can be applied to any platform where we are interested in understanding the dynamics and demographics of the underlying user population.
- Steven C. Amstrup, Trent L. McDonald, and Bryan F. J. Manly. 2010. Handbook of Capture-Recapture Analysis. Princeton University Press, Princeton, NJ.Google Scholar
- Joanna J. Arch and Alaina L. Carr. 2016. Using Mechanical Turk for research on cancer survivors. Psycho-Oncology Forthcoming (2016), --. PON-15-0731.R1.Google Scholar
- Antonio A. Arechar, Simon Gächter, and Lucas Molleman. 2017. Conducting interactive experiments online. Experimental Economics Forthcoming (09 May 2017), 1--33.Google Scholar
- K. P. Burnham and W. S. Overton. 1978. Estimation of the Size of a Closed Population when Capture Probabilities vary Among Animals. Biometrika 65, 3 (1978), 625--633.Google ScholarCross Ref
- Chris Callison-Burch and Mark Dredze. 2010. Creating Speech and Language Data with Amazon's Mechanical Turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk (CSLDAMT '10). Association for Computational Linguistics, Stroudsburg, PA, USA, 1--12. Google ScholarDigital Library
- Krista Casler, Lydia Bickel, and Elizabeth Hackett. 2013. Separate but Equal? A Comparison of Participants and Data Gathered via Amazon's MTurk, Social Media, and Face-to-Face Behavioral Testing. Computers in Human Behavior 29 (2013), 2156--2160. Google ScholarDigital Library
- Jesse Chandler, Gabriele Paolacci, and Pam Mueller. 2013. Risks and Rewards of Crowdsourcing Marketplaces. In Handbook of Human Computation. Springer, New York, 377--392.Google Scholar
- Jesse Chandler and Danielle Shapiro. 2016. Conducting Clinical Research Using Crowdsourced Convenience Samples. Annual Review of Clinical Psychology 12 (2016), 35--81.Google ScholarCross Ref
- Anne Chao. 1987. Estimating the population size for capture-recapture data with unequal catchability. Biometrics 43, 4 (Dec. 1987), 783--791.Google ScholarCross Ref
- R. M. Cormack. 1964. Estimates of Survival From the Sighting of Marked Animals. Biometrics 51 (1964), 429--438.Google ScholarCross Ref
- Francisco Cribari-Neto and Achim Zeileis. 2010. Beta Regression in R. Journal of Statistical Software, Articles 34, 2 (2010), 1--24.Google ScholarCross Ref
- Damien L. Crone and Lisa A. Williams. 2017. Crowdsourcing participants for psychological research in Australia: A test of Microworkers. Australian Psychological Society 69 (2017), 39--47. Issue 1.Google ScholarCross Ref
- Joost de Winter, Miltos Kyriakidis, Dimitra Dodou, and Riender Happee. 2015. Using CrowdFlower to Study the Relationship between Self-reported Violations and Traffic Accidents. Procedia Manufacturing 3 (2015), 2518--2525.Google ScholarCross Ref
- Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G Ipeirotis, and Philippe Cudré-Mauroux. 2015. The Dynamics of Micro-Task Crowdsourcing: The Case of Amazon MTurk. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, ACM, New York, NY, USA, 238--247. Google ScholarDigital Library
- Karën Fort, Gilles Adda, and K. Bretonnel Cohen. 2011. Amazon Mechanical Turk: Gold Mine or Coal Mine? Computational Linguistics 37 (2011), 413 -- 420. Issue 2. Google ScholarDigital Library
- Ilka H. Gleibs. 2017. Are all research fields equal? Rethinking practice for the use of data from crowd-sourcing market places. Behavior Research Methods 49 (August 2017), 1333--1342. Issue 4.Google Scholar
- Panagiotis Ipeirotis. 2010. Demographics of Mechanical Turk. Technical Report CeDER-10-01. New York University.Google Scholar
- G. M. Jolly. 1965. Explicit Estimates from Capture-Recapture Data With Both Death and Immigration- Stochastic Model. Biometrika 52 (1965), 225--247.Google ScholarCross Ref
- Jeremy Kees, Christopher Berry, Scot Burton, and Kim Sheehan. 2017. An Analysis of Data Quality: Professional Panels, Student Subject Pools, and Amazon's Mechanical Turk. Journal of Advertising 46, 1 (2017), 141--155.Google ScholarCross Ref
- Frederick Charles Lincoln. 1930. Calculating waterfowl abundance on the basis of banding returns. US Dept. of Agriculture, Washington, DC.Google Scholar
- Jianguo Lu and Dingding Li. 2010. Estimating deep web data source size by capture-recapture method. Information Retrieval 13, 1 (February 2010), 70--95. Google ScholarDigital Library
- Winter Mason and Duncan J. Watts. 2009. Financial Incentives and the "Performance of Crowds". In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '09). ACM, New York, NY, USA, 77--85. Google ScholarDigital Library
- Joshua D. Miller, Michael Crowe, Brandon Weiss, Jessica L. Maples-Keller, and Donald R. Lynam. 2017. Using online, crowdsourcing platforms for data collection in personality disorder research: The example of Amazon's Mechanical Turk. Personality Disorders: Theory, Research, and Treatment 8, 1 (2017), 26--34.Google ScholarCross Ref
- Shirley Pledger, Kenneth H. Pollock, and James L. Norris. 2003. Open CaptureRecapture Models with Heterogeneity: I. Cormack-Jolly-Seber Model. Biometrics 59, 4 (Dec. 2003), 786--794.Google Scholar
- Shirley Pledger, Kenneth H. Pollock, and James L. Norris. 2010. Open CaptureRecapture Models with Heterogeneity: II. Jolly-Seber Model. Biometrics 66, 3 (September 2010), 883--890.Google Scholar
- Joel Ross, Lilly Irani, M. Six Silberman, Andrew Zaldivar, and Bill Tomlinson. 2010. Who Are the Crowdworkers?: Shifting Demographics in Mechanical Turk. In CHI '10 Extended Abstracts on Human Factors in Computing Systems (CHI EA '10). ACM, New York, NY, USA, 2863--2872. Google ScholarDigital Library
- Carl James Schwarz and A. Neil Arnason. 1996. A General Methodology for the Analysis of Capture-Recapture Experiments in Open Populations. Biometrics 52, 3 (1996), 860--873.Google ScholarCross Ref
- George Seber. 1982. The Estimation of Animal Abundance and Related Parameters. Charles Griffin, London, UK.Google Scholar
- G. A. F. Seber. 1964. A Note on the Multiple Recapture Census. Biometrika 52 (1964), 249--259.Google ScholarCross Ref
- Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD '08). ACM, New York, NY, USA, 614--622. Google ScholarDigital Library
- Michael Smithson and Jay Verkuilen. 2006. A better lemon squeezer? Maximumlikelihood regression with beta-distributed dependent variables. Psychological methods 11 (2006), 54--71. Issue 1.Google Scholar
- Rion Snow, Brendan O'Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and Fast-but is It Good?: Evaluating Non-expert Annotations for Natural Language Tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP '08). Association for Computational Linguistics, Stroudsburg, PA, USA, 254--263. Google ScholarDigital Library
- Vanessa V. Sochat, Ian W. Eisenberg, A. Zeynep Enkavi, Jamie Li, Patrick G. Bissett, and Russell A. Poldrack. 2016. The Experiment Factory: Standardizing Behavioral Experiments. Frontiers in Psychology 7 (2016), 610.Google ScholarCross Ref
- Neil Stewart, Christoph Ungemach, Adam J. L. Harris, Daniel M. Bartels, Ben R. Newell, Gabriele Paolaccik, and Jesse Chandler. 2015. The average laboratory samples a population of 7,300 Amazon Mechanical Turk workers. Judgment and Decision Making 10, 5 (2015), 479--491.Google ScholarCross Ref
- Keela S. Thomson and Daniel M. Oppenheimer. 2016. Investigating an alternate form of the cognitive reflection test. Judgment and Decision Making 11, 1 (2016), 99--113.Google ScholarCross Ref
- Beth Trushkowsky, Tim Kraska, Michael J. Franklin, and Purnamrita Sarkar. 2016. Answering Enumeration Queries with the Crowd. Commun. ACM 59, 1 (2016), 118--127 Google ScholarDigital Library
Index Terms
- Demographics and Dynamics of Mechanical Turk Workers
Recommendations
A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk
CHI '18: Proceedings of the 2018 CHI Conference on Human Factors in Computing SystemsA growing number of people are working as part of on-line crowd work. Crowd work is often thought to be low wage work. However, we know little about the wage distribution in practice and what causes low/high earnings in this setting. We recorded 2,676 ...
Worker Demographics and Earnings on Amazon Mechanical Turk: An Exploratory Analysis
CHI EA '19: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing SystemsPrior research reported that workers on Amazon Mechanical Turk (AMT) are underpaid, earning about $2/h. But the prior research did not investigate the difference in wage due to worker characteristics (e.g., country of residence). We present the first ...
Who are the crowdworkers?: shifting demographics in mechanical turk
CHI EA '10: CHI '10 Extended Abstracts on Human Factors in Computing SystemsAmazon Mechanical Turk (MTurk) is a crowdsourcing system in which tasks are distributed to a population of thousands of anonymous workers for completion. This system is increasingly popular with researchers and developers. Here we extend previous ...
Comments