research-article

Experiences surveying the crowd: reflections on methods, participation, and reliability

Authors:

Catherine C. Marshall,

Frank M. ShipmanAuthors Info & Claims

WebSci '13: Proceedings of the 5th Annual ACM Web Science Conference

Pages 234 - 243

https://doi.org/10.1145/2464464.2464485

Published: 02 May 2013 Publication History

Abstract

Crowdsourcing services such as Amazon's Mechanical Turk (MTurk) provide new venues for recruiting participants and conducting studies; hundreds of surveys may be offered to workers at any given time. We reflect on the results of six related studies we performed on MTurk over a two year period. The studies used a combination of open-ended questions and structured hypothetical statements about story-like scenarios to engage the efforts of 1252 participants. We describe the method used in the studies and reflect on what we have learned about identified best practices. We analyze the aggregated data to profile the types of Turkers who take surveys and examine how the characteristics of the surveys may influence data reliability. The results point to the value of participant engagement, identify potential changes in MTurk as a study venue, and highlight how communication among Turkers influences the data that researchers collect.

References

[1]

Acquisti, A. and Grossklags, J. Privacy Attitudes and Privacy Behavior, in J. Camp and S. Lewis (Eds.) The Economics of Information Security, Kluwer, 165--178.

[2]

Alonso, O. Implementing crowdsourcing-based relevance experimentation: An industrial perspective, Information Retrieval Journal (2012), in press.

Digital Library

[3]

Antin J. & Shaw, A. Social desirability bias and self-reports of motivation: a study of amazon mechanical turk in the US and India. Proc. CHI '12. 2925--2934.

Digital Library

[4]

Boyle, J. The Public Domain: Enclosing the Commons of the Mind, Yale University Press, New Haven, 2008.

[5]

Dow, S., Kulkarni, A., Klemmer, S., and Hartmann, B. 2012. Shepherding the crowd yields better work. Proc. of CSCW '12. 1013--1022.

Digital Library

[6]

Downs, J., Holbrook, M., Sheng, S., and Cranor, L. Are your participants gaming the system?: Screening Mechanical Turk workers. Proc. CHI'10. 2399--2402.

Digital Library

[7]

Dwork, C. Differential privacy. ICALP, 2006, 1--12.

Digital Library

[8]

Eickhoff, C. and de Vries, A. P. How Crowdsourceable is Your Task? Proc. Workshop on Crowdsourcing for Search and Data Mining, 2011.

[9]

Fetterman, D. Ethnography. Sage, 1989.

[10]

Greengard, S. Digitally Possessed. Communications of the ACM, 55 (5), 2012, 14--16.

Digital Library

[11]

Herzog, W. & Stutzman, F. The Case for Online Obscurity, California Law Review, Vol. 101.

[12]

Hill, B., Monroy-Hernandez, A., and Olson, K. 2010. Responses to Remixing on a Social Media Website. Proc. AAAI Conf. on Weblogs and Social Media. 74--81.

[13]

Ipeirotis, P. Demographics of Mechanical Turk. NYU Tech Report, 2010.

[14]

Ipeirotis, P., Provost, F. and Wang, J. Quality Management on Amazon Mechanical Turk. KDD-HCOMP, 2010.

Digital Library

[15]

Jakobsson, M. Experimenting on Mechanical Turk: 5 How Tos. ITWorld, September 3, 2009.

[16]

Kiesler, S., and Sproull, L. Response Effects in the Electronic Survey. Public Opin Q 50 (3): 402--413.

[17]

Kittur, A., Chi, E., and Suh, B. Crowdsourcing User Studies with Mechanical Turk. Proc. CHI'08. 453--456.

Digital Library

[18]

Lessig, L. Remix, Penguin, New York, 2008.

[19]

MacCormick and Summers, (eds.) Interpreting Precedents, Ashgate/Dartmouth, 1997, pp. 528--9.

[20]

Marshall, C. C., and Shipman, F. M. Social media ownership: Using Twitter as a window onto current attitudes and beliefs. Proc. CHI'11, ACM, 1081--1090.

Digital Library

[21]

Marshall, C. C., and Shipman, F. M. The ownership and reuse of visual media. Proc. JCDL'11. ACM, 157--166.

Digital Library

[22]

Marshall, C. C., and Shipman, F. M. On the institutional archiving of social media. Proc. JCDL'12. ACM, 1--10.

Digital Library

[23]

Marshall, C. C. and Shipman, F. M. Saving, reusing, and remixing web video: using attitudes and practices to reveal social norms. Proc. WWW'13. ACM.

Digital Library

[24]

Mason, W. and Suri, S. Conducting Behavioral Research on Amazon's Mechanical Turk (October 12, 2010). Behavior Research Methods, Forthcoming.

[25]

Mason, W. & Watts, D. Financial incentives and the performance of crowds. Proc. SIGKDD 2009 workshop on human computation pp. 77--85.

Digital Library

[26]

Odom, W. Sellen, A., Harper, R., and Thereska, E. Lost in Translation: Understanding the Possession of Digital Things in the Cloud. Proc. CHI'12. 781--790.

Digital Library

[27]

Palen, L. and Dourish, P. 2003. Unpacking "privacy" for a networked world. Proc CHI 2003, 129--136.

Digital Library

[28]

Rissland and Ashley, "Hypotheticals as Heuristic Device." Proceedings of Strategic Computing Natural Language Workshop, Marina del Rey, California, May 1-2, 1986, p. 168.

Digital Library

[29]

Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. Who are the crowdworkers?: shifting demographics in mechanical turk. Proc. CHI EA '10. 2863--2872.

Digital Library

[30]

Rzeszotarski, J. and Kittur, A. 2011. Instrumenting the crowd: using implicit behavioral measures to predict task performance. Proc. UIST '11. 13--22.

Digital Library

[31]

Schmidt, L., Crowdsourcing for Human Subjects Research. Proc. CrowdConf 2010, SF, CA.

[32]

Schnoebelen T. and Kuperman, V. Using Amazon Mechanical Turk for linguistic research, PSIHOLOGIJA 43, 4, 441--464.

[33]

Shipman, F. M. and Marshall, C. C. Are user-contributed reviews community property? exploring the beliefs and practices of reviewers, Proc. WebSci, 2013.

Digital Library

[34]

Silberman, M., Irani, L., and Ross, J. Ethics and Tactics of Professional Crowdwork. XRDS 17, 2, 39--43

Digital Library

[35]

Strauss, A. and Corbin, J. Basics of Qualitative Research, Sage Publications, 1998.

[36]

United States Census Bureau. Computer and Internet Use in the United States: 2010. http://www.census.gov/hhes/computer/publications/2010.html. Retrieved 29 January 2013.

Cited By

Yang HSundar S(2024)Machine heuristic: concept explication and development of a measurement scaleJournal of Computer-Mediated Communication10.1093/jcmc/zmae01929:6Online publication date: 25-Sep-2024
https://doi.org/10.1093/jcmc/zmae019
He GKuiper LGadiraju U(2023)Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI SystemsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581025(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581025
Wijenayake SHettiachchi DGoncalves J(2023)Combining Worker Factors for Heterogeneous Crowd Task AssignmentProceedings of the ACM Web Conference 202310.1145/3543507.3583190(3794-3805)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583190
Show More Cited By

Index Terms

Experiences surveying the crowd: reflections on methods, participation, and reliability
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Demographics and Dynamics of Mechanical Turk Workers
WSDM '18: Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining

We present an analysis of the population dynamics and demographics of Amazon Mechanical Turk workers based on the results of the survey that we conducted over a period of 28 months, with more than 85K responses from 40K unique participants. The ...
Who are the crowdworkers?: shifting demographics in mechanical turk
CHI EA '10: CHI '10 Extended Abstracts on Human Factors in Computing Systems

Amazon Mechanical Turk (MTurk) is a crowdsourcing system in which tasks are distributed to a population of thousands of anonymous workers for completion. This system is increasingly popular with researchers and developers. Here we extend previous ...
Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments

The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WebSci '13: Proceedings of the 5th Annual ACM Web Science Conference

May 2013

481 pages

ISBN:9781450318891

DOI:10.1145/2464464

Conference Chairs:
Hugh Davis
University of Southampton
,
Harry Halpin
World Wide Web Consortium
,
Alex Pentland,
Program Chairs:
Mark Bernstein,
Lada Adamic,
Harith Alani,
Alexandre Monnin,
Richard Rogers

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

WebSci '13

Sponsor:

SIGWEB

WebSci '13: Web Science 2013

May 2 - 4, 2013

Paris, France

Acceptance Rates

Overall Acceptance Rate 245 of 933 submissions, 26%

Upcoming Conference

Websci '25

Sponsor:
sigweb

17th ACM Web Science Conference

May 20 - 24, 2025

New Brunswick , NJ , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

49
Total Citations
View Citations
557
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)3

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang HSundar S(2024)Machine heuristic: concept explication and development of a measurement scaleJournal of Computer-Mediated Communication10.1093/jcmc/zmae01929:6Online publication date: 25-Sep-2024
https://doi.org/10.1093/jcmc/zmae019
He GKuiper LGadiraju U(2023)Knowing About Knowing: An Illusion of Human Competence Can Hinder Appropriate Reliance on AI SystemsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581025(1-18)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581025
Wijenayake SHettiachchi DGoncalves J(2023)Combining Worker Factors for Heterogeneous Crowd Task AssignmentProceedings of the ACM Web Conference 202310.1145/3543507.3583190(3794-3805)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583190
Hatmanto ESari M(2023)Aligning Theory and Practice: Leveraging Chat GPT for Effective English Language Teaching and LearningE3S Web of Conferences10.1051/e3sconf/202344005001440(05001)Online publication date: 1-Nov-2023
https://doi.org/10.1051/e3sconf/202344005001
Pradhan VSchaekermann MLease M(2022)In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd WorkersFrontiers in Artificial Intelligence10.3389/frai.2022.8281875Online publication date: 18-May-2022
https://doi.org/10.3389/frai.2022.828187
Pandey RPurohit HCastillo CShalin V(2022)Modeling and mitigating human annotation errors to design efficient stream processing systems with human-in-the-loop machine learningInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2022.102772160:COnline publication date: 1-Apr-2022
https://dl.acm.org/doi/10.1016/j.ijhcs.2022.102772
Gavenas JHieronymi PMaoz U(2022)Diverging lay intuitions about concepts related to free will in arbitrary and deliberate decisionsConsciousness and Cognition10.1016/j.concog.2022.103434106(103434)Online publication date: Nov-2022
https://doi.org/10.1016/j.concog.2022.103434
Ramírez JSayin BBaez MCasati FCernuzzi LBenatallah BDemartini G(2021)On the State of Reporting in Crowdsourcing Experiments and a Checklist to Aid Current PracticesProceedings of the ACM on Human-Computer Interaction10.1145/34795315:CSCW2(1-34)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3479531
Osterbrink LAlpar P(2021)Silence of crowdworkers—reasons and implications for work conditions and qualityInternational Studies of Management & Organization10.1080/00208825.2021.192731151:2(136-161)Online publication date: 27-Jun-2021
https://doi.org/10.1080/00208825.2021.1927311
Braylan ALease M(2020)Modeling and Aggregation of Complex Annotations via Annotation DistancesProceedings of The Web Conference 202010.1145/3366423.3380250(1807-1818)Online publication date: 20-Apr-2020
https://dl.acm.org/doi/10.1145/3366423.3380250
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten