skip to main content
10.1145/2141512.2141556acmconferencesArticle/Chapter ViewAbstractPublication PagescscwConference Proceedingsconference-collections
poster

Diversity within the crowd

Published: 11 February 2012 Publication History

Abstract

Though crowdsourcing holds great promise, many struggle with framing tasks and determining which members of the crowd should be recruited to obtain reliable output. In some cases, expert knowledge is desired but, given the time and cost constraints of the problem, may not be available. In this case, it would be beneficial to augment the expert input that is available with input from members of the general population. We believe that reduced reliance on experts will in some cases lead to acceptable performance while reducing cost and latency. In this work, we show that we are able to approach the performance of an expert group for an image labeling task, while reducing our reliance on experts by incorporating non-expert responses.

References

[1]
Get Better Results with Less Effort with Mechanical Turk Masters! www.mechanicalturk.typepad.com, June 2011.
[2]
J. Heer and M. Bostock. Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Virtualization Design. CHI, ACM, 2010.
[3]
K. Heimerl and B. Hartmann. Communitysourcing: Engaging Local Crowds to Perform Expert Work Via Physical Kiosks. In Submission to CHI 2012.
[4]
L. Hong and S. E. Page. Problem Solving by Heterogeneous Agents. volume, 97, pages 123--163. Journal of Economic Theory, 2000.
[5]
D. Shahaf and E. Horvitz. Generalized Task Markets for Human and Machine Computation. AAAI, July 2010.
[6]
A. Sorokin and D. Forsyth. Utility Data Annotation with Amazon Mechanical Turk. Computer Vision and Pattern Recognition Workshops, IEEE, June 2008.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CSCW '12: Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work Companion
February 2012
388 pages
ISBN:9781450310512
DOI:10.1145/2141512
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 February 2012

Check for updates

Author Tags

  1. crowdsourcing
  2. diversity
  3. image labeling
  4. mechanical turk

Qualifiers

  • Poster

Conference

CSCW '12
Sponsor:
CSCW '12: Computer Supported Cooperative Work
February 11 - 15, 2012
Washington, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

Upcoming Conference

CSCW '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)2
  • Downloads (Last 6 weeks)0
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media