skip to main content
10.1145/3126973.3126994acmotherconferencesArticle/Chapter ViewAbstractPublication PagesiccseConference Proceedingsconference-collections
research-article

Empirical Study on Assessment Algorithms with Confidence in Crowdsourcing

Published: 06 July 2017 Publication History

Abstract

Evaluating the quality of workers is very important in crowdsourcing system and impactful methods are required in order to obtain the most appropriate quality. Previous work have introduced confidence intervals to estimate the quality of workers. However, we have found the size of the confidence interval is wide through analysis of experimental results, which leads to inaccurate worker error rates. In this paper, we propose an optimized algorithm of confidence interval to reduce the size of the confidence interval as narrow as possible and to estimate the quality of workers more precise. We verify our algorithm using the simulated data from our own crowdsourcing platform under realistic settings.

References

[1]
Guoliang Li, Jianan Wang, Yudian Zheng, and Michael Franklin. Crowdsourced data management: A survey. IEEE Transactions on Knowledge and Data Engineering, 28(9):2296--2319, 2016.
[2]
Xuan Liu, Meiyu Lu, Beng Chin Ooi, Yanyan Shen, Sai Wu, and Meihui Zhang. Cdas: a crowdsourcing data analytics system. Proceedings of the Vldb Endowment, 5(10):1040--1051, 2012.
[3]
Xuan Liu, Meiyu Lu, Beng Chin Ooi, Yanyan Shen, Sai Wu, and Meihui Zhang. Cdas: a crowdsourcing data analytics system. Proceedings of the Vldb Endowment, 5(10):1040--1051, 2012.
[4]
Yudian Zheng, Reynold Cheng, Silviu Maniu, and Luyi Mo. On optimality of jury selection in crowdsourcing. Intl.conf.on Extending Database Technology Brussels Belgium, 2015.
[5]
Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Anna Jerebko, Charles Florin, Gerardo Hermosillo Valadez, Luca Bogoni, and Linda Moy. Supervised learning from multiple experts:whom to trust when everyone lies a bit. pages 889--896, 2009.
[6]
Matteo Venanzi, John Guiver, Gabriella Kazai, Pushmeet Kohli, and Milad Shokouhi. Community-based bayesian aggregation models for crowdsourcing. pages 155--164, 2014.
[7]
Jacob Whitehill, Paul Ruvolo, Tingfan Wu, Jacob Bergsma, and Javier Movellan. Whose vote should count more: optimal integration of labels from labelers of unknown expertise. pages 2035--2043, 2009.
[8]
Manas Joglekar, Hector Garciamolina, and Aditya Parameswaran. Comprehensive and reliable crowd assessment algorithms. pages 195--206, 2014.
[9]
Stephen Guo, Aditya Parameswaran, and Hector Garcia-Molina. So who won?: dynamic max discovery with the crowd. pages 385--396, 2012.
[10]
Caleb Chen Cao, Jieying She, Yongxin Tong, and Lei Chen. Whom to ask?: jury selection for decision making tasks on micro-blog services. Proceedings of the Vldb Endowment, 5(11):1495--1506, 2012.
[11]
Leyla Kazemi, Cyrus Shahabi, and Lei Chen. Geotrucrowd:trustworthy query answering with spatial crowdsourcing. pages 314--323, 2013.
[12]
A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society, 28(1):20--28, 1979.
[13]
Panagiotis G. Ipeirotis, Foster Provost, and Jing Wang. Quality management on amazon mechanical turk. pages 64--67, 2010.
[14]
Larry Wasserman. All of statistics. Springer Texts in Statistics, 97(1-3):xx,442, 2004.
[15]
George Casella and Roger L. Berger. Statistical inference. Technometrics, 33(4):xii,328, 2001.
[16]
Adam Marcus, Eugene Wu, David Karger, Samuel Madden, and Robert Miller. Human-powered sorts and joins. Proceedings of the Vldb Endowment, 5(1):13--24, 2011.
[17]
Geoffrey J. Mclachlan and Thriyambakam Krishnan. The em algorithm and extensions (wiley series in probability and statistics). Journal of Classification, 15(1):154--156, 2008.
[18]
Aditya G Parameswaran, Hector Garciamolina, Hyunjung Park, Neoklis Polyzotis, Aditya Ramesh, and Jennifer Widom. Crowd-screen: algorithms for filtering data with humans. Sigmod, pages 361--372, 2012.
[19]
Aditya Parameswaran, Anish Das Sarma, Hector Garcia-Molina, Neoklis Polyzotis, and Jennifer Widom. Human-assisted graph search: it's okay to ask questions. Proceedings of the Vldb Endowment, 4(5):267--278, 2011.
[20]
Vikas C Raykar and Shipeng Yu. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. Journal of Machine Learning Research, 13(1):491--518, 2012.
[21]
Flvio Ribeiro, Dinei Florencio, and Vtor Nascimento. Crowdsourcing subjective image quality evaluation. pages 3097--3100, 2011.
[22]
Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. pages 614--622, 2008.
[23]
Yuandong Tian and Jun Zhu. Learning from crowds in the presence of schools of thought. pages 226--234, 2012.
[24]
Manas Joglekar, Hector Garciamolina, and Aditya Parameswaran. Evaluating the crowd with confidence. Computer Science, pages 686--694, 2013.
[25]
Edwin B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209--212, 1927.

Cited By

View all
  • (2020)A Smartphone Crowdsensing System Enabling Environmental Crowdsourcing for Municipality Resource Allocation with LSTM Stochastic PredictionSensors10.3390/s2014396620:14(3966)Online publication date: 16-Jul-2020

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
ICCSE'17: Proceedings of the 2nd International Conference on Crowd Science and Engineering
July 2017
158 pages
ISBN:9781450353755
DOI:10.1145/3126973
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 July 2017

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Crowdsourcing
  2. confidence interval
  3. quality of worker

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

ICCSE'17

Acceptance Rates

ICCSE'17 Paper Acceptance Rate 24 of 66 submissions, 36%;
Overall Acceptance Rate 92 of 247 submissions, 37%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)1
  • Downloads (Last 6 weeks)0
Reflects downloads up to 23 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2020)A Smartphone Crowdsensing System Enabling Environmental Crowdsourcing for Municipality Resource Allocation with LSTM Stochastic PredictionSensors10.3390/s2014396620:14(3966)Online publication date: 16-Jul-2020

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media