skip to main content
10.1145/2723372.2749430acmconferencesArticle/Chapter ViewAbstractPublication PagesmodConference Proceedingsconference-collections
research-article

QASCA: A Quality-Aware Task Assignment System for Crowdsourcing Applications

Published: 27 May 2015 Publication History

Abstract

A crowdsourcing system, such as the Amazon Mechanical Turk (AMT), provides a platform for a large number of questions to be answered by Internet workers. Such systems have been shown to be useful to solve problems that are difficult for computers, including entity resolution, sentiment analysis, and image recognition. In this paper, we investigate the online task assignment problem: Given a pool of n questions, which of the k questions should be assigned to a worker? A poor assignment may not only waste time and money, but may also hurt the quality of a crowdsourcing application that depends on the workers' answers. We propose to consider quality measures (also known as evaluation metrics) that are relevant to an application during the task assignment process. Particularly, we explore how Accuracy and F-score, two widely-used evaluation metrics for crowdsourcing applications, can facilitate task assignment. Since these two metrics assume that the ground truth of a question is known, we study their variants that make use of the probability distributions derived from workers' answers. We further investigate online assignment strategies, which enables optimal task assignments. Since these algorithms are expensive, we propose solutions that attain high quality in linear time. We develop a system called the Quality-Aware Task Assignment System for Crowdsourcing Applications (QASCA) on top of AMT. We evaluate our approaches on five real crowdsourcing applications. We find that QASCA is efficient, and attains better result quality (of more than 8% improvement) compared with existing methods.

References

[1]
A.P.Dawid and A.M.Skene. Maximum likelihood estimation of observer error-rates using em algorithm. Appl.Statist., 28(1):20--28, 1979.
[2]
M. Blum, R. W. Floyd, V. R. Pratt, R. L. Rivest, and R. E. Time bounds for selection. Journal of Computer and System Sciences, 7(4):448--461, 1973.
[3]
R. Boim, O. Greenshpan, T. Milo, S. Novgorodov, N. Polyzotis, and W. C. Tan. Asking the right questions in crowd data sourcing. In ICDE, 2012.
[4]
C.G. Small. Expansions and asymptotics for statistics. CRC Press, 2010.
[5]
X. Chen, P. N. Bennett, K. Collins-Thompson, and E. Horvitz. Pairwise ranking aggregation in a crowdsourced setting. In WSDM, pages 193--202, 2013.
[6]
X. Chen, Q. Lin, and D. Zhou. Optimistic knowledge gradient policy for optimal budget allocation in crowdsourcing. In ICML, pages 64--72, 2013.
[7]
J. Costa, C. Silva, M. Antunes, and B. Ribeiro. On using crowdsourcing and active learning to improve classification performance. In ISDA, 2011.
[8]
P. Dai, C. H. Lin, Mausam, and D. S. Weld. Pomdp-based control of workflows for crowdsourcing. Artif. Intell., 202:52--85, 2013.
[9]
S. B. Davidson, S. Khanna, T. Milo, and S. Roy. Using the crowd for top-k and group-by queries. In ICDT, pages 225--236, 2013.
[10]
A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the em algorithm. J.R. Statist. Soc. B, 30(1):1--38, 1977.
[11]
D. Deng, G. Li, S. Hao, J. Wang, and J. Feng. Massjoin: A mapreduce-based method for scalable string similarity joins. In ICDE, pages 340--351, 2014.
[12]
W. Dinkelbach. On nonlinear fractional programming. Management Science, 13(7):492--498, March,1967.
[13]
P. Efraimidis and P. G. Spirakis. Weighted random sampling with a reservoir. Inf. Process. Lett., 97(5):181--185, 2006.
[14]
M. J. Franklin, D. Kossmann, T. Kraska, S. Ramesh, and R. Xin. Crowddb: answering queries with crowdsourcing. In SIGMOD Conference, 2011.
[15]
J. Gao, X. Liu, B. C. Ooi, H. Wang, and G. Chen. An online cost sensitive decision-making method in crowdsourcing systems. In SIGMOD, 2013.
[16]
S. Guo, A. G. Parameswaran, and H. Garcia-Molina. So who won?: dynamic max discovery with the crowd. In SIGMOD Conference, pages 385--396, 2012.
[17]
K. Hara, V. Le, and J. Froehlich. Combining crowdsourcing and google street view to identify street-level accessibility problems. In CHI, 2013.
[18]
C.-J. Ho, S. Jabbari, and J. W. Vaughan. Adaptive task assignment for crowdsourced classification. In ICML (1), pages 534--542, 2013.
[19]
C.-J. Ho and J. W. Vaughan. Online task assignment in crowdsourcing markets. In AAAI, 2012.
[20]
J. J. Horton and L. B. Chilton. The labor economics of paid crowdsourcing. In ACM Conference on Electronic Commerce, pages 209--218, 2010.
[21]
N. Q. V. Hung, N. T. Tam, L. N. Tran, and K. Aberer. An evaluation of aggregation techniques in crowdsourcing. In WISE, pages 1--15. Springer, 2013.
[22]
P. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In SIGKDD workshop, pages 64--67, 2010.
[23]
P. G. Ipeirotis. Analyzing the amazon mechanical turk marketplace. ACM Crossroads, 17(2):16--21, 2010.
[24]
M. Jansche. A maximum expected utility framework for binary sequence labeling. In ACL, 2007.
[25]
S. R. Jeffery, M. J. Franklin, and A. Y. Halevy. Pay-as-you-go user feedback for dataspace systems. In SIGMOD, pages 847--860, 2008.
[26]
D. R. Karger, S. Oh, and D. Shah. Iterative learning for reliable crowdsourcing systems. In NIPS, pages 1953--1961, 2011.
[27]
A. H. Laender, M. A. Gonçalves, R. G. Cota, A. A. Ferreira, R. L. T. Santos, and A. J. Silva. Keeping a digital library clean: New solutions to old problems. In DocEng, pages 257--262. ACM, 2008.
[28]
D. D. Lewis. Evaluating and optimizing autonomous text classification systems. In SIGIR, pages 246--254, 1995.
[29]
X. Li, X. L. Dong, K. Lyons, W. Meng, and D. Srivastava. Truth finding on the deep web: Is the problem solved? PVLDB, 6(2):97--108, 2012.
[30]
X. Liu, M. Lu, B. C. Ooi, Y. Shen, S. Wu, and M. Zhang. Cdas: A crowdsourcing data analytics system. PVLDB, 5(10):1040--1051, 2012.
[31]
C. D. Manning, P. Raghavan, and H. Schütze. Introduction to information retrieval. Cambridge University Press, 2008.
[32]
C. D. Manning and H. Schütze. Foundations of statistical natural language processing. MIT Press, 2001.
[33]
A. Marcus, D. R. Karger, S. Madden, R. Miller, and S. Oh. Counting with the crowd. PVLDB, 6(2):109--120, 2012.
[34]
A. Marcus, E. Wu, D. R. Karger, S. Madden, and R. C. Miller. Human-powered sorts and joins. PVLDB, 5(1):13--24, 2011.
[35]
A. Marcus, E. Wu, S. Madden, and R. C. Miller. Crowdsourced databases: Query processing with people. In CIDR, pages 211--214, 2011.
[36]
D. Menestrina, S. E. Whang, and H. Garcia-Molina. Evaluating entity resolution results. PVLDB, 3(1--2):208--219, 2010.
[37]
K. Mo, E. Zhong, and Q. Yang. Cross-task crowdsourcing. In KDD, 2013.
[38]
L. Mo, R. Cheng, B. Kao, X. S. Yang, C. Ren, S. Lei, D. W. Cheung, and E. Lo. Optimizing plurality for human intelligence tasks. In CIKM, 2013.
[39]
S. Nowozin. Optimal decisions from probabilistic models: the intersection-over-union case. In CVPR, 2014.
[40]
A. G. Parameswaran, H. Garcia-Molina, H. Park, N. Polyzotis, A. Ramesh, and J. Widom. Crowdscreen: algorithms for filtering data with humans. In SIGMOD Conference, pages 361--372, 2012.
[41]
H. Park, R. Pang, A. G. Parameswaran, H. Garcia-Molina, N. Polyzotis, and J. Widom. Deco: A system for declarative crowdsourcing. PVLDB, 5(12), 2012.
[42]
R. Pochampally, A. D. Sarma, X. L. Dong, A. Meliou, and D. Srivastava. Fusing data with correlations. In SIGMOD, pages 433--444, 2014.
[43]
V. C. Raykar and S. Yu. Ranking annotators for crowdsourced labeling tasks. In NIPS, pages 1809--1817, 2011.
[44]
V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy. Learning from crowds. Journal of Machine Learning Research, 2010.
[45]
S. H. Rice. A stochastic version of the price equation reveals the interplay of deterministic and stochastic processes in evolution. BMC evolutionary biology, 8:262, 2008.
[46]
A. B. Sayeed, T. J. Meyer, H. C. Nguyen, O. Buzek, and A. Weinberg. Crowdsourcing the evaluation of a domain-adapted named entity recognition system. In HLT-NAACL, pages 345--348, 2010.
[47]
V. S. Sheng, F. J. Provost, and P. G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In KDD, 2008.
[48]
A. Sheshadri and M. Lease. SQUARE: A Benchmark for Research on Computing Crowd Consensus. In Proceedings of the 1st AAAI Conference on Human Computation (HCOMP), pages 156--164, 2013.
[49]
J. R. Talburt. Entity Resolution and Information Quality. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 2010.
[50]
B. Trushkowsky, T. Kraska, M. J. Franklin, and P. Sarkar. Crowdsourced enumeration queries. In ICDE, pages 673--684, 2013.
[51]
C. Van Rijsbergen. Information retrieval. Butterworths, 1979.
[52]
M. Venanzi, J. Guiver, G. Kazai, P. Kohli, and M. Shokouhi. Community-based bayesian aggregation models for crowdsourcing. In WWW, 2014.
[53]
P. Venetis and H. Garcia-Molina. Quality control for comparison microtasks. In Proceedings of the First International Workshop on Crowdsourcing and Data Mining, pages 15--21. ACM, 2012.
[54]
P. Venetis, H. Garcia-Molina, K. Huang, and N. Polyzotis. Max algorithms in crowdsourcing environments. In WWW, pages 989--998, 2012.
[55]
R. Vernica, M. J. Carey, and C. Li. Efficient parallel set-similarity joins using mapreduce. In SIGMOD, pages 495--506. ACM, 2010.
[56]
J. Wang, T. Kraska, M. J. Franklin, and J. Feng. CrowdER: crowdsourcing entity resolution. PVLDB, 5(11):1483--1494, 2012.
[57]
J. Wang, S. Krishnan, M. J. Franklin, K. Goldberg, T. Kraska, and T. Milo. A sample-and-clean framework for fast and accurate query processing on dirty data. In SIGMOD, pages 469--480. ACM, 2014.
[58]
J. Wang, G. Li, and J. Feng. Can we beat the prefix filtering?: an adaptive framework for similarity join and search. In SIGMOD, pages 85--96, 2012.
[59]
J. Wang, G. Li, T. Kraska, M. J. Franklin, and J. Feng. Leveraging transitive relations for crowdsourced joins. In SIGMOD, 2013.
[60]
S. E. Whang, P. Lofgren, and H. Garcia-Molina. Question selection for crowd entity resolution. PVLDB, 6(6):349--360, 2013.
[61]
J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. R. Movellan. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In NIPS, pages 2035--2043, 2009.
[62]
C. J. Zhang, L. Chen, H. V. Jagadish, and C. C. Cao. Reducing uncertainty of schema matching via crowdsourcing. PVLDB, 6(9):757--768, 2013.
[63]
B. Zhao, B. I. Rubinstein, J. Gemmell, and J. Han. A bayesian approach to discovering truth from conflicting sources for data integration. PVLDB, 2012.

Cited By

View all
  • (2025)Software Crowdsourcing Allocation Algorithm Based on Task PriorityAlgorithms and Architectures for Parallel Processing10.1007/978-981-96-1528-5_6(85-94)Online publication date: 15-Feb-2025
  • (2024)“I Prefer Regular Visitors to Answer My Questions”: Users’ Desired Experiential Background of Contributors for Location-based Crowdsourcing PlatformProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642520(1-18)Online publication date: 11-May-2024
  • (2024)Quasi Group Role Assignment With Agent Satisfaction in Self-Service Spatiotemporal CrowdsourcingIEEE Transactions on Computational Social Systems10.1109/TCSS.2024.341795911:5(7002-7019)Online publication date: Oct-2024
  • Show More Cited By

Index Terms

  1. QASCA: A Quality-Aware Task Assignment System for Crowdsourcing Applications

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGMOD '15: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data
    May 2015
    2110 pages
    ISBN:9781450327589
    DOI:10.1145/2723372
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 27 May 2015

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. crowdsourcing
    2. online task assignment
    3. quality control

    Qualifiers

    • Research-article

    Funding Sources

    • Huawei
    • "NExT Research Center"
    • NSFC project
    • Tencent
    • 973 Program of China
    • SAP
    • HKU
    • Chinese Special Project of Science and Technology

    Conference

    SIGMOD/PODS'15
    Sponsor:
    SIGMOD/PODS'15: International Conference on Management of Data
    May 31 - June 4, 2015
    Victoria, Melbourne, Australia

    Acceptance Rates

    SIGMOD '15 Paper Acceptance Rate 106 of 415 submissions, 26%;
    Overall Acceptance Rate 785 of 4,003 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)43
    • Downloads (Last 6 weeks)5
    Reflects downloads up to 19 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Software Crowdsourcing Allocation Algorithm Based on Task PriorityAlgorithms and Architectures for Parallel Processing10.1007/978-981-96-1528-5_6(85-94)Online publication date: 15-Feb-2025
    • (2024)“I Prefer Regular Visitors to Answer My Questions”: Users’ Desired Experiential Background of Contributors for Location-based Crowdsourcing PlatformProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642520(1-18)Online publication date: 11-May-2024
    • (2024)Quasi Group Role Assignment With Agent Satisfaction in Self-Service Spatiotemporal CrowdsourcingIEEE Transactions on Computational Social Systems10.1109/TCSS.2024.341795911:5(7002-7019)Online publication date: Oct-2024
    • (2024)BClean: A Bayesian Data Cleaning System2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00263(3407-3420)Online publication date: 13-May-2024
    • (2024)HITSnDIFFs: From Truth Discovery to Ability Discovery by Recovering Matrices with the Consecutive Ones Property2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00025(235-248)Online publication date: 13-May-2024
    • (2024)A Fuzzy Ranking Collaborative Multi-Tasks Data Collection Scheme in Ubiquitous EnvironmentsIEEE Access10.1109/ACCESS.2024.344018512(130777-130798)Online publication date: 2024
    • (2024)A Dual-Embedding Based Reinforcement Learning Scheme for Task Assignment Problem in Spatial CrowdsourcingWorld Wide Web10.1007/s11280-024-01325-928:1Online publication date: 27-Dec-2024
    • (2024)Call Centre Optimization Based on Personalized Requests DistributionOptimization, Learning Algorithms and Applications10.1007/978-3-031-53025-8_10(135-147)Online publication date: 1-Feb-2024
    • (2023)Optimal budget allocation for crowdsourcing labels for graphsProceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence10.5555/3625834.3625943(1154-1163)Online publication date: 31-Jul-2023
    • (2023)Online Coalitional Skill FormationProceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems10.5555/3545946.3598676(494-503)Online publication date: 30-May-2023
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media