skip to main content
10.1145/2983323.2983716acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
research-article

Active Content-Based Crowdsourcing Task Selection

Published: 24 October 2016 Publication History

Abstract

Crowdsourcing has long established itself as a viable alternative to corpus annotation by domain experts for tasks such as document relevance assessment. The crowdsourcing process traditionally relies on high degrees of label redundancy in order to mitigate the detrimental effects of individually noisy worker submissions. Such redundancy comes at the cost of increased label volume, and, subsequently, monetary requirements. In practice, especially as the size of datasets increases, this is undesirable. In this paper, we focus on an alternate method that exploits document information instead, to infer relevance labels for unjudged documents. We present an active learning scheme for document selection that aims at maximising the overall relevance label prediction accuracy, for a given budget of available relevance judgements by exploiting system-wide estimates of label variance and mutual information.
Our experiments are based on TREC 2011 Crowdsourcing Track data and show that our method is able to achieve state-of-the-art performance while requiring 17% - 25% less budget.

References

[1]
O. Alonso, D. E. Rose, and B. Stewart. Crowdsourcing for relevance evaluation. SIGIR Forum, 42(2):9--15, Nov. 2008.
[2]
J. A. Aslam, V. Pavlu, and E. Yilmaz. A statistical method for system evaluation using incomplete judgments. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '06, pages 541--548, New York, NY, USA, 2006. ACM.
[3]
C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '04, pages 25--32, New York, NY, USA, 2004. ACM.
[4]
J. Callan, M. Hoy, C. Yoo, and L. Zhao. Clueweb09 data set, 2009.
[5]
B. Carterette, J. Allan, and R. Sitaraman. Minimal test collections for retrieval evaluation. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '06, pages 268--275, New York, NY, USA, 2006. ACM.
[6]
C. Cleverdon. Readings in information retrieval. In K. Sparck Jones and P. Willett, editors, Readings in Information Retrieval, chapter The Cran eld Tests on Index Language Devices, pages 47--59. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1997.
[7]
M. Davtyan, C. Eickho, and T. Hofmann. Exploiting document content for efficient aggregation of crowdsourcing votes. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 783--790. ACM, 2015.
[8]
D. E. Difallah, M. Catasta, G. Demartini, and P. Cudré-Mauroux. Scaling-up the crowd: Micro-task pricing schemes for worker retention and latency improvement. In Second AAAI Conference on Human Computation and Crowdsourcing, 2014.
[9]
D. E. Difallah, G. Demartini, and P. Cudré-Mauroux. Pick-a-crowd: tell me what you like, and i'll tell you what to do. In Proceedings of the 22nd international conference on World Wide Web, pages 367--374. International World Wide Web Conferences Steering Committee, 2013.
[10]
C. Eickho and A. P. de Vries. Increasing cheat robustness of crowdsourcing tasks. Information retrieval, 16(2):121--137, 2013.
[11]
C. Eickho, C. G. Harris, A. P. de Vries, and P. Srinivasan. Quality through ow and immersion: gamifying crowdsourced relevance assessments. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval, pages 871--880. ACM, 2012.
[12]
C. Grady and M. Lease. Crowdsourcing document relevance assessment with mechanical turk. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, CSLDAMT '10, pages 172--179, Stroudsburg, PA, USA, 2010. Association for Computational Linguistics.
[13]
M. Hirth, T. Hoβfeld, and P. Tran-Gia. Cheat-detection mechanisms for crowdsourcing. University of Würzburg, Tech. Rep, 474, 2010.
[14]
J. Howe. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. Crown Publishing Group, New York, NY, USA, 1 edition, 2008.
[15]
T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '05, pages 154--161, New York, NY, USA, 2005. ACM.
[16]
G. Kazai, J. Kamps, M. Koolen, and N. Milic-Frayling. Crowdsourcing for book search evaluation: impact of hit design on comparative system ranking. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 205--214. ACM, 2011.
[17]
G. Kazai, J. Kamps, and N. Milic-Frayling. Worker types and personality traits in crowdsourcing relevance labels. In Proceedings of the 20th ACM international conference on Information and knowledge management, pages 1941--1944. ACM, 2011.
[18]
G. Kazai, J. Kamps, and N. Milic-Frayling. An analysis of human factors and label accuracy in crowdsourcing relevance judgments. Information retrieval, 16(2):138--178, 2013.
[19]
G. Kazai and N. Milic-Frayling. On the evaluation of the quality of relevance assessments collected through crowdsourcing. In SIGIR Workshop on Future of IR Evaluation. Association for Computing Machinery, Inc., July 2009.
[20]
A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '08, pages 453--456, New York, NY, USA, 2008. ACM.
[21]
A. Krause, A. Singh, and C. Guestrin. Near-optimal sensor placements in gaussian processes: Theory, efficient algorithms and empirical studies. The Journal of Machine Learning Research, 9:235--284, 2008.
[22]
Q. V. Le and T. Mikolov. Distributed representations of sentences and documents. arXiv preprint arXiv:1405.4053, 2014.
[23]
M. Lease. On quality control and machine learning in crowdsourcing. Human Computation, 11:11, 2011.
[24]
M. Lease and G. Kazai. Overview of the trec 2011 crowdsourcing track. In Proceedings of the text retrieval conference (TREC), 2011.
[25]
D. J. MacKay. Information-based objective functions for active data selection. Neural computation, 4(4):590--604, 1992.
[26]
C. C. Marshall and F. M. Shipman. The ownership and reuse of visual media. In Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries, pages 157--166. ACM, 2011.
[27]
G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. An analysis of approximations for maximizing submodular set functionsâ Ti. Mathematical Programming, 14(1):265--294, 1978.
[28]
J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pages 1532--1543, 2014.
[29]
N. Ramakrishnan, C. Bailey-Kellogg, S. Tadepalli, V. Pandey, et al. Gaussian processes for active data mining of spatial aggregates. In SDM, pages 427--438. SIAM, 2005.
[30]
M. Sabou, K. Bontcheva, L. Derczynski, and A. Scharl. Corpus annotation through crowdsourcing: Towards best practice guidelines. In LREC, pages 859--866, 2014.
[31]
N. Srinivas, A. Krause, S. M. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint arXiv:0912.3995, 2009.
[32]
C. J. van Rijsbergen and K. SPARCK JONES. A test for the separation of relevant and non-relevant documents in experimental retrieval collections. Journal of Documentation, 29(3):251--257, 1973.
[33]
E. M. Voorhees and D. K. Harman. TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing). The MIT Press, 2005.
[34]
J. Wang, S. Faridani, and P. G. Ipeirotis. Estimating the completion time of crowdsourced tasks using survival analysis models. Crowdsourcing for Search and Data Mining (CSDM 2011), page 31, 2011.
[35]
F. Wilcoxon. Individual comparisons by ranking methods. Biometrics bulletin, 1(6):80--83, 1945.
[36]
Y. Yan, G. M. Fung, R. Rosales, and J. G. Dy. Active learning from crowds. In Proceedings of the 28th international conference on machine learning (ICML-11), pages 1161--1168, 2011.

Cited By

View all
  • (2022)Cost-effective crowdsourced join queries for entity resolution without prior knowledgeFuture Generation Computer Systems10.1016/j.future.2021.09.008127:C(240-251)Online publication date: 1-Feb-2022
  • (2022)Hint: harnessing the wisdom of crowds for handling multi-phase tasksNeural Computing and Applications10.1007/s00521-021-06825-735:31(22911-22933)Online publication date: 17-Jan-2022
  • (2021)Task Selection Based on Worker Performance Prediction in Gamified CrowdsourcingAgents and Multi-Agent Systems: Technologies and Applications 202110.1007/978-981-16-2994-5_6(65-75)Online publication date: 8-Jun-2021
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CIKM '16: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management
October 2016
2566 pages
ISBN:9781450340731
DOI:10.1145/2983323
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 October 2016

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. active learning
  2. crowdsourcing
  3. relevance assessment

Qualifiers

  • Research-article

Conference

CIKM'16
Sponsor:
CIKM'16: ACM Conference on Information and Knowledge Management
October 24 - 28, 2016
Indiana, Indianapolis, USA

Acceptance Rates

CIKM '16 Paper Acceptance Rate 160 of 701 submissions, 23%;
Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)2
  • Downloads (Last 6 weeks)0
Reflects downloads up to 28 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2022)Cost-effective crowdsourced join queries for entity resolution without prior knowledgeFuture Generation Computer Systems10.1016/j.future.2021.09.008127:C(240-251)Online publication date: 1-Feb-2022
  • (2022)Hint: harnessing the wisdom of crowds for handling multi-phase tasksNeural Computing and Applications10.1007/s00521-021-06825-735:31(22911-22933)Online publication date: 17-Jan-2022
  • (2021)Task Selection Based on Worker Performance Prediction in Gamified CrowdsourcingAgents and Multi-Agent Systems: Technologies and Applications 202110.1007/978-981-16-2994-5_6(65-75)Online publication date: 8-Jun-2021
  • (2020)Search Result Explanations Improve Efficiency and TrustProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401279(1597-1600)Online publication date: 25-Jul-2020
  • (2019)The Practice of CrowdsourcingSynthesis Lectures on Information Concepts, Retrieval, and Services10.2200/S00904ED1V01Y201903ICR06611:1(1-149)Online publication date: 28-May-2019
  • (2019)Rehumanized CrowdsourcingProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300773(1-12)Online publication date: 2-May-2019
  • (2018)Cognitive Biases in CrowdsourcingProceedings of the Eleventh ACM International Conference on Web Search and Data Mining10.1145/3159652.3159654(162-170)Online publication date: 2-Feb-2018
  • (2018)Quality Control in CrowdsourcingACM Computing Surveys10.1145/314814851:1(1-40)Online publication date: 4-Jan-2018
  • (2017)Budgeted Task Scheduling for Crowdsourced Knowledge AcquisitionProceedings of the 2017 ACM on Conference on Information and Knowledge Management10.1145/3132847.3133002(1059-1068)Online publication date: 6-Nov-2017

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media