skip to main content
10.1145/2987386.2987413acmconferencesArticle/Chapter ViewAbstractPublication PagesracsConference Proceedingsconference-collections
research-article

A Task Ontology-based Model for Quality Control in Crowdsourcing Systems

Published: 11 October 2016 Publication History

Abstract

In the era of big data, a vast amount of data is created every day. Crowdsourcing systems have recently gained significance as an interesting practice in managing and performing big data operations. Crowdsourcing has facilitated the process of performing tasks that cannot be adequately solved by machines including image labeling, transcriptions, data validation and sentiment analysis. However, quality control remains one of the biggest challenges for crowdsourcing. Current crowdsourcing systems use the same quality control mechanism for evaluating different types of tasks. In this paper, we argue that quality mechanisms vary by task type. We propose a task ontology-based model to identify the most appropriate quality mechanism for a given task. The proposed model has been enriched by a reputation system to collect requesters' feedback on quality mechanisms. Accordingly, the reputation of each mechanism can be established and used for mapping between tasks and mechanisms. Description of the model's framework, algorithms, and its components' interaction are presented.

References

[1]
L. Litman, J. Robinson, and C. Rosenzweig, "The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk.," Behavior research methods, Springer US, Jun. 2014.
[2]
R. Buettner, "A Systematic Literature Review of Crowdsourcing Research from a Human Resource Management Perspective," in Proceedings of the 48th Hawaii International Conference on System Sciences, 2015.
[3]
M. Allahbakhsh, B. Benatallah, A. Ignjatovic, H. Motahari-Nezhad, E. Bertino, and S. Dustdar, "Quality Control in Crowdsourcing Systems: Issues and Directions," IEEE Internet Computing, vol. 17, no. 2, pp. 76--81, 2013.
[4]
D. Chang, C. H. Chen, and K. M. Lee, "A crowdsourcing development approach based on a neuro-fuzzy network for creating innovative product concepts," Neurocomputing, vol. 142, pp. 60--72, 2014.
[5]
M. Franklin, D. Kossman, T. Kraska, S. Ramesh, and R. Xin, "CrowdDB: answering queries with crowdsourcing," in Proceedings of the 2011 ACM SIGMOD International Conference on Management of data, 2011, pp. 61--72.
[6]
A. Kittur, B. Smus, S. Khamkar, and R. Kraut, "Crowdforge: Crowdsourcing complex work," in Proc. 24th Ann. ACM Symp. User Interface Software and Technology UIST'11, 2011, pp. 43--52.
[7]
J. Wang, P. G. Ipeirotis, and F. Provost, "A Framework for Quality Assurance in Crowdsourcing," NYU Working Paper, number 2451/31833, 2013.
[8]
J. Parson, D. Braga, M. Tjalve, and J. Oh, "Evaluating Voice Quality and Speech Synthesis Using Crowdsourcing," in Text, Speech, and Dialogue Lecture Notes in Computer Science Volume 8082, Springer Berlin Heidelberg, 2013, pp. 233--240.
[9]
G. Xintong, W. Hongzhi, Y. Song, and G. Hong, "Brief survey of crowdsourcing for data mining," Expert Systems with Applications, vol. 41, no. 17, pp. 7987--7994, Jul. 2014.
[10]
S. Egelman, E. H. Chi, and S. Dow, "Crowdsourcing in HCI Research," in Ways of Knowing in HCI, J. S. Olson and W. A. Kellogg, Eds. New York, NY: Springer New York, 2014, pp. 267--289.
[11]
H. J. Khasraghi and A. Aghaie, "Crowdsourcing contests: understanding the effect of competitors' participation history on their performance," Behaviour & Information Technology, vol. 33, no. 12, pp. 1383--1395, Mar. 2014.
[12]
A. Kittur, J. Nickerson, M. Bernstein, E. Gerber, A. Shaw, J. Zimmerman, M. Lease, and J. Horton, "The future of crowd work," in proceedings of the 2013 conference on Computer supported cooperative work (CSCW '13), 2013, pp. 1301--1317.
[13]
Y. Zhao and Q. Zhu, "Evaluation on crowdsourcing research: Current status and future direction," Information Systems Frontiers, vol. 16, no. 3, pp. 417--434, Apr. 2012.
[14]
J. Howe, Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. New York, NY: Crown Business, 2008.
[15]
D. C. Brabham, "Crowdsourcing as a Model for Problem Solving: An Introduction and Cases," The International Journal of Research into New Media Technologies, vol. 14, no. 1, pp. 75--90, Feb. 2008.
[16]
G. D. Saxton, O. Oh, and R. Kishore, "Rules of Crowdsourcing: Models, Issues, and Systems of Control," Information Systems Management, vol. 30, no. 1, pp. 2--20, Jan. 2013.
[17]
A. Doan, R. Ramakrishnan, and A. Y. Halevy, "Crowdsourcing systems on the World-Wide Web," Communications of the ACM, vol. 54, no. 4, pp. 86--96, 2011.
[18]
MTurk, "MTurk: Amazon Mechanical Turk," 2016. {Online}. Available: http://www.mturk.com/. {Accessed: 01-Jan-2016}.
[19]
Upwork, "Upwork," 2016. {Online}. Available: https://www.upwork.com/. {Accessed: 24-Mar-2016}.
[20]
Freelancer, "Freelancer," 2016. {Online}. Available: https://www.freelancer.com/. {Accessed: 24-Mar-2016}.
[21]
Threadless, "Threadless," 2016. {Online}. Available: https://www.threadless.com/. {Accessed: 04-Jan-2016}.
[22]
IStockPhoto, "iStockPhoto," 2016. {Online}. Available: http://www.istockphoto.com. {Accessed: 04-Jan-2016}.
[23]
C. Keimel, J. Habigt, C. Horch, and K. Diepold, "Qualitycrowd---a framework for crowd-based quality evaluation," in Proceedings of Picture Coding Symposium (PCS 2012), 2012, pp. 245--248.
[24]
CNNiReport, "CNN's iReport," 2016. {Online}. Available: http://ireport.cnn.com/. {Accessed: 04-Jan-2016}.
[25]
CastingWords, "CastingWords," 2016. {Online}. Available: https://castingwords.com/. {Accessed: 24-Mar-2016}.
[26]
L. Cilliers and S. Flowerday, Eds., "Information security in a public safety, participatory crowdsourcing smart city project," in Proceedings of the World Congress on the 2014 Internet Security (WorldCIS), 2014, pp. 36--41.
[27]
D. Schuurman, B. Baccarne, L. De Marez, and P. Mechant, "Smart ideas for smart cities: Investigating crowdsourcing for generating and selecting ideas for ICT innovation in a city context," Journal of Theoretical and Applied Electronic Commerce Research, vol. 7, no. 3, pp. 49--62, 2012.
[28]
M. Olson, "The amateur search," ACM SIGMOD Record, vol. 37, no. 2, pp. 21--24, 2008.
[29]
T. Hobfeld, M. Hirth, P. Korshunov, P. Hanhart, B. Gardlo, C. Keimel, and C. Timmerer, "Survey of web-based crowdsourcing frameworks for subjective quality assessment," in Proceedings of the 16th International Workshop on Multimedia Signal Processing (MMSP), 2014, pp. 22--24.
[30]
N. Luz, N. Silva, and P. Novais, "Generating Human-Computer Micro-task Workflows from Domain Ontologies," Human-Computer Interaction Theories, Methods, and Tools, Lecture Notes in Computer Science, Springer International Publishing, vol. 8510, pp. 98--109, 2014.
[31]
D. Schall, Service-Oriented Crowdsourcing, 2012 editi. Springer, 2012.
[32]
E. Tromp and M. Pechenizkiy, "SentiCorr: Multilingual Sentiment Analysis of Personal Correspondence," 2011 IEEE 11th International Conference on Data Mining Workshops, pp. 1247--1250, Dec. 2011.
[33]
M. Post, C. Callison-Burch, and M. Osborne, "Constructing parallel corpora for six indian languages via crowdsourcing," in Proceedings of the 7th Workshop on Statistical Machine Translation, 2012, pp. 401--409.
[34]
A. Quinn and B. Bederson, "Human computation: a survey and taxonomy of a growing field," in CHI '11 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2011, pp. 1403--1412.
[35]
N. Savage, "Gaining wisdom from crowds," Communications of the ACM, vol. 55, no. 3, pp. 13--15, Mar. 2012.
[36]
A. Kittur, E. H. Chi, and B. Suh, "Crowdsourcing user studies with Mechanical Turk," in Proceeding of the twenty-sixth annual CHI conference on Human factors in computing systems - CHI '08, 2008, pp. 453--456.
[37]
M.-C. Yuen, I. King, and K.-S. Leung, "A Survey of Crowdsourcing Systems," in Proceedings of 2011 IEEE Third Int'l Conference on Privacy, Security, Risk and Trust and 2011 IEEE Third Int'l Conference on Social Computing, 2011, pp. 766--773.
[38]
P. G. Ipeirotis, F. Provost, and J. Wang, "Quality management on Amazon Mechanical Turk," in Proceedings of the ACM SIGKDD Workshop on Human Computation - HCOMP '10, 2010, p. 64.
[39]
Samasource, "Samasource," 2016. {Online}. Available: http://www.samasource.org/. {Accessed: 04-Jan-2016}.
[40]
M. Allahbakhsh, A. Ignjatovic, B. Benatallah, S.-M.-R. Beheshti, E. Bertino, and N. Foo, "Reputation Management in Crowdsourcing Systems," in Proceedings of the 8th IEEE International Conference on Collaborative Computing: Networking, Applications and Worksharing, 2012.
[41]
N. Luz, N. Silva, and P. Novais, "A survey of task-oriented crowdsourcing," Artificial Intelligence Review, vol. 44, no. 2, pp. 187--213, Aug. 2015.
[42]
K. El Maarry, W. Balke, H. Cho, S. Hwang, and Y. Baba, "Skill Ontology-Based Model for Quality Assurance in Crowdsourcing," Database Systems for Advanced Applications, LNCS, vol. 8505, pp. 376--387, 2014.
[43]
E. Maximilien and M. Singh, "A framework and ontology for dynamic web services selection," Internet Computing, IEEE, no. October, pp. 84--93, 2004.
[44]
P. Iske and W. Boersma, "Connected brains: Question and answer systems for knowledge sharing: concepts, implementation and return on investment," Journal of Knowledge Management, vol. 9, no. 1, pp. 126--145, 2005.
[45]
X. Lam, T. Vu, and T. Le, "Addressing cold-start problem in recommendation systems," Proceedings of the 2nd international, pp. 208--211, 2008.
[46]
A. I. Schein, A. Popescul, L. H. Ungar, and D. M. Pennock, "Methods and metrics for cold-start recommendations," Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval SIGIR 02, vol. 46, no. Sigir, pp. 253--260, 2002.
[47]
H. Al-Dossari and J. Shao, "Modelling Confidence for Quality of Service Assessment in Cloud Computing," in CONF-IRM 2013 Proceedings, paper 48, 2013.
[48]
Z. Huang, H. Chen, and D. Zeng, "Applying associative retrieval techniques to alleviate the sparsity problem in collaborative filtering," ACM Transactions on Information Systems, vol. 22, no. 1, pp. 116--142, 2004.
[49]
M. Papagelis, D. Plexousakis, and T. Kutsuras, "Alleviating the sparsity problem of collaborative filtering using trust inferences," Proceedings of the Third international conference on Trust Management, pp. 224--239, 2005.
[50]
J. Zhang and R. Cohen, "A personalized approach to address unfair ratings in multi-agent reputation systems.," in Proceedings of the AAMAS Workshop on Trust in Agent Societies, 2006.
[51]
B. Pang and L. Lee, "Opinion Mining and Sentiment Analysis," Foundations and Trends in Information Retrieval, vol. 2, no. 1-2, pp. 1--135, 2008.

Cited By

View all
  • (2023)Towards Artificial Intelligence Augmenting Facilitation: AI Affordances in Macro-Task CrowdsourcingGroup Decision and Negotiation10.1007/s10726-022-09801-132:1(75-124)Online publication date: 10-Jan-2023
  • (2019)Key Crowdsourcing Technologies for Product Design and DevelopmentInternational Journal of Automation and Computing10.1007/s11633-018-1138-716:1(1-15)Online publication date: 1-Feb-2019
  • (2018)An Algorithm of Crowdsourcing Answer Integration Based on Specialty Categories of WorkersProceedings of the Fifth Euro-China Conference on Intelligent Data Analysis and Applications10.1007/978-3-030-03766-6_4(25-35)Online publication date: 25-Dec-2018
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
RACS '16: Proceedings of the International Conference on Research in Adaptive and Convergent Systems
October 2016
266 pages
ISBN:9781450344555
DOI:10.1145/2987386
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 October 2016

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Big Data
  2. Crowd Computing
  3. Crowdsourcing
  4. HITs
  5. Human Computation
  6. MTurk
  7. Ontology
  8. Quality Control
  9. Reputation

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

RACS '16
Sponsor:

Acceptance Rates

RACS '16 Paper Acceptance Rate 40 of 161 submissions, 25%;
Overall Acceptance Rate 393 of 1,581 submissions, 25%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)1
  • Downloads (Last 6 weeks)0
Reflects downloads up to 22 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Towards Artificial Intelligence Augmenting Facilitation: AI Affordances in Macro-Task CrowdsourcingGroup Decision and Negotiation10.1007/s10726-022-09801-132:1(75-124)Online publication date: 10-Jan-2023
  • (2019)Key Crowdsourcing Technologies for Product Design and DevelopmentInternational Journal of Automation and Computing10.1007/s11633-018-1138-716:1(1-15)Online publication date: 1-Feb-2019
  • (2018)An Algorithm of Crowdsourcing Answer Integration Based on Specialty Categories of WorkersProceedings of the Fifth Euro-China Conference on Intelligent Data Analysis and Applications10.1007/978-3-030-03766-6_4(25-35)Online publication date: 25-Dec-2018
  • (2017)Towards a classification model for tasks in crowdsourcingProceedings of the Second International Conference on Internet of things, Data and Cloud Computing10.1145/3018896.3018916(1-7)Online publication date: 22-Mar-2017

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media