research-article

Quality-aware Dynamic Task Assignment in Human+AI Crowd

Authors:

Masaki Kobayashi,

Kei Wakabayashi,

Atsuyuki MorishimaAuthors Info & Claims

WWW '20: Companion Proceedings of the Web Conference 2020

Pages 118 - 119

https://doi.org/10.1145/3366424.3382727

Published: 20 April 2020 Publication History

Get Access

Abstract

Today, crowdsourcing the creation of AI programs is a common practice. However, combining the created programs with other programs and human computations to obtain efficient human-in-the-loop solutions is not trivial. Our paper proposes a framework to address the problem of dynamically assigning tasks to a crowd of AI and human workers, and presents the results of a preliminary experiment.

References

[1]

Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh. 2018. Quality Control in Crowdsourcing: A Survey of Quality Attributes, Assessment Techniques, and Assurance Actions. ACM Comput. Surv. 51, 1, Article 7 (Jan. 2018), 40 pages. https://doi.org/10.1145/3148148

Digital Library

Google Scholar

[2]

Yan Yan, Romer Rosales, Glenn Fung, and Jennifer G. Dy. 2011. Active Learning from Crowds. In ICML’11 (Bellevue, Washington, USA). Omnipress, USA, 1161–1168. http://dl.acm.org/citation.cfm?id=3104482.3104628

Google Scholar

Cited By

View all

Flathmann CSchelble BRosopa PMcNeese NMallick RMadathil K(2023)Examining the impact of varying levels of AI teammate influence on human-AI teamsInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103061177:COnline publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103061
Li KLi GWang YHuang YLiu ZWu Z(2021)CrowdRL: An End-to-End Reinforcement Learning Framework for Data Labelling2021 IEEE 37th International Conference on Data Engineering (ICDE)10.1109/ICDE51399.2021.00032(289-300)Online publication date: Apr-2021
https://doi.org/10.1109/ICDE51399.2021.00032
Correia AJameel SSchneider DParedes HFonseca B(2020)A Workflow-Based Methodological Framework for Hybrid Human-AI Enabled Scientometrics2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9378096(2876-2883)Online publication date: 10-Dec-2020
https://doi.org/10.1109/BigData50022.2020.9378096
Show More Cited By

Index Terms

Quality-aware Dynamic Task Assignment in Human+AI Crowd

Index terms have been assigned to the content through auto-classification.

Recommendations

Quality-aware Online Task Assignment in Mobile Crowdsourcing

In recent years, mobile crowdsourcing has emerged as a powerful computation paradigm to harness human power to perform spatial tasks such as collecting real-time traffic information and checking product prices in a specific supermarket. A fundamental ...
Crowdsourcing usage, task assignment methods, and crowdsourcing platforms: A systematic literature review
Abstract
Crowdsourcing is simply the outsourcing of different tasks or work to a diverse group of individuals in an open call for the purpose of utilizing human intelligence. Crowdsourcing nowadays used to support and enhance software engineering in ...
A workload-dependent task assignment policy for crowdsourcing

Crowdsourcing marketplaces have emerged as an effective tool for high-speed, low-cost labeling of massive data sets. Since the labeling accuracy can greatly vary from worker to worker, we are faced with the problem of assigning labeling tasks to workers ...

Comments

Information & Contributors

Information

Published In

WWW '20: Companion Proceedings of the Web Conference 2020

April 2020

854 pages

ISBN:9781450370240

DOI:10.1145/3366424

Editors:
Amal El Fallah Seghrouchni
Sorbonne University, France
,
Gita Sukthankar
University of Central Florida, United States
,
Tie-Yan Liu
Microsoft Research Asia, China
,
Maarten van Steen
University of Twente, Netherlands

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 April 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '20

Sponsor:

SIGWEB

WWW '20: The Web Conference 2020

April 20 - 24, 2020

Taipei, Taiwan

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
216
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)2

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Flathmann CSchelble BRosopa PMcNeese NMallick RMadathil K(2023)Examining the impact of varying levels of AI teammate influence on human-AI teamsInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103061177:COnline publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103061
Li KLi GWang YHuang YLiu ZWu Z(2021)CrowdRL: An End-to-End Reinforcement Learning Framework for Data Labelling2021 IEEE 37th International Conference on Data Engineering (ICDE)10.1109/ICDE51399.2021.00032(289-300)Online publication date: Apr-2021
https://doi.org/10.1109/ICDE51399.2021.00032
Correia AJameel SSchneider DParedes HFonseca B(2020)A Workflow-Based Methodological Framework for Hybrid Human-AI Enabled Scientometrics2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9378096(2876-2883)Online publication date: 10-Dec-2020
https://doi.org/10.1109/BigData50022.2020.9378096
Inoguchi MTamura KUo KKobayashi M(2020)Validation of CyborgCrowd Implementation Possibility for Situation Awareness in Urgent Disaster Response -Case Study of International Disaster Response in 2019-2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9377838(3062-3071)Online publication date: 10-Dec-2020
https://doi.org/10.1109/BigData50022.2020.9377838

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Cited By

Index Terms

Recommendations

Quality-aware Online Task Assignment in Mobile Crowdsourcing

Crowdsourcing usage, task assignment methods, and crowdsourcing platforms: A systematic literature review

A workload-dependent task assignment policy for crowdsourcing

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

HTML Format

Share

Share this Publication link

Share on social media

Affiliations