research-article

A Winners-Take-All Incentive Mechanism for Crowd-Powered Systems

Authors:

Lei YingAuthors Info & Claims

NetEcon '18: Proceedings of the 13th Workshop on Economics of Networks, Systems and Computation

Article No.: 3, Pages 1 - 6

https://doi.org/10.1145/3230654.3230657

Published: 18 June 2018 Publication History

Abstract

This paper studies incentive mechanisms for crowd-powered systems, including applications such as collection of personal data for big-data analytics and crowdsourcing. In big-data analytics using personal data, an individual may control the quality of reported data via a privacy-preserving mechanism that randomizes the answer. In crowdsourcing, the quality of the reported answer depends on the amount of effort spent by a worker or a team. In these applications, incentive mechanisms are critical for eliciting data/answers with target quality. This paper focuses the following two fundamental questions: what is the minimum payment required to incentivize an individual to submit a data/answer with quality level ∈? and what incentive mechanisms can achieve the minimum payment?

Let ∈i denote the quality of the data/answer reported by individual i. In this paper, we first derive a lower bound on the minimum amount of payment required for guaranteeing quality level ∈i. Inspired by the lower bound, we propose an incentive mechanism, named Winners-Take-All (WINTALL). WINTALL first decides a winning answer based on the reported data, cost functions of individuals, and some prior distribution; and then pays to individuals whose reported data match the winning answer. Under some assumptions, we show that the expected payment of WINTALL matches the lower bound. In the application of private discrete distribution estimation, we show that WINTALL simply rewards individuals whose reported answers match the most popular answer from the reported ones (the prior distribution is not needed in this case).

References

[1]

https://images.apple.com/privacy/docs/Differential_Privacy_Overview.pdf.

[2]

Cai, Y., Daskalakis, C., and Papadimitriou, C. Optimum statistical estimation with strategic data sources. In Proc. Conf. Learning Theory (COLT) (2015), pp. 280--296.

[3]

Cummings, R., Ligett, K., Roth, A., Wu, Z. S., and Ziani, J. Accuracy for sale: Aggregating data with a variance constraint. In Proc. Conf. Innovations in Theoretical Computer Science (2015), pp. 317--324.

Digital Library

[4]

Dasgupta, A., and Ghosh, A. Crowdsourced judgement elicitation with endogenous proficiency. In Proc. Int. Conf. World Wide Web (WWW) (2013), pp. 319--330.

Digital Library

[5]

Erlingsson, Ú., Pihur, V., and Korolova, A. RAPPOR: Randomized aggregatable privacy-preserving ordinal response. In Proc. ACM Conf. Computer and Communications Security (CCS) (Scottsdale, AZ, 2014), pp. 1054--1067.

Digital Library

[6]

Kairouz, P., Bonawitz, K., and Ramage, D. Discrete distribution estimation under local privacy. In Int. Conf. Machine Learning (ICML) (2016), pp. 2436--2444.

Digital Library

[7]

Khetan, A., and Oh, S. Achieving budget-optimality with adaptive schemes in crowdsourcing. In Advances Neural Information Processing Systems (NIPS) (2016), pp. 4844--4852.

Digital Library

[8]

Liu, y., and Liu, M. An online learning approach to improving the quality of crowdsourcing. In Proc. Ann. ACM SIGMETRICS Conf. (New York, NY, USA, 2015), pp. 217--230.

Digital Library

[9]

Miller, N., Resnick, P., and Zeckhauser, R. Eliciting informative feedback: The peer-prediction method. In Computing with Social Trust, Human--Computer Interaction Series. Springer London, 2009, pp. 185--212.

[10]

Prelec, D. A bayesian truth serum for subjective data. Science 306, 5695 (2004), 462--466.

[11]

Radanovic, G., and faltings, B. A robust bayesian truth serum for non-binary signals. In AAAI Conf. Artificial Intelligence (2013), pp. 833--839.

Digital Library

[12]

Radanovic, G., and Faltings, B. Incentives for truthful information elicitation of continuous signals. In AAAI Conf. Artificial Intelligence (2014), pp. 770--776.

Digital Library

[13]

Radanovic, G., and Faltings, B. Incentive schemes for participatory sensing. In Proc. Int. Conf. Autonomous Agents and Multiagent Systems (2015), International Foundation for Autonomous Agents and Multiagent Systems, pp. 1081--1089.

Digital Library

[14]

Shah, N., and Zhou, D. No oops, you won't do it again: Mechanisms for self-correction in crowdsourcing. In International Conference on Machine Learning (2016), pp. 1--10.

Digital Library

[15]

Shah, N., Zhou, D., and Peres, Y. Approval voting and incentives in crowdsourcing. In International Conference on Machine Learning (2015), pp. 10--19.

Digital Library

[16]

Shah, N. B., and Zhou, D. Double or nothing: Multiplicative incentive mechanisms for crowdsourcing. In Advances in neural information processing systems (2015), pp. 1--9.

Digital Library

[17]

Shnayder, V., Agarwal, A., Frongillo, R., and Parkes, D. C. Informed truthfulness in multi-task peer prediction. In Proceedings of the 2016 ACM Conference on Economics and Computation (2016), pp. 179--196.

Digital Library

[18]

Von Ahn, L., and Dabbish, L. Labeling images with a computer game. In Proc. the SIGCHI conf. Human factors in computing systems (2004), pp. 319--326.

Digital Library

[19]

Wang, W., Ying, L., and Zhang, J. A game-theoretic approach to quality control for collecting privacy-preserving data. In Proc. Annu. Allerton Conf. Communication, Control and Computing (Monticello, IL, Sept. 2015).

[20]

Wang, W., Ying, L., and Zhang, J. The value of privacy: Strategic data subjects, incentive mechanisms and fundamental limits. In Proc. Ann. ACM SIGMETRICS Conf. (Antibes Juan-les-Pins, France, June 2016).

Digital Library

[21]

Wauthier, F. L., and Jordan, M. I. Bayesian bias mitigation for crowdsourcing. In Advances Neural Information Processing Systems (NIPS) (2011), pp. 1800--1808.

Digital Library

[22]

Witkowski, J., and Parkes, D. C. A robust bayesian truth serum for small populations. In AAAI Conf. Artificial Intelligence (2012), vol. 12, pp. 1492--1498.

Digital Library

Cited By

Zhou YNelakurthi AMaciejewski RFan WHe J(2020)Crowd Teaching with Imperfect LabelsProceedings of The Web Conference 202010.1145/3366423.3380099(110-121)Online publication date: 20-Apr-2020
https://dl.acm.org/doi/10.1145/3366423.3380099
Zhou YYing LHe J(2019)Multi-task Crowdsourcing via an Optimization FrameworkACM Transactions on Knowledge Discovery from Data10.1145/331022713:3(1-26)Online publication date: 29-May-2019
https://dl.acm.org/doi/10.1145/3310227
Zhou YMa FGao JHe JTeredesai AKumar VLi YRosales RTerzi EKarypis G(2019)Optimizing the Wisdom of the CrowdProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining10.1145/3292500.3332277(3231-3232)Online publication date: 25-Jul-2019
https://dl.acm.org/doi/10.1145/3292500.3332277

Recommendations

Incentive Mechanism Design to Meet Task Criteria in Crowdsourcing: How to Determine Your Budget

In crowdsourcing markets, a requester announces a task and calls for contribution from potential participants. With strategic participants, the requester needs to reward the participants to introduce the incentives of participation. However, it is ...
An incentive mechanism to promote honesty among seller agents in electronic marketplaces

With the rapid development of online shopping, usually, sellers and buyers have virtual identities, which are not verified beforehand. As a result, establishing trust between sellers and buyers is much harder than before where sellers and buyers would ...
Incentive-Compatible Learning of Reserve Prices for Repeated Auctions
How can an auctioneer optimize revenue by learning the reserve prices from the bids in the previous auctions? How should the long-term incentives and strategic behavior of the bidders be taken into account? Motivated in part by applications in online ...
Large fractions of online advertisements are sold via repeated second-price auctions. In these auctions, the reserve price is the main tool for the auctioneer to boost revenues. In this work, we investigate the following question: how can the auctioneer ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

NetEcon '18: Proceedings of the 13th Workshop on Economics of Networks, Systems and Computation

June 2018

35 pages

ISBN:9781450359160

DOI:10.1145/3230654

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMETRICS: ACM Special Interest Group on Measurement and Evaluation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

SIGMETRICS '18

Sponsor:

SIGMETRICS

SIGMETRICS '18: ACM SIGMETRICS / International Conference on Measurement and Modeling of Computer Systems

June 18, 2018

CA, Irvine, USA

Acceptance Rates

NetEcon '18 Paper Acceptance Rate 10 of 18 submissions, 56%;

Overall Acceptance Rate 10 of 18 submissions, 56%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
91
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhou YNelakurthi AMaciejewski RFan WHe J(2020)Crowd Teaching with Imperfect LabelsProceedings of The Web Conference 202010.1145/3366423.3380099(110-121)Online publication date: 20-Apr-2020
https://dl.acm.org/doi/10.1145/3366423.3380099
Zhou YYing LHe J(2019)Multi-task Crowdsourcing via an Optimization FrameworkACM Transactions on Knowledge Discovery from Data10.1145/331022713:3(1-26)Online publication date: 29-May-2019
https://dl.acm.org/doi/10.1145/3310227
Zhou YMa FGao JHe JTeredesai AKumar VLi YRosales RTerzi EKarypis G(2019)Optimizing the Wisdom of the CrowdProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining10.1145/3292500.3332277(3231-3232)Online publication date: 25-Jul-2019
https://dl.acm.org/doi/10.1145/3292500.3332277

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten