Abstract
We develop flexible reward plans to elicit truthful predictive probability distribution over a set of uncertain events from workers. In general, strictly proper scoring rules for categorical events only reward a worker for an event that actually occurred. However, different incorrect predictions vary in quality, and the principal would like to assign different rewards to them, according to her subjective similarity among events; e.g. a prediction of overcast is closer to sunny than rainy.
We propose concrete methods so that the principal can assign rewards for incorrect predictions according to her similarity between events. We focus on two representative examples of strictly proper scoring rules: spherical and quadratic, where a worker’s expected utility is represented as the inner product of her truthful predictive probability and her declared probability. In this paper, we generalize the inner product by introducing a reward matrix that defines a reward for each prediction-outcome pair. We first show that if the reward matrix is symmetric and positive definite, both the spherical and quadratic proper scoring rules guarantee the maximization of a worker’s expected utility when she truthfully declares her prediction. We next compare our rules with the original spherical/quadratic proper scoring rules in terms of the variance of rewards obtained by workers. Finally, we show our experimental results using Amazon Mechanical Turk.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Akasiadis, C., Chalkiadakis, G.: Agent cooperatives for effective power consumption shifting. In: Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI 2013), pp. 1263–1269 (2013)
Brier, G.W.: Verification of forecasts expressed in terms of probability. Monthly Weather Review 78(1), 1–3 (1950)
Chen, Y., Gao, X.A., Goldstein, R., Kash, I.A.: Market manipulation with outside incentives. In: Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI 2011), pp. 614–619 (2011)
Chen, Y., Pennock, D.M.: Designing markets for prediction. AI Magazine 31(4), 42–52 (2010)
Conitzer, V.: Prediction markets, mechanism design, and cooperative game theory. In: Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence (UAI 2009), pp. 101–108 (2009)
Gneiting, T., Raftery, A.E.: Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102(477), 359–378 (2007)
Huang, E.H., Shoham, Y.: Price manipulation in prediction markets:analysis and mitigation. In: Proceedings of the 13th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2014), pp. 213–220 (2014)
Law, E., Ahn, L.V.: Human Computation. Morgan & Claypool Publishers (2011)
Matheson, J.E., Winkler, R.L.: Scoring rules for continuous probability distributions. Management Science 22(10), 1087–1096 (1976)
Prelec, D.: A bayesian truth serum for subjective data. Science 306(5695), 462–466 (2004)
Radanovic, G., Faltings, B.: A robust bayesian truth serum for non-binary signals. In: Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI 2013), pp. 833–839 (2013)
Robu, V., Kota, R., Chalkiadakis, G., Rogers, A., Jennings, N.R.: Cooperative virtual power plant formation using scoring rules. In: Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI 2012), pp. 370–376 (2012)
Savage, L.J.: Elicitation of personal probabilities and expectations. Journal of the American Statistical Association 66(336), 783–801 (1971)
Witkowski, J., Parkes, D.C.: A robust bayesian truth serum for small populations. In: Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI 2012), pp. 1492–1498 (2012)
Wolfers, J., Zitzewitz, E.: Prediction markets. Journal of Economic Perspectives 18(2), 107–126 (2004)
Zhang, P., Chen, Y.: Elicitability and knowledge-free elicitation with peer prediction. In: Proceedings of the 13th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2014), pp. 245–252 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Sakurai, Y., Shinoda, M., Oyama, S., Yokoo, M. (2015). Flexible Reward Plans for Crowdsourced Tasks. In: Chen, Q., Torroni, P., Villata, S., Hsu, J., Omicini, A. (eds) PRIMA 2015: Principles and Practice of Multi-Agent Systems. PRIMA 2015. Lecture Notes in Computer Science(), vol 9387. Springer, Cham. https://doi.org/10.1007/978-3-319-25524-8_25
Download citation
DOI: https://doi.org/10.1007/978-3-319-25524-8_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-25523-1
Online ISBN: 978-3-319-25524-8
eBook Packages: Computer ScienceComputer Science (R0)