Hostname: page-component-669899f699-tzmfd Total loading time: 0 Render date: 2025-05-02T18:49:28.276Z Has data issue: false hasContentIssue false
  • English
  • Français

A survey of incentive engineering for crowdsourcing

Published online by Cambridge University Press:  17 April 2018

Conor Muldoon ,
Michael J. O’Grady  and
Gregory M. P. O’Hare
Conor Muldoon
Affiliation:
School of Computer Science, University College Dublin, Belfield, Dublin 4, Ireland e-mail: conor.muldoon@ucd.ie
Michael J. O’Grady
Affiliation:
School of Computer Science, University College Dublin, Belfield, Dublin 4, Ireland e-mail: michael.j.ogrady@ucd.ie
Gregory M. P. O’Hare
Affiliation:
School of Computer Science, University College Dublin, Belfield, Dublin 4, Ireland e-mail: gregory.ohare@ucd.ie
Get access Rights & Permissions [Opens in a new window]

Abstract

With the growth of the Internet, crowdsourcing has become a popular way to perform intelligence tasks that hitherto would be either performed internally within an organization or not undertaken due to prohibitive costs and the lack of an appropriate communications infrastructure. In crowdsourcing systems, whereby multiple agents are not under the direct control of a system designer, it cannot be assumed that agents will act in a manner that is consistent with the objectives of the system designer or principal agent. In situations whereby agents’ goals are to maximize their return in crowdsourcing systems that offer financial or other rewards, strategies will be adopted by agents to game the system if appropriate mitigating measures are not put in place. The motivational and incentivization research space is quite large; it incorporates diverse techniques from a variety of different disciplines including behavioural economics, incentive theory, and game theory. This paper specifically focusses on game theoretic approaches to the problem in the crowdsourcing domain and places it in the context of the wider research landscape. It provides a survey of incentive engineering techniques that enable the creation of apt incentive structures in a range of different scenarios.

Type
Principles and Practice of Multi-Agent Systems
Information
The Knowledge Engineering Review , Volume 33 , 2018 , e2
Copyright
© Cambridge University Press, 2018 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Adriaens, T., Sutton-Croft, M., Owen, K., Brosens, D., van Valkenburg, J., Kilbey, D., Groom, Q., Ehmig, C., Thürkow, F., Van Hende, P. & Schneider, K. 2015. Trying to engage the crowd in recording invasive alien species in Europe: experiences from two smartphone applications in northwest Europe. Management of Biological Invasions 6(2), 215–225.Google Scholar
Babaioff, M., Dughmi, S., Kleinberg, R. & Slivkins, A. 2015. Dynamic pricing with limited supply. ACM Transactions on Economics and Computation 3(1), 4.Google Scholar
Benkler, Y. & Nissenbaum, H. 2006. Commons-based peer production and virtue. Journal of Political Philosophy 14(4), 394–419.Google Scholar
Berg, N. & Gigerenzer, G. 2010. As-if behavioral economics: Neoclassical economics in disguise? History of Economic Ideas 18, 133–165.Google Scholar
Blum, A. & Monsour, Y. 2007. Learning, regret minimization, and equilibria. In Algorithmic Game Theory, Nisan, N., Roughgarden, T., Tardos, E. & Vazirani, V. V. (eds). Cambridge University Press, 79–102.Google Scholar
Bosha, E., Cilliers, L. & Flowerday, S. 2017. Incentive theory for a participatory crowdsourcing project in a developing country. SA Journal of Information Management 19(1), 7.Google Scholar
Brabham, D. C. 2013. Crowdsourcing. MIT Press.Google Scholar
Camerer, C. F. 2003. Behavioural studies of strategic thinking in games. Trends in Cognitive Sciences 7(5), 225–231.Google Scholar
Cameron, J., Banko, K. M & Pierce, W. D. 2001. Pervasive negative effects of rewards on intrinsic motivation: the myth continues. The Behavior Analyst 24(1), 1.Google Scholar
Cooper, H. M. 1988. Organizing knowledge syntheses: a taxonomy of literature reviews. Knowledge in Society 1(1), 104–126.Google Scholar
David, G., Michael, R., Erwin, F. & Martin, S. 2012. Crowdsourcing information systems: definition, typology and design. In ICIS 2012 The 33rd International Conference on Information Systems, Joey F George (ed.). Association for Information Systems/AIS Electronic Library (AISeL), 3562–3572.Google Scholar
Dayama, P., Narayanaswamy, B., Garg, D. & Narahari, Y. 2015. Truthful interval cover mechanisms for crowdsourcing applications. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, 1091–1099. International Foundation for Autonomous Agents and Multiagent Systems.Google Scholar
Drexler, K. E. & Miller, M. S. 1988. Incentive engineering for computational resource management. The Ecology of Computation 2, 231–266.Google Scholar
Ebden, M., Huynh, D., Moreau, L. & Roberts, S. 2015. Incentive engineering through subgraph matching with application to task allocation. HAIDM.Google Scholar
Endriss, U., Kraus, S., Lang, J. & Wooldridge, M. 2011. Designing incentives for boolean games. In The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 1, 79–86. International Foundation for Autonomous Agents and Multiagent Systems.Google Scholar
Ghani, E., Kerr, W. R. & Stanton, C. 2014. Diasporas and outsourcing: evidence from odesk and India. Management Science 60(7), 1677–1697.Google Scholar
Ghezzi, A., Gabelloni, D., Martini, A. & Natalicchio, A. 2017. Crowdsourcing: a review and suggestions for future research. International Journal of Management Reviews 0, 1–21.Google Scholar
Groves, T. 1973. Incentives in teams. Econometrica: Journal of the Econometric Society 41, 617–631.Google Scholar
Guo, B., Wang, Z., Yu, Z., Wang, Y., Yen, N. Y., Huang, R. & Zhou, X. 2015. Mobile crowd sensing and computing: the review of an emerging human-powered sensing paradigm. ACM Computing Surveys (CSUR) 48(1), 7.Google Scholar
Hamari, J. 2013. Transforming homo economicus into homo ludens: a field experiment on gamification in a utilitarian peer-to-peer trading service. Electronic Commerce Research and Applications 12(4), 236–245.Google Scholar
Higgins, C. I., Williams, J., Leibovici, D. G., Simonis, I., Davis, M. J., Muldoon, C., van Genuchten, P., O’Hare, G. M. P. & Wiemann, S. 2016. Citizen OBservatory WEB (COBWEB): a generic infrastructure platform to facilitate the collection of citizen science data for environmental monitoring. International Journal of Spatial Data Infrastructures Research 11(1), 20–48.Google Scholar
Howe, J. 2006a. Crowdsourcing: a definition. Retrieved 3 June 2006 from http://crowdsourcing.typepad.com/cs/2006/06/crowdsourcing_a.html.Google Scholar
Howe, J. 2006b. The rise of crowdsourcing. Wired Magazine 14(6), 1–4.Google Scholar
Kahneman, D. 2003. Maps of bounded rationality: psychology for behavioral economics. The American Economic Review 93(5), 1449–1475.Google Scholar
Koutsoupias, E. & Papadimitriou, C. 2009. Worst-case equilibria. Computer Science Review 3(2), 65–69.Google Scholar
Mao, A., Kamar, E., Chen, Y., Horvitz, E., Schwamb, M. E., Lintott, C. J. & Smith, A. M. 2013. Volunteering versus work for pay: incentives and tradeoffs in crowdsourcing. In First AAAI Conference on Human Computation and Crowdsourcing.Google Scholar
McLeod, S.A. 2007. BF Skinner: Operant conditioning. Retrieved September 9, 2009.Google Scholar
Morschheuser, B., Hamari, J. & Koivisto, J. 2016. Gamification in crowdsourcing: a review. In 2016 49th Hawaii International Conference on System Sciences (HICSS), 4375–4384. IEEE.Google Scholar
Muller, C.L., Chapman, L., Johnston, S., Kidd, C., Illingworth, S., Foody, G., Overeem, A. & Leigh, R.R. 2015. Crowdsourcing for climate and atmospheric sciences: current status and future potential. International Journal of Climatology 35(11), 3185–3203.Google Scholar
Myerson, R. B. 1981. Optimal auction design. Mathematics of Operations Research 6(1), 58–73.Google Scholar
Myerson, R. B. 2013. Game Theory. Analysis of Conflict. Harvard University Press.Google Scholar
Nash, J. 1951. Non-cooperative games. Annals of Mathematics 54, 286–295.Google Scholar
Nisan, N., Roughgarden, T., Tardos, E. & Vazirani, V. V. 2007. Algorithmic Game Theory, 1. Cambridge University Press.Google Scholar
O’Grady, M. J., Muldoon, C., Carr, D., Wan, J., Kroon, B. & O’Hare, G. M. P. 2016. Intelligent Sensing for Citizen Science - Challenges and Future Directions. Mobile Networks and Applications 21(2), 375–385.Google Scholar
Parkes, D. C. & Wellman, M. P. 2015. Economic reasoning and artificial intelligence. Science 349(6245), 267–272.Google Scholar
Petersen, K., Vakkalanka, S. & Kuzniarz, L. 2015. Guidelines for conducting systematic mapping studies in software engineering: an update. Information and Software Technology 64, 1–18.Google Scholar
Rabin, M. 2002. A perspective on psychology and economics. European Economic Review 46(4), 657–685.Google Scholar
Roughgarden, T. 2003. The price of anarchy is independent of the network topology. Journal of Computer and System Sciences 67(2), 341–364.Google Scholar
Roughgarden, T. 2010. Algorithmic game theory. Communications of the ACM 53(7), 78–86.Google Scholar
See, L., Fritz, S., Dias, E., Hendriks, E., Mijling, B., Snik, F., Stammes, P., Vescovi, F. D., Zeug, G., Mathieu, P.-P., Desnos, Y.-L. & Rast, M. 2016a. Supporting earth-observation calibration and validation: a new generation of tools for crowdsourcing and citizen science. IEEE Geoscience and Remote Sensing Magazine 4(3), 38–50.Google Scholar
See, L., Mooney, P., Foody, G., Bastin, L., Comber, A., Estima, J., Fritz, S., Kerle, N., Jiang, B., Laakso, M., Liu, H.-Y., Milčinski, G., Nikšič, M., Painho, M., Pődör, A., Olteanu-Raimond, A.-M. & Rutzinger, M. 2016b. Crowdsourcing, citizen science or volunteered geographic information? The current state of crowdsourced geographic information. ISPRS International Journal of Geo-Information 5(5), 55.Google Scholar
See, L., Schepaschenko, D., Lesiv, M., McCallum, I., Fritz, S., Comber, A., Perger, C., Schill, C., Zhao, Y., Maus, V., Siraj, M.A., Albrecht, F., Cipriani, A., Vakolyuk, M., Garcia, A., Rabia, A.H., Singha, K., Marcarini, A.A., Kattenborn, T., Hazarka, R., Schepaschenko, M., van der Velde, M., Kraxner, F. & Obersteiner, M. 2015. Building a hybrid land cover map with crowdsourcing and geographically weighted regression. ISPRS Journal of Photogrammetry and Remote Sensing 103, 48–56.Google Scholar
Senaratne, H., Mobasheri, A., Ali, A. L., Capineri, C. & Haklay, M. 2017. A review of volunteered geographic information quality assessment methods. International Journal of Geographical Information Science 31(1), 139–167.Google Scholar
Shaw, A. D., Horton, J. J. & Chen, D. L. 2011. Designing incentives for inexpert human raters. In Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, 275–284. ACM.Google Scholar
Shah, N. B. & Zhou, D. 2015. Double or nothing: multiplicative incentive mechanisms for crowdsourcing. Advances in Neural Information Processing Systems 1, 1–9.Google Scholar
Shah-Mansouri, H. & Wong, V. W. S. 2015. Profit maximization in mobile crowdsourcing: a truthful auction mechanism. In2015 IEEE International Conference on Communications (ICC), 3216–3221. IEEE.Google Scholar
Shoham, Y. 2008. Computer science and game theory. Communications of the ACM 51(8), 74–79.Google Scholar
Singla, A. & Krause, A. 2013. Truthful incentives in crowdsourcing tasks using regret minimization mechanisms. In Proceedings of the 22nd International Conference on World Wide Web, 1167–1178. International World Wide Web Conferences Steering Committee.Google Scholar
Stewart, O., Lubensky, D. & Huerta, J. M. 2010. Crowdsourcing participation inequality: a SCOUT model for the enterprise domain. In Proceedings of the ACM SIGKDD Workshop on Human Computation, 30–33. ACM.Google Scholar
Tran-Thanh, L., Stein, S., Rogers, A. & Jennings, N. R. 2014. Efficient crowdsourcing of unknown experts using bounded multi-armed bandits. Artificial Intelligence 214, 89–111.Google Scholar
Tversky, A. & Kahneman, D. 1992. Advances in prospect theory: cumulative representation of uncertainty. Journal of Risk and uncertainty 5(4), 297–323.Google Scholar
Vickrey, W. 1961. Counterspeculation, auctions, and competitive sealed tenders. The Journal of finance 16(1), 8–37.Google Scholar
Von Neumann, J. & Morgenstern, O. 1944. Theorem 3.1.18, Theory of games and economic behavior, Princeton University Press.Google Scholar
Wang, L., Zhang, D., Wang, Y., Chen, C., Han, X. & M’hamed, A. 2016. Sparse mobile crowdsensing: challenges and opportunities. IEEE Communications Magazine 54(7), 161–167.Google Scholar
Yang, D., Xue, G., Fang, X. & Tang, J. 2012. Crowdsourcing to smartphones: incentive mechanism design for mobile phone sensing. In Proceedings of the 18th Annual International Conference on Mobile Computing and Networking, 173–184. ACM.Google Scholar
Yang, J., Adamic, L. A. & Ackerman, M. S. 2008. Crowdsourcing and knowledge sharing: strategic user behavior on taskcn. In Proceedings of the 9th ACM Conference on Electronic Commerce, 246–255. ACM.Google Scholar
Zhang, Y. & Van der Schaar, M. 2012. Reputation-based incentive protocols in crowdsourcing applications. In FOCOM, 2012 Proceedings IEEE, 2140–2148. IEEE.Google Scholar
Zhao, Y. & Han, Q. 2016. Spatial crowdsourcing: current state and future directions. IEEE Communications Magazine 54(7), 102–107.Google Scholar