Jiuchuan Jiang
Bo An
Skip Abstract Section
Abstract
Crowdsourcing has recently been significantly explored. Although related surveys have been conducted regarding this subject, each has mainly consisted of a review of a single aspect of crowdsourcing systems or on the application of crowdsourcing in a specific application domain. A crowdsourcing system is a comprehensive set of multiple entities, including various elements and processes. Multiagent computing has already been widely envisioned as a powerful paradigm for modeling autonomous multi-entity systems with adaptation to dynamic environments. Therefore, this article presents a novel multiagent perspective and approach to understanding crowdsourcing systems, which can be used to correlate the research on crowdsourcing and multiagent systems and inspire possible interdisciplinary research between the two areas. This article mainly discusses the following two aspects: (1) The multiagent perspective can be used for conducting a comprehensive survey on the state of the art of crowdsourcing, and (2) the multiagent approach can bring about concrete enhancements for crowdsourcing technology and inspire future research directions that enable crowdsourcing research to overcome the typical challenges in crowdsourcing technology. Finally, this article discusses the advantages and disadvantages of the multiagent perspective by comparing it with two other popular perspectives on crowdsourcing: the business perspective and the technical perspective.
- J. Howe. 2006. The Rise of Crowdsourcing. Wired Magazine, 14, 6 (2006), 1--4.Google Scholar
- Long Tran-Thanh, Trung Dong Huynh, Avi Rosenfeld, Sarvapali Ramchurn, and Nicholas R. Jennings. 2014. BudgetFix: Budget limited crowdsourcing for interdependent task allocation with quality guarantees. In Proceedings of the 13th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’14). 477--484. Google ScholarDigital Library
- T. Kohler and M. Nickel. 2017. Crowdsourcing business models that last. J. Bus. Strategy 38, 2 (2017), 25--32.Google ScholarCross Ref
- M. C. Yuen, I. King, and K. S. Leung. 2011. A survey of crowdsourcing systems. In Proceedings of the IEEE 3rd International Conference on Social Computing (SocialCom’11). 766--773.Google Scholar
- Q. Li, A. Vempaty, L. R. Varshney, and P. K. Varshney. 2017. Multi-object classification via crowdsourcing with a reject option. IEEE Trans. Sign. Process. 65, 4 (2017), 1068--1081. Google ScholarDigital Library
- Nuno Luz, Nuno Silva, and Paulo Novais. 2015. A survey of task-oriented crowdsourcing. Artif. Intell. Rev. 44, 2 (2015), 187--213. Google ScholarDigital Library
- Robbie T. Nakatsu, Elissa B. Grossman, and Charalambos L. Lacovou. 2014. A taxonomy of crowdsourcing based on task complexity. J. Inf. Sci. 40, 6 (2014), 823--834. Google ScholarDigital Library
- X. Guo, H. Wang, Y. Song, and H. Gao. 2014. Brief survey of crowdsourcing for data mining. Expert Syst. Appl. 41, 17 (2014), 7987--7994. Google ScholarDigital Library
- A. Doan, R. Ramakrishnan, and A. Y. Halevy. 2011. Crowdsourcing systems on the world-wide web. Commun. ACM 54, 4 (2011), 86--96. Google ScholarDigital Library
- S. Badihi and A. Heydarnoori. 2017. CrowdSummarizer: Automated generation of code summaries for java programs through crowdsourcing. IEEE Softw. 34, 2 (2017), 71--80. Google ScholarDigital Library
- G. Zou, A. Gil, and M. Tharayil. 2014. An agent-based model for crowdsourcing systems. In Proceedings of the 2014 Winter Simulation Conference (WSC-14). 407--418. Google ScholarDigital Library
- L. Hetmank. 2013. Components and functions of crowdsourcing systems-a systematic literature review. Retrieved from http://aisel.aisnet.org/wi2013/4.Google Scholar
- G. Goel, A. Nikzad, and A. Singla. 2014. Mechanism design for crowdsourcing markets with heterogeneous tasks. In Proceedings of the 2nd AAAI Conference on Human Computation and Crowdsourcing (HCOMP’14), 77--86, Pittsburgh, Pennsylvania, USA, November 2-4, 2014.Google Scholar
- K. Osipov and G. Sukthankar. 2012. AmalgaCloud: Social network adaptation for humana and computational agent team formation. In Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering (ASE’12). 61--73.Google Scholar
- D. Yang, G. Xue, X. Fang, and J. Tang. 2012. Crowdsourcing to smartphones; Incentive mechanism design for mobile phone sensing. In Proceedings of the 18th Annual International Conference on Mobile Computing and Networking (MobiCom’12). 173--184. Google ScholarDigital Library
- P. E. Petruzzi, J. Pitt, and D. Busquets. 2017. Electronic social capital for self-organising muti-agent systems. ACM Trans. Auton. Adapt. Syst. 12, 3 (2017), Article 3. Google ScholarDigital Library
- B. P. Gerkey and M. J. Mataric. 2004. A formal analysis and taxonomy of task allocation in multi-robot systems. Int. J. Robot. Res. 23, 9 (2004), 939--954.Google ScholarCross Ref
- Ross Conroy, Yifeng Zeng, Marc Cavazza, and Yingke Chen. 2015. Learning behaviors in agents systems with interactive dynamic influence diagrams. In Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI’15). 39--45. Google ScholarDigital Library
- Y. Jiang, Y. Zhou, and W. Wang. 2013. Task allocation for undependable multiagent systems in social networks. IEEE Trans. Parallel Distrib. Syst. 24, 8 (2013), 1671--1681. Google ScholarDigital Library
- B. T. R. Savarimuthu, M. Purvis, and M. Fleurke. 2004. Monitoring and controlling of a multi-agent based workflow system. In Proceedings of the 2nd Workshop on Australasian Information Security, Data Mining and Web Intelligence, and Software Internationalisation (ACSW Frontiers’04). 127--132. Google ScholarDigital Library
- Y. Jiang. 2016. A survey of task allocation and load balancing in distributed systems. IEEE Transactions on Parallel and Distributed Systems 27, 2, 585--599, 2016. Google ScholarDigital Library
- R. Zlot and A. Stentz. 2006. Market-based multirobot coordination for complex tasks. Int. J. Robot. Res. 25, 1 (2006), 73--101.Google ScholarCross Ref
- Long Tran-Thanh, Sebastian Stein, Alex Rogers, and Nicholas R. Jennings. 2014. Efficient crowdsourcing of unknown experts using multi-armed bandits. Artif. Intell. 214 (2014), 89--111. Google ScholarDigital Library
- J. T. Jacques and P. O. Kristensson. 2013. Crowdsourcing a HIT: Measuring workers’ pre-task interactions on microtask markets. In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing (HCOMP’13). 86--93.Google Scholar
- Long Tran-Thanh, M. Venanzi, A. Rogers, and N. R. Jennings. 2013. Efficient budget allocation with accuracy guarantees for crowdsourcing classification tasks. In Proceedings of the 12th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’13). 901--908. Google ScholarDigital Library
- A. Kittur, B. Smus, and R. Kraut. 2011. Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST’11). 43--52. Google ScholarDigital Library
- H. Jiang and S. Matsubara. 2014. Efficient task decomposition in crowdsourcing. In Proceedings of the 17th International Conference on Principles and Practice of Multi-Agent Systems (PRIMA’14), LNAI Vol. 8861. 65--73.Google Scholar
- Victor R. Lesser. 1999. Cooperative multiagent systems: A personal view of the state of the art. IEEE Trans. Knowl. Data Eng. 11, 1 (1999), 133--142. Google ScholarDigital Library
- J. Lee and L. F. Lai. 2002. A high-level petri nets-based approach to verifying task structures. IEEE Trans. Knowl. Data Eng. 14, 2 (2002), 316--335. Google ScholarDigital Library
- Y. Jiang, J. Hu, and D. Lin. 2011. Decision making of networked multiagent systems for interaction structures. IEEE Trans. Syst. Man Cybernet. A 41, 6 (2011), 1107--1121. Google ScholarDigital Library
- A. Garvey, M. Humphrey, and V. R. Lesser. 1993. Task interdependencies in design-to-time real-time scheduling. In Proceedings of the 11th National Conference on Artificial Intelligence (AAAI’93). 580--585. Google ScholarDigital Library
- J. Yu and R. Buyya. 2006. A taxonomy of workflow management systems for grid computing. J. Grid Comput. 3, 3 (2006), 171--200.Google ScholarCross Ref
- Peng Dai, Christopher H. Lin, and Daniel S. Weld. 2013. POMDP-based control of workflows for crowdsourcing. Artif. Intell. 202 (2013), 52--85. Google ScholarDigital Library
- P. Tichy, P. Slechta, R. J. Staron, F. P. Maturana, and K. H. Hall. 2006. Multiagent technology for fault tolerance and flexible control. IEEE Trans. Syst. Man Cybernet. C 36, 5 (2006), 700--704. Google ScholarDigital Library
- Gagan Goel, Afshin Nikzad, and Adish Singla. 2014. Allocating tasks to workers with matching constraints: Truthful mechanisms for crowdsourcing markets. In Proceedings of the 23rd International Conference on World Wide Web Companion (WWW’14). 279--280. Google ScholarDigital Library
- Sarvapali D. Ramchurn, Claudio Mezzetti, Andrea Giovannucii, Juan A. Rodriguez-Aguilar, Rajdeep K. Dash, and Nicholas R. Jennings. 2009. Trust-based mechanisms for robust and efficient task allocation in the presence of execution un-certainty. J. Artif. Intell. Res. 35, 1 (2009), 119--159. Google ScholarDigital Library
- T. D. Huynh, N. R. Jennings, and N. R. Shadbolt. 2006. An integrated trust and reputation model for open multi-agent systems. Auton. Agents Multi-Agent Syst. 13, 2 (2006), 119--154. Google ScholarDigital Library
- Ruggiero Cavallo, David C. Parkes, and Satinder Singh. 2012. Optimal coordinated planning amongst self-interested agents with private state. arXiv:12066820 {cs.GT}.Google Scholar
- David R. Karger, Sewoong Oh, and Devavrat Shah. 2014. Budget-optimal task allocation for reliable crowdsourcing systems. Operat. Res. 62, 1 (2014), 1--24. Google ScholarDigital Library
- L. Mo et al. 2013. Optimizing plurality for human intelligence tasks. In Proceedings of the 22nd ACM International Conference on Conference on Information & Knowledge Management (CIKM’13). 1929--1938. Google ScholarDigital Library
- Han Yu, Zhiqi Shen, Chunyan Miao, and Bo An. 2013. A reputation-aware decision-making approach for improving the efficiency of crowdsourcing systems. In Proceedings of the 12th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’13). 1315--1316. Google ScholarDigital Library
- M. Venanzi, Alex Rogers, and Nicholas R. Jennings. 2013. Trust-based fusion of untrustworthy information in crowdsourcing applications. In Proceedings of the 12th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’13). 829--836. Google ScholarDigital Library
- Yu Zhang and Mihaela van derSchaar. 2012. Reputation-based incentive protocols in crowdsourcing applications. In Proceedings of the 31st Annual IEEE International Conference on Computer Communications (INFOCOM’12). 2140--2148.Google Scholar
- G. Radanovic and R. Faltings. 2015. Incentive schemes for participatory sensing. In Proceedings of the 14th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’15). 1081--1089. Google ScholarDigital Library
- D. Zhao, X. Y. Li, and H. Ma. 2014. How to crowdsource tasks truthfully without sacrificing utility: Online incentive mechanism with budget constraint. In Proceedings of the 33rd Annual IEEE International Conference on Computer Communications (INFOCOM’14). 1213--1221.Google Scholar
- Anand Kulkarni, Matthew Can, and Björn Hartmann. 2012. Collaboratively crowdsourcing workflows with turkomatic. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW’12). 1003--1012. Google ScholarDigital Library
- H. Rahman, S. B. Roy, S. Thirumuruganathan, S. Amer-Yahia, and G. Das. 2015. Task assignment optimization in collaborative crowdsourcing. In Proceedings of the 2015 IEEE International Conference on Data Mining (ICDM’15). 949--954. Google ScholarDigital Library
- A. Kobren and C. H. Tan. 2015. Getting more for less: Optimized crowdsourcing with dynamic tasks and goals. In Proceedings of the 24th International World Wide Web Conference (WWW’15). 592--602. Google ScholarDigital Library
- C. J. Ho and J. W. Vaughan. 2012. Online task assignment in crowdsourcing markets. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI’12). 45--51. Google ScholarDigital Library
- Bergenti Federico, Enrico Franchi, and Agostino Poggi. 2013. Agent-based interpretations of classic network models. Comput. Math. Organiz. Theory 19, 2 (2013), 105--127. Google ScholarDigital Library
- Jiming Liu, Xiaolong Jin, and Yuanshi Wang. 2005. Agent-based load balancing on homogeneous minigrids: Macroscopic modeling and characterization. IEEE Trans. Parallel Distrib. Syst. 16, 7 (2005), 586--598. Google ScholarDigital Library
- F. Daniel, P. Kucherbaev, C. Cappiello, B. Benatallah, and M. Allahbakhsh. 2018. Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions. ACM Comput. Surv. 51, 1 (2018), Article 7. Google ScholarDigital Library
- A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, and J. Aru. 2017. Multiagent cooperation and competition with deep reinforcement learning. PLoS ONE 12, 4 (2017), e0172395.Google ScholarCross Ref
- Y. Jiang, Yifeng Zhou, and Yunpeng Li. 2015. Reliable task allocation with load balancing in multiplex networks. ACM Trans. Auton. Adapt. Syst. 10, 1 (2015), Article 3. Google ScholarDigital Library
- Dayong Ye, Minjie Zhang, and Athanasios V. Vasilakos. 2017. A survey of self-organization mechanisms in multi-agent systems. IEEE Trans. Syst. Man Cybernet. Syst. 47, 3 (2017), 441--461.Google ScholarCross Ref
- P. Xiong, L. Zhang, and T. Zhu. 2017. Reward-based spatial crowdsourcing with differential privacy preservation. Enterp. Inf. Syst. 11, 10 (2017), 1500--1517. Google ScholarDigital Library
- F. Zambonelli. 2011. Pervasive urban crowdsourcing: Visions and challenges. In Proceedings of the 7th IEEE International Workshop on Pervasive Learning, Life, and Lei-sure (PerEL’11). 578--583.Google ScholarCross Ref
- Retrieved from https://answers.yahoo.com/.Google Scholar
- Lingjie Duan, Kubo T. Sugiyama, and K. Jianwei Huang. 2012. Incentive mechanisms for smartphone collaboration in data acquisition and distributed computing. In Proceedings of the 2012 IEEE International Conference on Computer Communications (INFOCOM’12). 1701--1709.Google Scholar
- A. Biswas, D. Chander, K. Dasgupta, K. Mukherjee, M. Singh, and T. Mukherjee. 2015. PISCES: Participatory incentive strategies for effective community engagement in smart cities. In Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing (HCOMP’15). 23--31.Google Scholar
- Y. Pan and E. Blevis. 2011. A survey of crowdsourcing as a means of collaboration and the implications of crowdsourcing for interaction design. In Proceedings of the 2011 International Conference on Collaboration Technologies and Systems (CTS’11). 397--403.Google Scholar
- David R. Karger, Sewoong Oh, and Devavrat Shah. 2011. Iterative learning for reliable crowdsourcing systems. In Proceedings of the 25th Annual Conference on Neural Information Processing Systems (NIPS’11). 1953--1961. Google ScholarDigital Library
- L. G. Jaimes, I. Vergara-Laurens, and M. A. Labrador. 2012. A location-based incentive mechanism for participatory sensing systems with budget constraints. In Proceedings of the 2012 IEEE International Conference on Pervasive Computing and Communications (PerCom’12). 103--108.Google Scholar
- Donghui Lin and Toru Ishida. 2014. Coordination of local process views in interorganizational business process. IEICE Trans. Inf. Syst. E97-D, 5 (2014), 1119--1126.Google ScholarCross Ref
- Danny Weyns, Robrecht Haesevoets, and Alexander Helleboogh. 2010. The MACODO middleware for context-driven dynamic agent organizations. ACM Trans. Auton. Adapt. Syst. 5, 1 (2010), Article 3. Google ScholarDigital Library
- G. Weiss. 1999. Multiagent Systems: A Modern Approach to Distributed Artificial Intelligence. The MIT Press, Cambridge, MA, 1999. Google ScholarDigital Library
- Reza Olfati-Saber. 2006. Flocking for multi-agent dynamic systems: Algorithms and theory. IEEE Trans. Autom. Contr. 51, 3 (2006), 401--420.Google ScholarCross Ref
- Onn Shehory and Sarit Kraus. 1998. Methods for task allocation via agent coalition formation. Artif. Intell. 101, 1--2 (1998), 165--200. Google ScholarDigital Library
- S. D. Ramchurn, D. Huynh, and N. R. Jennings. 2004. Trust in multi-agent systems. Knowl. Eng. Rev. 19, 1 (2004), 1--25. Google ScholarDigital Library
- Alessandro Bozzon, Marco Brambilla, Stefano Ceri, Matteo Silvestri, and Giuliano Vesci. 2013. Choosing the Right crowd: Expert finding in social networks. In Proceedings of the 16th International Conference on Extending Database Technology (EDB’13). 637--648. Google ScholarDigital Library
- A. Kulkarni, P. Narula, D. Rolnitzky, and N. Kontny. 2014. Wish: Amplifying creative ability with expert crowds. In Proceedings of the 2nd AAAI Conference on Human Computation and Crowdsourcing (HCOMP’14). 112--120.Google Scholar
- Yukino Baba, Hisashi Kashima, Kei Kinoshita, Goushi Yamaguchi, and Yosuke Akiyoshi. 2014. Leveraging non-expert crowdsourcing workers for improper task detection in crowdsourcing marketplaces. Expert Syst. Appl. 41, 6 (2014), 2678--2687. Google ScholarDigital Library
- Theodoros Lappas, Kun Liu, and Evimaria Terzi. 2009. Finding a team of experts in social networks. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’09). 127--132. Google ScholarDigital Library
- Anirban Majumder, Samik Datta, and K. V. M. Naidu. 2012. Capacitated team formation problem on social networks. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’12). 1005--1013. Google ScholarDigital Library
- S. Jain, B. Narayanaswamy, and Y. Narahari. 2014. A multiarmed. bandit incentive mechanism for crowdsourcing demand response in smart grids. In Proceedings of the 28th AAAI Conference on Artificial Intelligence (AAAI’14). 721--727. Google ScholarDigital Library
- W. Hu, L. Liu, and G. Feng. 2017. Output consensus of heterogeneous linear multi-agent systems by distributed event-triggered/self-triggered strategy. IEEE Trans. Cybernet. 47, 8 (2017), 1914--1924.Google ScholarCross Ref
- S. Assadi, J. Hsu, and S. Jabbari. 2015. Online assignment of heterogeneous tasks in crowdsourcing markets. In Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing (HCOMP’15). 2--11.Google Scholar
- M. S. Bernstein, G. Little, R. C. Miller, B. Hartmann, M. S. Ackerman, D. R. Karger, D. Crowell, and K. Panovich. 2015. Soylent: A word processor with a crowd in-side. Commun. ACM 58, 8 (2015), 313--322. Google ScholarDigital Library
- J. Ferber and O. Gutknecht. 1998. A meta-model for the analysis and design of organizations in multi-agent systems. In Proceedings of the 3rd International Conference on Multi Agent Systems (ICMAS’98). 128--135. Google ScholarDigital Library
- Aniket Kittur, Ed H. Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI 2008 Conference on Human Factors in Computing Systems (CHI’08). 453--456. Google ScholarDigital Library
- J. Chamberlain. 2014. Groupsourcing: Distributed problem solving using social networks. In Proceedings of the 2nd AAAI Conference on Human Computation and Crowdsourcing (HCOMP’14). 22--29.Google Scholar
- Y. Jiang and J. C. Jiang. 2014. Understanding social networks from a multiagent perspective. IEEE Trans. Parallel Distrib. Syst. 25, 10 (2014), 2743--2759.Google ScholarCross Ref
- Y. Jiang and Z. Huang. 2012. The rich get richer: Preferential attachment in the task allocation of cooperative networked multiagent systems with resource caching. IEEE Trans. Syst. Man Cybernet. A 42, 5 (2012), 1040--1052. Google ScholarDigital Library
- Qing Liu, Tie Luo, Ruiming Tang, and Stéphane Bressan. 2015. An efficient and truthful pricing mechanism for team formation in crowdsourcing markets. In Proceedings of the 2015 IEEE International Conference on Communications (ICC’15). 567--572.Google ScholarCross Ref
- Mehdi Kargar, Aijun An, and Morteza Zihayat. 2012. Efficient bi-objective team formation in social networks. In Proceedings of the 2012 European Conference on Ma-chine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD’12), LNCS Vol. 7524. 483--498.Google ScholarCross Ref
- Markus Rokicki, Sergej Zerr, and Stefan Siersdorfer. 2015. Groupsourcing: Team competition designs for crowdsourcing. In Proceedings of the 24th International Conference on World Wide Web (WWW’15). 906--915. Google ScholarDigital Library
- A. Singla et al. 2014. Building collaborative teams: A local navigation-based approach. Retrieved from http://research.microsoft.com/en-us/um/people/ryenw/papers/SinglaNIPS2014.pdf.Google Scholar
- Ece Kamar and Eric Horvitz. 2012. Incentives for truthful reporting in crowdsourcing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS’12). 1329--1330. Google ScholarDigital Library
- W. Feng, Z. Yan, H. Zhang, K. Zeng, Y. Xiao, and Y. T. Hou. 2018. A survey on security, privacy and trust in mobile crowdsourcing. IEEE IoT Journal, 2018.Google Scholar
- Vamshi Ambati, Stephan Vogel, and Jaime Carbonell. 2012. Collaborative workflow for crowdsourcing translation. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW’12). 1191--1194. Google ScholarDigital Library
- Jiang Yang, Lada A. Adamic, and Mark S. Ackerman. 2008. Crowdsourcing and knowledge sharing: Strategic user behavior on taskcn. In Proceedings of the 9th ACM Conference on Electronic Commerce (EC’08). 246--255. Google ScholarDigital Library
- Aniket Kittur, Susheel Khamkar, Paul André, and Robert E. Kraut. 2012. CrowdWeaver: Visually managing complex crowd work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW’12). 1033--1036. Google ScholarDigital Library
- Peter Stone and Manuela Veloso. 1999. Task decomposition, dynamic role assignment, and low-bandwidth communication for real-time strategic teamwork. Artif. Intell. 110, 2 (1999), 241--273. Google ScholarDigital Library
- Guido Sautter and Klemens Böhm. 2013. High-throughput crowdsourcing mechanisms for complex tasks. Soc. Netw. Anal. Min. 3, 4 (2013), 873--888.Google ScholarCross Ref
- Y. Jiang and J. Jiang. 2009. Contextual resource negotiation-based task allocation and load balancing in complex software systems. IEEE Trans. Parallel Distrib. Syst. 20, 5 (2009), 641--653. Google ScholarDigital Library
- Shibo He, Dong-Hoon Shin, Junshan Zhang, and Jiming Chen. 2014. Toward optimal allocation of location dependent tasks in crowdsensing. In Proceedings of the 33rd Annual IEEE International Conference on Computer Communications (INFOCOM’14). 745--753.Google ScholarCross Ref
- Cen Chen, Shih-Fen Cheng, Archan Misra, and Hoong Chuin Lau. 2015. Multi-agent task assignment for mobile crowdsourcing under trajectory uncertainties. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems (AAMAS’15). 1715--1716. Google ScholarDigital Library
- Blazej Bulka, Matthew Gaston, and Marie desJardins. 2007. Local strategy learning in networked multi-agent team formation. Auton. Agents Multi-Agent Syst. 15, 1 (2007), 29--45. Google ScholarDigital Library
- Alessandro Bozzon, Ilio Catallo, Eleonora Ciceri, Piero Fraternali, Davide Martinenghi, and Marco Tagliasacchi. 2012. A framework for crowdsourced multimedia processing and querying. In Proceedings of the 21st International Conference on World Wide Web on Workshop on CrowdSearch (WWW’12). 42--47.Google Scholar
- Panos Ipeirotis, Greg Little, and Thomas W. Malone. 2014. Composing and analyzing crowdsourcing workflows. Retrieved from http://www.ipeirotis.com/wp-content/uploads/2014/03/main.pdf.Google Scholar
- Christopher H. Lin, Daniel S. Weld, and others. 2012. Dynamically switching between synergistic workflows for crowdsourcing. In Proceedings of the 26th AAAI Conference on Artificial Intelligence (AAAI’12). 87--93. Google ScholarDigital Library
- Cheong Ha Park, KyoungHee Son, Joon Hyub Lee, and Seok-Hyung. 2013. Crowd vs. Crowd: Large-scale cooperative design through open team competition. In Proceedings of the 16th ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW’13). Google ScholarDigital Library
- Luca de Alfaro, V. Polychronopoulos, and M. Shavlovsky. 2015. Reliable aggregation of boolean crowdsourced tasks. In Proceedings of the 3rd AAAI Conference on Hu-man Computation and Crowdsourcing (HCOMP’15). 42--51.Google Scholar
- D. Yue, G. Yu, D. Shen, and X. Yu. 2014. A weighted aggregation rule in crowdsourcing systems for high result accuracy. In Proceeding of the 12th International Conference on Dependable, Autonomic and Secure Computing (DASC’14). 265--270. Google ScholarDigital Library
- Mihai Georgescu and Xiaofei Zhu. 2014. Aggregation of crowdsourced labels based on worker history. In Proceedings of the 4th International Conference on Web Intelligence, Mining and Semantics (WIMS’14). Google ScholarDigital Library
- T. Matsui, Y. Baba, T. Kamishima, and H. Kashima. 2013. Crowdsourcing quality control for item ordering tasks. Human Computation and Crowdsourcing: Works in Progress and Demonstration Abstracts, AAAI Technical Report CR-13-01, pp. 52--53.Google Scholar
- J. Hui, A. Glenn, R. Jue, E. Gerber, and S. Dow. 2015. Using anonymity and communal efforts to improve quality of crowdsourced feedback. In Proceedings of the 3rd AAAI Conference on Human Computation and Crowdsourcing (HCOMP’15). 72--81.Google Scholar
- Ming Yin and Yiling Chen. 2015. Bonus or not? learn to reward in crowdsourcing. In Proceedings of the 24th International Joint Conference on Artificial Intelligence (IJCAI’15). 201--207. Google ScholarDigital Library
- D. Dipalantino and M. Vojnovic. 2009. Crowdsourcing and all-pay auctions. In Proceedings of the 10th ACM Conference on Electronic Commerce (EC’09). 119--128. Google ScholarDigital Library
- Benjamin Satzger, Harald Psaier, Daniel Schall, and Schahram Dustdar. 2013. Auction-based crowdsourcing supporting skill management. Inf. Syst. 38, 4 (2013), 547--560. Google ScholarDigital Library
- Maria Daltayanni, Luca de Alfaro, and Panagiotis Papadimitriou. 2015. WorkerRank: Using employer implicit judgments to infer worker reputation. In Proceedings of the 8th ACM International Conference on Web Search and Data Mining (WSDM’15). 263--272. Google ScholarDigital Library
- M. Kokkodis and P. G. Ipeirotis. 2013. Have you done anything like that? Predicting performance using inter-category reputation. In Proceedings of the 6th ACM Inter-national Conference on Web Search and Data Mining (WSDM’13). 435--444. Google ScholarDigital Library
- D. R. Karger, S. Oh, and D. Shah. 2011. Budget-optimal crowdsourcing using low-rank matrix approximations. In Proceedings of the 49th Annual Allerton Conference on Communication, Control, and Computing (Allerton’11). 284--291.Google Scholar
- R. K. Ganti, F. Ye, and H. Lei. 2011. Mobile crowdsensing: Current stage and future challenges. IEEE Commun. Mag. 49, 11 (2011), 32--39.Google ScholarCross Ref
- A. Anagnostopoulos, L. Becchetti, A. Fazzone I. Mele, and M. Riondato. 2015. The importance of being expert: Efficient max-fining in crowdsourcing. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (SIGMOD’15). 983--998. Google ScholarDigital Library
- J. G. T. Zañudo, G. Yang, and R. Albert. 2017. Structure-based control of complex networks with nonlinear dynamics. Proc. Natl. Acad. Sci. U.S.A. 114, 28 (2017), 7234--7239.Google ScholarCross Ref
- M. B. Dias, R. Zlot, N. Kalra, and A. Stentz. 2006. Market-based multirobot coordination: A survey and analysis. Proc. IEEE 94, 7 (2006), 1257--1270.Google ScholarCross Ref
- C. Sahin et al. 2017. A multi-agent based approach to dynamic scheduling with flexible processing capabilities. J. Intell. Manufactur. 28, 8 (2017), 1827--1845. Google ScholarDigital Library
- S. B. Roy, I. Lykourentzou, S. Thirumuruganathan, S. Amer-Yahia, and G. Das. 2015. Crowds, not drones: Modeling human factors in interactive crowdsourcing. Retrieved September 21, 2015 from https://hal.inria.fr/hal-00923542/.Google Scholar
- G. Meng and D. V. Dimarogonas. 2017. Task and motion coordination for heterogeneous multiagent systems with loosely coupled local tasks. IEEE Trans. Autom. Sci. Eng. 14, 2 (2017), 797--808.Google ScholarCross Ref
- D. Grosu and A. T. Chronopoulos. 2004. Algorithmic mechanism design for load balancing in distributed systems. IEEE Trans. Syst. Man Cybernet. B 34, 1 (2004), 77--84. Google ScholarDigital Library
- A. Kittur, J. V. Nickerson, E. Gerber, A. Shaw, J. Zimmerman, M. Lease, and J. Horton. 2013. The future of crowd work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW’13). 1301--1318. Google ScholarDigital Library
- A. J. Quinn and B. B. Bederson. 2011. Human computation: A survey and taxonomy of a growing field. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’11). 1403--1402. Google ScholarDigital Library
- K. Mao, L. Capra, M. Harman, and Y. Jia. 2017. A survey of the use of crowdsourcing in software engineering. J. Syst. Softw. 126 (2017), 57--84.Google ScholarCross Ref
- J. Jiang, B. An, Y. Jiang, and D. Lin. 2017. Context-aware reliable crowdsourcing in social networks. (unpublished).Google Scholar
Index Terms
- Understanding Crowdsourcing Systems from a Multiagent Perspective and Approach
Recommendations
Incentivizing Sequential Crowdsourcing Systems
AAMAS '23: Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent SystemsA crowdsourcing system such as Amazon's Mechanical Turk allows a crowdsource campaign initiator to recruit a large number of workers to accomplish a task. The proper design of such a crowdsourcing system becomes very challenging when the task involves ...
Organizing Multiagent Systems
Despite all the research done in the last years on the development of methodologies for designing MAS, there is no methodology suitable for the specification and design of MAS in complex domains where both the agent view and the organizational view can ...
Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing SystemsCrowdsourcing is increasingly being used as a means to tackle problems requiring human intelligence. With the ever-growing worker base that aims to complete microtasks on crowdsourcing platforms in exchange for financial gains, there is a need for ...
Comments