Markus Rokicki
Sergej Zerr
ABSTRACT
Many modern data analytics applications in areas such as crisis management, stock trading, and healthcare, rely on components capable of nearly real-time processing of streaming data produced at varying rates. In addition to automatic processing methods, many tasks involved in those applications require further human assessment and analysis. However, current crowdsourcing platforms and systems do not support stream processing with variable loads. In this paper, we investigate how incentive mechanisms in competition based crowdsourcing can be employed in such scenarios. More specifically, we explore techniques for stimulating workers to dynamically adapt to both anticipated and sudden changes in data volume and processing demand, and we analyze effects such as data processing throughput, peak-to-average ratios, and saturation effects. To this end, we study a wide range of incentive schemes and utility functions inspired by real world applications. Our large-scale experimental evaluation with more than 900 participants and more than 6200 hours of work spent by crowd workers demonstrates that our competition based mechanisms are capable of adjusting the throughput of online workers and lead to substantial on-demand performance boosts.
- GamifIR '14: Proceedings of the First International Workshop on Gamification for Information Retrieval, 2014. ACM.Google Scholar
- M. S. Bernstein, J. Brandt, R. C. Miller, and D. R. Karger. Crowds in two seconds: Enabling realtime crowd-powered interfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 33--42. ACM, 2011. Google ScholarDigital Library
- A. Biem, H. Feng, A. V. Riabov, and D. S. Turaga. Real-time analysis and management of big time-series data. IBM J. Res. Dev., 57(3--4):1:8--1:8, May 2013. Google ScholarDigital Library
- J. P. Bigham, C. Jayant, H. Ji, G. Little, A. Miller, R. C. Miller, R. Miller, A. Tatarowicz, B. White, S. White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pages 333--342, 2010. ACM. Google ScholarDigital Library
- S. Dahi and S. Tabbane. Sigmoid utility function formulation for handoff reducing access model in cognitive radio. In Communications and Information Technologies (ISCIT), 2013 13th International Symposium on, pages 166--170, 2013. Google ScholarCross Ref
- A. Damasceno, R. A. Mini, J. M. Almeida, and H. Marques-Neto. A base station workload-aware dynamic pricing scheme for mobile internet access. In Proceedings of the 10th International Workshop on Mobility in the Evolving Internet Architecture, MobiArch '15, pages 45--50, 2015. ACM. Google ScholarDigital Library
- A. Das Sarma, A. Parameswaran, H. Garcia-Molina, and A. Halevy. Crowd-powered find algorithms. In Data Engineering (ICDE), 2014 IEEE 30th International Conference on, pages 964--975. 2014, IEEE. Google ScholarCross Ref
- D. E. Difallah, M. Catasta, G. Demartini, and P. Cudré-Mauroux. Scaling-up the crowd: Micro-task pricing schemes for worker retention and latency improvement. In Second AAAI Conference on Human Computation and Crowdsourcing, 2014.Google Scholar
- J. Fan, M. Zhang, S. Kok, M. Lu, and B. C. Ooi. Crowdop: Query optimization for declarative crowdsourcing systems. Knowledge and Data Engineering, IEEE Transactions on, 27(8):2078--2092, Aug 2015.Google Scholar
- S. Faradani, B. Hartmann, and P. G. Ipeirotis. What's the right price? pricing tasks for finishing on time. Human computation, 11, 2011. Google ScholarDigital Library
- O. Feyisetan, E. Simperl, M. V. Kleek, and N. Shadbolt. Improving paid microtasks through gamification and adaptive furtherance incentives. In Proceedings of the 24th International Conference on World Wide Web, WWW '15, pages 333--343, 2015. International World Wide Web Conferences Steering Committee. Google ScholarDigital Library
- Y. Gao and A. Parameswaran. Finish them!: Pricing algorithms for human computation. Proceedings of the VLDB Endowment, 7(14):1965--1976, 2014. Google ScholarDigital Library
- J. He, M. Bron, L. Azzopardi, and A. de Vries. Studying user browsing behavior through gamified search tasks. In Proceedings of the First International Workshop on Gamification for Information Retrieval, GamifIR '14, pages 49--52, 2014. ACM. Google ScholarDigital Library
- M. Imran, C. Castillo, J. Lucas, P. Meier, and S. Vieweg. Aidr: Artificial intelligence for disaster response. In Proceedings of the companion publication of the 23rd international conference on World wide web companion, pages 159--162, 2014. International World Wide Web Conferences Steering Committee. Google ScholarDigital Library
- M. Imran, C. Castillo, J. Lucas, M. Patrick, and J. Rogstadius. Coordinating human and machine intelligence to classify microblog communications in crises. Proc. of ISCRAM, 2014.Google Scholar
- M. Imran, I. Lykourentzou, Y. Naudet, and C. Castillo. Engineering crowdsourced stream processing systems. arXiv preprint arXiv:1310.5463, 2013.Google Scholar
- P. G. Ipeirotis and P. K. Paritosh. Managing crowdsourced human computation: a tutorial. In Proceedings of the 20th international conference companion on World wide web, pages 287--288. ACM, 2011. Google ScholarDigital Library
- E. D. Jensen. Asynchronous decentralized realtime computer systems. In Real Time Computing, pages 347--371. Springer, 1994. Google ScholarCross Ref
- E. D. Jensen, C. D. Locke, and H. Tokuda. A time-driven scheduling model for real-time operating systems. In RTSS, volume 85, pages 112--122, 1985.Google Scholar
- T. Johnson, S. Muthukrishnan, and I. Rozenbaum. Sampling algorithms in a stream operator. In Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data, SIGMOD '05, pages 1--12, 2005. ACM. Google ScholarDigital Library
- N. Kumar, A. C. Berg, P. N. Belhumeur, and S. K. Nayar. Attribute and Simile Classifiers for Face Verification. In IEEE International Conference on Computer Vision (ICCV), Oct 2009. Google ScholarCross Ref
- P. Li, B. Ravindran, and E. Jensen. Adaptive time-critical resource management using time/utility functions: past, present, and future. In Computer Software and Applications Conference, 2004. COMPSAC 2004. Proceedings of the 28th Annual International, volume 2, pages 12--13 vol.2, Sept 2004. Google ScholarDigital Library
- W. Mason and D. J. Watts. Financial incentives and the "performance of crowds". SIGKDD Explor. Newsl., 11(2):100--108, May 2010. Google ScholarDigital Library
- E. Minack, W. Siberski, and W. Nejdl. Incremental diversification for very large sets: A streaming-based approach. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '11, pages 585--594, 2011. ACM. Google ScholarDigital Library
- A.-H. Mohsenian-Rad and A. Leon-Garcia. Optimal residential load control with price prediction in real-time electricity pricing environments. Smart Grid, IEEE Transactions on, 1(2):120--133, Sept 2010.Google Scholar
- J. Noronha, E. Hysen, H. Zhang, and K. Z. Gajos. Platemate: crowdsourcing nutritional analysis from food photographs. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 1--12. 2011, ACM. Google ScholarDigital Library
- C. Perlich, F. Provost, and J. S. Simonoff. Tree induction vs. logistic regression: A learning-curve analysis. The Journal of Machine Learning Research, 4:211--255, 2003. Google ScholarDigital Library
- D. Pothineni, P. Mishra, A. Rasheed, and D. Sundararajan. Incentive design to mould online behavior: A game mechanics perspective. In Proceedings of the First International Workshop on Gamification for Information Retrieval, GamifIR '14, pages 27--32, 2014. ACM. Google ScholarDigital Library
- S. Roche, E. Propeck-Zimmermann, and B. Mericskay. Geoweb and crisis management: issues and perspectives of volunteered geographic information. GeoJournal, 78(1):21--40, 2013. Google ScholarCross Ref
- M. Rokicki, S. Chelaru, S. Zerr, and S. Siersdorfer. Competitive game designs for improving the cost effectiveness of crowdsourcing. In Proceedings of the 23rd ACM International Conference on Information and Knowledge Management, CIKM '14, pages 1469--1478, 2014. ACM. Google ScholarDigital Library
- M. Rokicki, S. Zerr, and S. Siersdorfer. Groupsourcing: Team competition designs for crowdsourcing. In Proceedings of the 24th International Conference on World Wide Web, WWW '15, pages 906--915, 2015.International World Wide Web Conferences Steering Committee. Google ScholarDigital Library
- P. Samadi, A.-H. Mohsenian-Rad, R. Schober, V. Wong, and J. Jatskevich. Optimal real-time pricing algorithm based on utility maximization for smart grid. In Smart Grid Communications (SmartGridComm), 2010 First IEEE International Conference on, pages 415--420, Oct 2010. Google ScholarCross Ref
- D. R. Stockwell and A. T. Peterson. Effects of sample size on accuracy of species distribution models. Ecological modelling, 148(1):1--13, 2002. Google ScholarCross Ref
- J. Wang and B. Ravindran. Time-utility function-driven switched ethernet: Packet scheduling algorithm, implementation, and feasibility analysis. Parallel and Distributed Systems, IEEE Transactions on, 15(2):119--133, 2004. Google ScholarDigital Library
- Y. Wang, J.-G. Kim, and S.-F. Chang. Content-based utility function prediction for real-time mpeg-4 video transcoding. In Image Processing, 2003. ICIP 2003. Proceedings. 2003 International Conference on, volume 1, pages I--189--92 vol.1, Sept 2003. Google ScholarCross Ref
- %B. L. Welch.% The significance of the difference between two means when the% population variances are unequal.% Biometrika, 29(3--4):350--362, 1938.Google Scholar
- M. Xiao, N. B. Shroff, and E. K. Chong. A utility-baed power-control scheme in wireless cellular systems. Networking, IEEE/ACM Transactions on, 11(2):210--221, 2003. Google ScholarDigital Library
- D. Yang, E. A. Rundensteiner, and M. O. Ward. Summarization and matching of density-based clusters in streaming environments. Proc. VLDB Endow., 5(2):121--132, Oct. 2011. Google ScholarDigital Library
- S. Yousefi, M. P. Moghaddam, and V. J. Majd. Optimal real time pricing in an agent-based retail market using a comprehensive demand response model. Energy, 36(9):5716--5727, 2011. Google ScholarCross Ref
Index Terms
- Just in Time: Controlling Temporal Performance in Crowdsourcing Competitions
Recommendations
Competitive Game Designs for Improving the Cost Effectiveness of Crowdsourcing
CIKM '14: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge ManagementCrowd based online work is leveraged in a variety of applications such as semantic annotation of images, translation of texts in foreign languages, and labeling of training data for machine learning models. However, annotating large amounts of data ...
Groupsourcing: Team Competition Designs for Crowdsourcing
WWW '15: Proceedings of the 24th International Conference on World Wide WebMany data processing tasks such as semantic annotation of images, translation of texts in foreign languages, and labeling of training data for machine learning models require human input, and, on a large scale, can only be accurately solved using crowd ...
Just the Right Mood for HIT!: Analyzing the Role of Worker Moods in Conversational Microtask Crowdsourcing
Web EngineeringAbstractConversational agents are playing an increasingly important role in providing users with natural communication environments, improving outcomes in a variety of domains in human-computer interaction. Crowdsourcing marketplaces are simultaneously ...
Comments