Aniket Kittur
Susheel Khamkar
Paul André
Robert Kraut
ABSTRACT
Though toolkits exist to create complex crowdsourced workflows, there is limited support for management of those workflows. Managing crowd workers and tasks requires significant iteration and experimentation on task instructions, rewards, and flows. We present CrowdWeaver, a system to visually manage complex crowd work. The system supports the creation and reuse of crowdsourcing and computational tasks into integrated task flows, manages the flow of data between tasks, and allows tracking and notification of task progress, with support for real-time modification. We describe the system and demonstrate its utility through case studies and user feedback.
Supplemental Material
- Faridani, S., Hartmann, B., & Ipeirotis, P. G. What's the Right Price? Pricing Tasks for Finishing on Time. Proc. HCOMP 2011.Google Scholar
- Johnston, W. M., Hanna, J. R., & Millar, R. J. Advances in dataflow programming languages. ACM Computing Surveys (CSUR) 36, 1 (2004), 1--34. Google ScholarDigital Library
- Kittur, A., Chi, E., & Suh, B. Crowdsourcing user studies with Mechanical Turk. Proc. CHI 2008. Google ScholarDigital Library
- Kittur, A., Smus, B., Khamkar, S. & Kraut, R. E. CrowdForge: Crowdsourcing complex work. Proc. UIST 2011. Google ScholarDigital Library
- Kulkarni, A., Can, M., & Hartmann, B. Collaboratively Crowdsourcing Workflows with Turkomatic. Proc. CSCW 2012. Google ScholarDigital Library
- Little, G., Chilton, L. B., Goldman, M., & Miller, R. C. Turkit: Human computation algorithms on Mechanical Turk. Proc. UIST 2010, 57--66. Google ScholarDigital Library
- Yu, J., Benatallah, B., Casati, F., & Daniel, F. Understanding mashup development. IEEE Internet Computing, (2008), 44-?52. Google ScholarDigital Library
Index Terms
- CrowdWeaver: visually managing complex crowd work
Recommendations
CrowdForge: crowdsourcing complex work
UIST '11: Proceedings of the 24th annual ACM symposium on User interface software and technologyMicro-task markets such as Amazon's Mechanical Turk represent a new paradigm for accomplishing work, in which employers can tap into a large population of workers around the globe to accomplish tasks in a fraction of the time and money of more ...
CrowdScape: interactively visualizing user behavior and output
UIST '12: Proceedings of the 25th annual ACM symposium on User interface software and technologyCrowdsourcing has become a powerful paradigm for accomplishing work quickly and at scale, but involves significant challenges in quality control. Researchers have developed algorithmic quality control approaches based on either worker outputs (such as ...
Workflow transparency in a microtask marketplace
GROUP '12: Proceedings of the 2012 ACM International Conference on Supporting Group WorkInterdependent tasks in Mechanical Turk (MTurk) can be managed efficiently with a workflow, a sequence of tasks through which work passes to its completion. We ask if workers should be informed about the workflow, which we call workflow transparency. ...
Comments