Mitchell Gordon
ABSTRACT
We introduce LegionTools, a toolkit and interface for managing large, synchronous crowds of online workers for experiments. This poster contributes the design and implementation of a state-of-the-art crowd management tool, along with a publicly-available, open-source toolkit that future system builders can use to coordinate synchronous crowds of online workers for their systems and studies.
We describe the toolkit itself, along with the underlying design rationale, in order to make it clear to the community of system builders at UIST when and how this tool may be beneficial to their project. We also describe initial deployments of the system in which workers were synchronously recruited to support real-time crowdsourcing systems, including the largest synchronous recruitment and routing of workers from Mechanical Turk that we are aware of. While the version of LegionTools discussed here focuses on Amazon's Mechanical Turk platform, it can be easily extended to other platforms as APIs become available.
- Bernstein, M. S., Brandt, J., Miller, R. C., and Karger, D. R. Crowds in two seconds: Enabling realtime crowd-powered interfaces. In Proc. of UIST (2011). Google Scholar
Digital Library
- Bigham et al., J. P. Vizwiz: Nearly real-time answers to visual questions. In Proc. of UIST (2010). Google ScholarDigital Library
- Lasecki, W. S., Gordon, M., Koutra, D., Jung, M. F., Dow, S. P., and Bigham, J. P. Glance: Rapidly coding behavioral video with the crowd. In Proc. of UIST (2014). Google ScholarDigital Library
- Lasecki, W. S., Murray, K. I., White, S., Miller, R. C., and Bigham, J. P. Real-time crowd control of existing interfaces. In Proc. of UIST (2011). Google ScholarDigital Library
Index Terms
- LegionTools: A Toolkit + UI for Recruiting and Routing Crowds to Synchronous Real-Time Tasks
Recommendations
Crowds in two seconds: enabling realtime crowd-powered interfaces
UIST '11: Proceedings of the 24th annual ACM symposium on User interface software and technologyInteractive systems must respond to user input within seconds. Therefore, to create realtime crowd-powered interfaces, we need to dramatically lower crowd latency. In this paper, we introduce the use of synchronous crowds for on-demand, realtime ...
How many crowdsourced workers should a requester hire?
Recent years have seen an increased interest in crowdsourcing as a way of obtaining information from a potentially large group of workers at a reduced cost. The crowdsourcing process, as we consider in this paper, is as follows: a requester hires a ...
Who are the crowdworkers?: shifting demographics in mechanical turk
CHI EA '10: CHI '10 Extended Abstracts on Human Factors in Computing SystemsAmazon Mechanical Turk (MTurk) is a crowdsourcing system in which tasks are distributed to a population of thousands of anonymous workers for completion. This system is increasingly popular with researchers and developers. Here we extend previous ...
Comments