Susumu Saito
Teppei Nakano
ABSTRACT
Working time estimation is known to be helpful for allowing crowd workers to select lucrative microtasks. We previously proposed a machine learning method for estimating the working times of microtasks, but a practical evaluation was not possible because it was unclear what errors would be problematic for workers across different scales of microtask working times. In this study, we formulate MicroLapse, a function that expresses a maximal error in working time prediction that workers can accept for a given working time length. We collected 60,760 survey answers from 660 Amazon Mechanical Turk workers to formulate MicroLapse. Our evaluation of our previous method based on MicroLapse demonstrated that our working time prediction method was fairly successful for shorter microtasks, which could not have been concluded in our previous paper.
- 2019. TurkerView. Retrieved Jun 22, 2019 from https://turkerview.com/Google Scholar
- 2019. Turkopticon 2. Retrieved Jun 22, 2019 from https://turkopticon.info/Google Scholar
- Chris Callison-Burch. 2014. Crowd-workers: Aggregating information across turkers to help them find higher paying work. In Second AAAI Conference on Human Computation and Crowdsourcing.Google ScholarCross Ref
- Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics(2001), 1189--1232.Google Scholar
- Benjamin V Hanrahan, Jutta K Willamowski, Saiganesh Swaminathan, and David B Martin. 2015. TurkBench: Rendering the market for Turkers. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1613--1616.Google ScholarDigital Library
- Lilly C Irani and M Silberman. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 611--620.Google ScholarDigital Library
- Susumu Saito, Chun-Wei Chiang, Saiph Savage, Teppei Nakano, Tetsunori Kobayashi, and Jeffrey P. Bigham. 2019. TurkScanner: Predicting the Hourly Wage of Microtasks. In The World Wide Web Conference (WWW '19). ACM, New York, NY, USA, 3187--3193. https://doi.org/10.1145/3308558.3313716Google ScholarDigital Library
Index Terms
- MicroLapse: Measuring Workers' Leniency to Prediction Errors of Microtasks' Working Times
Recommendations
Modus Operandi of Crowd Workers: The Invisible Role of Microtask Work Environments
The ubiquity of the Internet and the widespread proliferation of electronic devices has resulted in flourishing microtask crowdsourcing marketplaces, such as Amazon MTurk. An aspect that has remained largely invisible in microtask crowdsourcing is that ...
A Community Rather Than A Union: Understanding Self-Organization Phenomenon on MTurk and How It Impacts Turkers and Requesters
CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing SystemsThis paper aims to understand the self-organization phenomenon among the workers of Amazon Mechanical Turk (MTurk), a well-known crowdsourcing platform. Specifically, we explored 1) why MTurk workers self-organize into online communities (Turker ...
Make Hay While the Crowd Shines: Towards Efficient Crowdsourcing on the Web
WWW '15 Companion: Proceedings of the 24th International Conference on World Wide WebWithin the scope of this PhD proposal, we set out to investigate two pivotal aspects that influence the effectiveness of crowdsourcing: (i) microtask design, and (ii) workers behavior. Leveraging the dynamics of tasks that are crowdsourced on the one ...
Comments