skip to main content
10.1145/3025453.3025969acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

ReTool: Interactive Microtask and Workflow Design through Demonstration

Published: 02 May 2017 Publication History

Abstract

In addition to simple form filling, there is an increasing need for crowdsourcing workers to perform freeform interactions directly on content in microtask crowdsourcing (e.g. proofreading articles or specifying object boundary in an image). Such microtasks are often organized within well-designed workflows to optimize task quality and workload distribution. However, designing and implementing the interface and workflow for such microtasks is challenging because it typically requires programming knowledge and tedious manual effort. We present ReTool, a web-based tool for requesters to design and publish interactive microtasks and workflows by demonstrating the microtasks for text and image content. We evaluated ReTool against a task-design tool from a popular crowdsourcing platform and showed the advantages of ReTool over the existing approach.

Supplementary Material

suppl.mov (pn4168-file3.mp4)
Supplemental video
suppl.mov (pn4168p.mp4)
Supplemental video

References

[1]
Amazon. 2016. Amazon Mechanical Turk Homepage. (2016). https://www.mturk.com.
[2]
Amazon. 2016. Developer Tools and Software Development Kits (SDKs) of Amazon Mechanical Turk. (2016). https://requester.mturk.com/developer/tools.
[3]
Paul André, Aniket Kittur, and Steven P. Dow. 2014. Crowd synthesis: extracting categories and clusters from complex data. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing CSCW '14. ACM Press, New York, New York, USA, 989--998.
[4]
Michael S Bernstein, Joel Brandt, Robert C Miller, and David R Karger. 2011. Crowds in Two Seconds: Enabling Realtime Crowd-powered Interfaces. In Proceedings of the 24th annual ACM symposium on User interface software and technology - UIST '11. ACM Press, New York, New York, USA, 33.
[5]
M S Bernstein, G Little, R C Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich. 2010. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 313--322.
[6]
Jeffrey P Bigham, Samual Samuel White, Tom Yeh, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Robin Miller, Aubrey Tatarowicz, and Brandyn White. 2010. VizWiz: Nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology - UIST '10. 333--342.
[7]
Steve Branson, Catherine Wah, Florian Schroff, Boris Babenko, Peter Welinder, Pietro Perona, and Serge Belongie. 2010. Visual recognition with humans in the loop. In Proceedings of the 11th European conference on Computer vision: Part IV - ECCV '10, Vol. 6314. Springer Berlin Heidelberg, Berlin, Heidelberg, 438--451.
[8]
Lydia B. Chilton, Greg Little, Darren Edge, Daniel S. Weld, and James A. Landay. 2013. Cascade: Crowdsourcing taxonomy creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI '13. 1999.
[9]
CrowdFlower. 2016. CrowdFlower Homepage. (2016). https://www.crowdflower.com.
[10]
Jia Deng, Olga Russakovsky, Jonathan Krause, Michael S. Bernstein, Alex Berg, and Li Fei-Fei. 2014. Scalable multi-label annotation. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems CHI '14. ACM Press, New York, New York, USA, 3099--3102.
[11]
Daniel Conrad Halbert. 1984. Programming by example. Ph.D. Dissertation. University of California, Berkeley.
[12]
Kotaro Hara, Vicki Le, and Jon Froehlich. 2013. Combining Crowdsourcing and Google Street View to Identify Street-level Accessibility Problems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 631--640.
[13]
Juho Kim, Phu Tran Nguyen, Sarah Weir, Philip J. Guo, Robert C. Miller, and Krzysztof Z. Gajos. 2014. Crowdsourcing step-by-step information extraction to enhance existing how-to videos. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems CHI '14. 4017--4026.
[14]
Aniket Kittur, Susheel Khamkar, Paul André, and Robert Kraut. 2012. CrowdWeaver. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work CSCW '12. ACM, 1033.
[15]
Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E. Kraut. 2011. CrowdForge: crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology UIST '11. ACM, ACM Press, New York, New York, USA, 43.
[16]
Anand Kulkarni, Matthew Can, and Björn Hartmann. 2012. Collaboratively crowdsourcing workflows with turkomatic. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work CSCW '12. ACM, 1003.
[17]
Walter S. Lasecki, Mitchell Gordon, Steven P. Dow, and Jeffrey P. Bigham. 2014. Glance: enabling rapid interactions with data using the crowd. In Proceedings of the extended abstracts of the 32nd annual ACM conference on Human factors in computing systems CHI EA '14. ACM Press, New York, New York, USA, 511--514.
[18]
James R. Lewis. 1995. IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction 7, 1 (jan 1995), 57--78.
[19]
Greg Little, Lydia B Chilton, Max Goldman, and Robert C Miller. 2010. TurKit: human computation algorithms on mechanical turk. In Proceedings of the 23nd annual ACM symposium on User interface software and technology - UIST '10. ACM, ACM Press, New York, New York, USA, 57.
[20]
Jon Noronha, Eric Hysen, Haoqi Zhang, and Krzysztof Z Gajos. 2011. Platemate: crowdsourcing nutritional analysis from food photographs. In Proceedings of the 24th annual ACM symposium on User interface software and technology - UIST '11. ACM Press, New York, New York, USA, 1.
[21]
Hao Su, Jia Deng, and Li Fei-fei. 2012. Crowdsourcing Annotations for Visual Object Detection. In Proc. AAAI Human Computation? 12. 40--46.
[22]
Jiannan Wang, Tim Kraska, Michael J Franklin, and Jianhua Feng. 2012. CrowdER: Crowdsourcing Entity Resolution. Proceedings of the VLDB Endowment 5 (aug 2012), 1483--1494. http://arxiv.org/abs/1208.1927

Cited By

View all
  • (2022)A Survey on Task Assignment in CrowdsourcingACM Computing Surveys10.1145/349452255:3(1-35)Online publication date: 3-Feb-2022
  • (2021)An Unsupervised Aspect-Aware Recommendation Model with Explanation Text GenerationACM Transactions on Information Systems10.1145/348361140:3(1-29)Online publication date: 29-Nov-2021
  • (2021)A panoramic survey of natural language processing in the Arab worldCommunications of the ACM10.1145/344773564:4(72-81)Online publication date: 22-Mar-2021
  • Show More Cited By

Index Terms

  1. ReTool: Interactive Microtask and Workflow Design through Demonstration

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems
    May 2017
    7138 pages
    ISBN:9781450346559
    DOI:10.1145/3025453
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 May 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Badges

    • Honorable Mention

    Author Tags

    1. crowdsourcing
    2. freeform interactive microtasks
    3. programming by demonstration
    4. retool
    5. workflow

    Qualifiers

    • Research-article

    Funding Sources

    • NUS-Tsinghua Extreme Search project

    Conference

    CHI '17
    Sponsor:

    Acceptance Rates

    CHI '17 Paper Acceptance Rate 600 of 2,400 submissions, 25%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)32
    • Downloads (Last 6 weeks)5
    Reflects downloads up to 13 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2022)A Survey on Task Assignment in CrowdsourcingACM Computing Surveys10.1145/349452255:3(1-35)Online publication date: 3-Feb-2022
    • (2021)An Unsupervised Aspect-Aware Recommendation Model with Explanation Text GenerationACM Transactions on Information Systems10.1145/348361140:3(1-29)Online publication date: 29-Nov-2021
    • (2021)A panoramic survey of natural language processing in the Arab worldCommunications of the ACM10.1145/344773564:4(72-81)Online publication date: 22-Mar-2021
    • (2018)Designing Pronunciation Learning ToolsProceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3173930(1-13)Online publication date: 21-Apr-2018
    • (2018)Information Visualization Evaluation Using CrowdsourcingComputer Graphics Forum10.1111/cgf.1344437:3(573-595)Online publication date: 10-Jul-2018
    • (2017)Video Annotation by Cascading MicrotasksProceedings of the 23rd Brazillian Symposium on Multimedia and the Web10.1145/3126858.3126897(49-56)Online publication date: 17-Oct-2017

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media