skip to main content
10.1145/2998181.2998332acmconferencesArticle/Chapter ViewAbstractPublication PagescscwConference Proceedingsconference-collections
research-article

Communicating Context to the Crowd for Complex Writing Tasks

Published: 25 February 2017 Publication History

Abstract

Crowd work is typically limited to simple, context-free tasks because they are easy to describe and understand. In contrast, complex tasks require communication between the requester and workers to achieve mutual understanding, which can be more work than it is worth. This paper explores the notion of structured communication: using structured microtasks to support communication in the domain of complex writing. Our studies compare a variety of communication mechanisms with respect to the costs to the requester in providing information and the value of that information to workers while performing the task. We find that different mechanisms are effective at different stages of writing. For early drafts, asking the requester to state the biggest problem in the current write-up is valuable and low cost, while later it is more useful for the worker if the requester highlights the text that needs to be improved. These findings can be used to enable richer, more interactive crowd work than what currently seems possible. We incorporate the findings in a workflow for crowdsourcing written content using appropriately timed mechanisms for communicating with the crowd.

References

[1]
Paul André, Robert E. Kraut, and Aniket Kittur. "Effects of simultaneous and sequential work structures on distributed collaborative interdependent tasks." In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 139--148. ACM, 2014.
[2]
Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich. "Soylent: a word processor with a crowd inside." In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pp. 313--322. ACM, 2010.
[3]
Carrie J. Cai, Shamsi T. Iqbal, and Jaime Teevan. "Chain reactions: The impact of order on microtask chains." In Proceedings of the 34th Annual ACM Conference on Human Factors in Computing Systems (CHI'16). ACM, vol. 6. 2016.
[4]
Michelene TH Chi, Paul J. Feltovich, and Robert Glaser. "Categorization and representation of physics problems by experts and novices." Cognitive science 5, no. 2 (1981): 121--152.
[5]
Lydia B. Chilton, Greg Little, Darren Edge, Daniel S. Weld, and James A. Landay. "Cascade: Crowdsourcing taxonomy creation." In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1999--2008. ACM, 2013.
[6]
Shayan Doroudi, Ece Kamar, Emma Brunskill, and Eric Horvitz. "Toward a Learning Science for Complex Crowdsourcing Tasks." In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 2623--2634. ACM, 2016.
[7]
Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. "Shepherding the crowd yields better work." In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, pp. 10131022. ACM, 2012.
[8]
Linda Flower. "Writer-based prose: A cognitive basis for problems in writing." College English (1979): 1937.
[9]
Adam Fourney, Ben Lafreniere, Parmit Chilana, and Michael Terry. "InterTwine: creating interapplication information scent to support coordinated use of software." In Proceedings of the 27th annual ACM symposium on User interface software and technology, pp. 429--438. ACM, 2014.
[10]
Nick Greer, Jaime Teevan, and Shamsi T. Iqbal. An introduction to technological support for writing. Technical Report. Microsoft Research Tech Report MSR-TR-2016-001, 2016.
[11]
Jonathan Grudin. "Computer-supported cooperative work: History and focus." Computer 5 (1994): 19--26.
[12]
Nathan Hahn, Joseph Chang, Ji Eun Kim, and Aniket Kittur. "The Knowledge Accelerator: Big Picture Thinking in Small Pieces." In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 2258--2270. ACM, 2016.
[13]
R.J. Havighurst. Human Development and Education.
[14]
Suzanne Hidi and Pietro Boscolo. "Motivation and writing." Handbook of writing research (2006): 144157.
[15]
Andrea B. Hollingshead, Joseph E. McGrath, and Kathleen M. O'Connor. "Group task performance and communication technology a longitudinal study of computer-mediated versus face-to-face work groups." Small group research 24, no. 3 (1993): 307--333.
[16]
Joy Kim, Justin Cheng, and Michael S. Bernstein. "Ensemble: exploring complementary strengths of leaders and crowds in creative collaboration." In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, pp. 745--755. ACM, 2014.
[17]
Juho Kim, Haoqi Zhang, Paul André, Lydia B. Chilton, Wendy Mackay, Michel Beaudouin-Lafon, Robert C. Miller, and Steven P. Dow. "Cobi: A communityinformed conference scheduling tool." In Proceedings of the 26th annual ACM symposium on User interface software and technology, pp. 173--182. ACM, 2013.
[18]
Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. "The future of crowd work." In Proceedings of the 2013 conference on Computer supported cooperative work, pp. 13011318. ACM, 2013.
[19]
Travis Kriplean, Caitlin Bonnar, Alan Borning, Bo Kinney, and Brian Gill. "Integrating on-demand factchecking with public dialogue." In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing, pp. 1188--1199. ACM, 2014.
[20]
Anand Kulkarni, Matthew Can, and Björn Hartmann. "Collaboratively crowdsourcing workflows with turkomatic." In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work, pp. 1003--1012. ACM, 2012.
[21]
Anand Kulkarni, Philipp Gutheim, Prayag Narula, David Rolnitzky, Tapan Parikh, and Björn Hartmann. "Mobileworks: Designing for quality in a managed crowdsourcing architecture." Internet Computing, IEEE 16, no. 5 (2012): 28--35.
[22]
Walter S. Lasecki, Juho Kim, Nicholas Rafter, Onkur Sen, Jeffrey P. Bigham, and Michael S. Bernstein. "Apparition: Crowdsourced User Interfaces That Come To Life As You Sketch Them." In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 1925--1934. ACM, 2015.
[23]
Thomas D. LaToza, W. Ben Towne, Christian M. Adriano, and André Van Der Hoek. "Microtask programming: Building software with a crowd." InProceedings of the 27th annual ACM symposium on User interface software and technology, pp. 43--54. ACM, 2014.
[24]
Jean Lave and Etienne Wenger. Situated learning: Legitimate peripheral participation. Cambridge university press, 1991. Longmans, Green and Co, 1955.
[25]
Gary M. Olson and Judith S. Olson. "Distance matters." Human-computer interaction 15, no. 2 (2000): 139--178.
[26]
Irene Rae, Gina Venolia, John C. Tang, and David Molnar. "A framework for understanding and designing telepresence." In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing, pp. 1552--1566. ACM, 2015.
[27]
Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S. Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S. Bernstein. "Expert crowdsourcing with flash teams." In Proceedings of the 27th annual ACM symposium on User interface software and technology, pp. 75--85. ACM, 2014.
[28]
Haoqi Zhang, Edith Law, Rob Miller, Krzysztof Gajos, David Parkes, and Eric Horvitz. "Human computation tasks with global constraints." In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 217--226. ACM, 2012.

Cited By

View all
  • (2025)Designing LLM Chains by Adapting Techniques from Crowdsourcing WorkflowsACM Transactions on Computer-Human Interaction10.1145/3716134Online publication date: 7-Feb-2025
  • (2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
  • (2023)Addressing Interpersonal Harm in Online Gaming Communities: The Opportunities and Challenges for a Restorative Justice ApproachACM Transactions on Computer-Human Interaction10.1145/360362530:6(1-36)Online publication date: 25-Sep-2023
  • Show More Cited By

Index Terms

  1. Communicating Context to the Crowd for Complex Writing Tasks

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CSCW '17: Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing
    February 2017
    2556 pages
    ISBN:9781450343350
    DOI:10.1145/2998181
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 25 February 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. communication
    2. context.
    3. crowdsourcing

    Qualifiers

    • Research-article

    Conference

    CSCW '17
    Sponsor:
    CSCW '17: Computer Supported Cooperative Work and Social Computing
    February 25 - March 1, 2017
    Oregon, Portland, USA

    Acceptance Rates

    CSCW '17 Paper Acceptance Rate 183 of 530 submissions, 35%;
    Overall Acceptance Rate 2,235 of 8,521 submissions, 26%

    Upcoming Conference

    CSCW '25

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)26
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 18 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Designing LLM Chains by Adapting Techniques from Crowdsourcing WorkflowsACM Transactions on Computer-Human Interaction10.1145/3716134Online publication date: 7-Feb-2025
    • (2024)The State of Pilot Study Reporting in Crowdsourcing: A Reflection on Best Practices and GuidelinesProceedings of the ACM on Human-Computer Interaction10.1145/36410238:CSCW1(1-45)Online publication date: 26-Apr-2024
    • (2023)Addressing Interpersonal Harm in Online Gaming Communities: The Opportunities and Challenges for a Restorative Justice ApproachACM Transactions on Computer-Human Interaction10.1145/360362530:6(1-36)Online publication date: 25-Sep-2023
    • (2023)Supporting Requesters in Writing Clear Crowdsourcing Task Descriptions Through Computational Flaw AssessmentProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584039(737-749)Online publication date: 27-Mar-2023
    • (2023)Behind the scenesInformation and Management10.1016/j.im.2023.10384160:7Online publication date: 1-Nov-2023
    • (2023)Crowdsourced Argumentation Feedback for Persuasive WritingHuman Interface and the Management of Information10.1007/978-3-031-35129-7_33(461-475)Online publication date: 23-Jul-2023
    • (2022)Being a Solo Endeavor or Team Worker in Crowdsourcing Contests? It is a Long-term Decision You Need to MakeProceedings of the ACM on Human-Computer Interaction10.1145/35555956:CSCW2(1-32)Online publication date: 11-Nov-2022
    • (2022)Organizing in the End of Employment: Information Sharing, Data Stewardship, and Digital WorkerismProceedings of the 1st Annual Meeting of the Symposium on Human-Computer Interaction for Work10.1145/3533406.3533424(1-9)Online publication date: 8-Jun-2022
    • (2022)True or False: Studying the Work Practices of Professional Fact-CheckersProceedings of the ACM on Human-Computer Interaction10.1145/35129746:CSCW1(1-44)Online publication date: 7-Apr-2022
    • (2022)The Invisible Labor of Access in Academic Writing Practices: A Case Analysis with Dyslexic AdultsProceedings of the ACM on Human-Computer Interaction10.1145/35129676:CSCW1(1-25)Online publication date: 7-Apr-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media