Molly Q. Feldman
Skip Abstract Section
Abstract
Writing is a common task for crowdsourcing researchers exploring complex and creative work. To better understand how we write with crowds, we conducted both a literature review of crowd-writing systems and structured interviews with designers of such systems. We argue that the cognitive process theory of writing described by Flower and Hayes (1981), originally proposed as a theory of how solo writers write, offers a useful analytic lens for examining the design of crowd-writing systems. This lens enabled us to identify system design challenges that are inherent to the process of writing as well as design challenges that are introduced by crowdsourcing. The findings present both similarities and differences between how solo writers write versus how we write with crowds. To conclude, we discuss how the research community might apply and transcend the cognitive process model to identify opportunities for future research in crowd-writing systems.
- Elena Agapie, Jaime Teevan, and Andrés Monroy-Hernández. 2015. Crowdsourcing in the field: A case study using local crowds for event reporting. In Third AAAI Conference on Human Computation and Crowdsourcing.Google Scholar
Cross Ref
- Ali Alkhatib, Michael S Bernstein, and Margaret Levi. 2017. Examining crowd work and gig work through the historical lens of piecework. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 4599--4616.Google ScholarDigital Library
- Nancy Allen, Dianne Atkinson, Meg Morgan, Teresa Moore, and Craig Snow. 1987. What experienced collaborators say about collaborative writing. Iowa State Journal of Business and Technical Communication, Vol. 1, 2 (1987), 70--90.Google ScholarCross Ref
- Amazon. 2020. Amazon Mechanical Turk. https://web.archive.org/web/20200612183816/https://www.mturk.com/.Google Scholar
- Daniel W Barowy, Charlie Curtsinger, Emery D Berger, and Andrew McGregor. 2012. Automan: A platform for integrating human-based and digital computation. In Proceedings of the ACM international conference on Object oriented programming systems languages and applications. 639--654.Google ScholarDigital Library
- Anne Becker. 2006. A review of writing model research based on cognitive processes. Revision: History, theory, and practice (2006), 25--49.Google Scholar
- Michael S Bernstein, Greg Little, Robert C Miller, Björn Hartmann, Mark S Ackerman, David R Karger, David Crowell, and Katrina Panovich. 2010. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 313--322.Google ScholarDigital Library
- Patrick Biernacki and Dan Waldorf. 1981. Snowball sampling: Problems and techniques of chain referral sampling. Sociological methods & research, Vol. 10, 2 (1981), 141--163.Google Scholar
- Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et almbox. 2010. VizWiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. 333--342.Google ScholarDigital Library
- Minsuk Chang, Léonore V Guillain, Hyeungshik Jung, Vivian M Hare, Juho Kim, and Maneesh Agrawala. 2018. Recipescape: An interactive tool for analyzing cooking instructions at scale. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--12.Google ScholarDigital Library
- Neil Cohn. 2013. Visual narrative structure. Cognitive science, Vol. 37, 3 (2013), 413--452.Google Scholar
- Justin Cranshaw, Emad Elwany, Todd Newman, Rafal Kocielnik, Bowen Yu, Sandeep Soni, Jaime Teevan, and Andrés Monroy-Hernández. 2017. Calendar. help: Designing a workflow-based scheduling agent with humans in the loop. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2382--2393.Google ScholarDigital Library
- Djellel Eddine Difallah, Michele Catasta, Gianluca Demartini, Panagiotis G Ipeirotis, and Philippe Cudré-Mauroux. 2015. The dynamics of micro-task crowdsourcing: The case of amazon mturk. In Proceedings of the 24th international conference on world wide web. 238--247.Google ScholarDigital Library
- Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the crowd yields better work. In Proceedings of the ACM 2012 conference on computer supported cooperative work. ACM, 1013--1022.Google ScholarDigital Library
- Ryan Drapeau, Lydia B Chilton, Jonathan Bragg, and Daniel S Weld. 2016. Microtalk: Using argumentation to improve crowdsourcing accuracy. In Fourth AAAI Conference on Human Computation and Crowdsourcing.Google ScholarCross Ref
- Linda Flower and John R Hayes. 1981. A cognitive process theory of writing. College composition and communication, Vol. 32, 4 (1981), 365--387.Google ScholarCross Ref
- Snehal Gaikwad, Durim Morina, Rohit Nistala, Megha Agarwal, Alison Cossette, Radhika Bhanu, Saiph Savage, Vishwajeet Narwal, Karan Rajpal, Jeff Regino, et almbox. 2015. Daemo: A self-governed crowdsourcing marketplace. In Adjunct proceedings of the 28th annual ACM symposium on user interface software & technology. 101--102.Google ScholarDigital Library
- Elena L Glassman, Aaron Lin, Carrie J Cai, and Robert C Miller. 2016. Learnersourcing personalized hints. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, 1626--1636.Google ScholarDigital Library
- Google. 2020. Google Docs. Online. https://web.archive.org/web/20200416081437/https://www.google.com/docs/about/Google Scholar
- Nick Greer, Jaime Teevan, and Shamsi T Iqbal. 2016. An introduction to technological support for writing. Technical Report. Technical Report. Microsoft Research Tech Report MSR-TR-2016-001.Google Scholar
- Nathan Hahn, Joseph Chang, Ji Eun Kim, and Aniket Kittur. 2016. The Knowledge Accelerator: Big picture thinking in small pieces. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2258--2270.Google ScholarDigital Library
- David Hicks, Peter E Doolittle, and E Thomas Ewing. 2004. The SCIM-C strategy: Expert historians, historical inquiry, and multimedia. Social Education, Vol. 68, 3 (2004), 221--226.Google Scholar
- Hwajung Hong, Eric Gilbert, Gregory D Abowd, and Rosa I Arriaga. 2015. In-group questions and out-group answers: crowdsourcing daily living advice for individuals with autism. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 777--786.Google ScholarDigital Library
- Jeff Howe. 2019. Crowdsourcing: A Definition. https://web.archive.org/web/20191027010513/https://crowdsourcing.typepad.com/.Google Scholar
- Chieh-Yang Huang, Shih-Hong Huang, and Ting-Hao Kenneth Huang. 2020. Heteroglossia: In-Situ Story Ideation with the Crowd. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM.Google ScholarDigital Library
- Ting-Hao Huang, Joseph Chee Chang, and Jeffrey P Bigham. 2018. Evorus: A crowd-powered conversational assistant built to automate itself over time. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1--13.Google ScholarDigital Library
- Lilly C Irani and M Six Silberman. 2013. Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems. 611--620.Google ScholarDigital Library
- Youxuan Jiang, Catherine Finegan-Dollak, Jonathan K Kummerfeld, and Walter Lasecki. 2018. Effective crowdsourcing for a new type of summarization task. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). 628--633.Google ScholarCross Ref
- Harmanpreet Kaur, Alex C Williams, Anne Loomis Thompson, Walter S Lasecki, Shamsi T Iqbal, and Jaime Teevan. 2018. Creating Better Action Plans for Writing Tasks via Vocabulary-Based Planning. Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (2018), 86.Google ScholarDigital Library
- Joy Kim and Andres Monroy-Hernandez. 2016. Storia: Summarizing social media content based on narrative theory using crowdsourcing. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, 1018--1027.Google ScholarDigital Library
- Joy Kim, Sarah Sterman, Allegra Argent Beal Cohen, and Michael S Bernstein. 2017. Mechanical novel: Crowdsourcing complex work through reflection and revision. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 233--245.Google ScholarDigital Library
- Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work. 1301--1318.Google ScholarDigital Library
- Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E Kraut. 2011. Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 43--52.Google ScholarDigital Library
- Nicolas Kokkalis, Thomas Köhn, Carl Pfeiffer, Dima Chornyi, Michael S Bernstein, and Scott R Klemmer. 2013. EmailValet: Managing email overload through private, accountable crowdsourcing. In Proceedings of the 2013 conference on Computer supported cooperative work. 1291--1300.Google ScholarDigital Library
- Robert Kraut, Jolene Galegher, Robert Fish, and Barbara Chalfonte. 1992. Task requirements and media choice in collaborative writing. Human-Computer Interaction, Vol. 7, 4 (1992), 375--407.Google ScholarDigital Library
- Walter Lasecki, Christopher Miller, Adam Sadilek, Andrew Abumoussa, Donato Borrello, Raja Kushalnagar, and Jeffrey Bigham. 2012. Real-time captioning by groups of non-experts. In Proceedings of the 25th annual ACM symposium on User interface software and technology. 23--34.Google ScholarDigital Library
- Walter S Lasecki, Juho Kim, Nick Rafter, Onkur Sen, Jeffrey P Bigham, and Michael S Bernstein. 2015. Apparition: Crowdsourced user interfaces that come to life as you sketch them. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 1925--1934.Google ScholarDigital Library
- Walter S Lasecki, Rachel Wesley, Jeffrey Nichols, Anand Kulkarni, James F Allen, and Jeffrey P Bigham. 2013. Chorus: a crowd-powered conversational assistant. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 151--162.Google ScholarDigital Library
- Edith Law and Luis von Ahn. 2011. Human computation. Synthesis lectures on artificial intelligence and machine learning, Vol. 5, 3 (2011), 1--121.Google ScholarDigital Library
- Mary M Lay and William M Karis. 1991. Collaborative writing in industry: Investigations in theory and practice. Baywood Publishing Company.Google Scholar
- C Michael Levy and Sarah Ransdell. 2013. The science of writing: Theories, methods, individual differences and applications. Routledge.Google Scholar
- Tianyi Li, Kurt Luther, and Chris North. 2018. CrowdIA: Solving Mysteries with Crowdsourced Sensemaking. Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (2018), 105.Google ScholarDigital Library
- Chi-Chin Lin, Yi-Ching Huang, and Jane Yung-jen Hsu. 2014. Crowdsourced explanations for humorous internet memes based on linguistic theories. In Second AAAI Conference on Human Computation and Crowdsourcing.Google ScholarCross Ref
- Greg Little, Lydia B Chilton, Max Goldman, and Robert C Miller. 2010. Turkit: human computation algorithms on mechanical turk. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. 57--66.Google ScholarDigital Library
- Paul Benjamin Lowry, Aaron Curtis, and Michelle René Lowry. 2004. Building a taxonomy and nomenclature of collaborative writing to improve interdisciplinary research and practice. The Journal of Business Communication (1973), Vol. 41, 1 (2004), 66--99.Google ScholarCross Ref
- Kurt Luther, Jari-Lee Tolentino, Wei Wu, Amy Pavel, Brian P Bailey, Maneesh Agrawala, Björn Hartmann, and Steven P Dow. 2015. Structuring, aggregating, and evaluating crowdsourced design critique. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing. ACM, 473--485.Google ScholarDigital Library
- Narges Mahyar, Michael R James, Michelle M Ng, Reginald A Wu, and Steven P Dow. 2018. CommunityCrit: Inviting the Public to Improve and Evaluate Urban Design Ideas through Micro-Activities. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 195.Google ScholarDigital Library
- Thomas W Malone and Kevin Crowston. 1994. The interdisciplinary study of coordination. ACM Computing Surveys (CSUR), Vol. 26, 1 (1994), 87--119.Google ScholarDigital Library
- Brian McInnis, Dan Cosley, Chaebong Nam, and Gilly Leshed. 2016. Taking a HIT: Designing around rejection, mistrust, risk, and workers' experiences in Amazon Mechanical Turk. In Proceedings of the 2016 CHI conference on human factors in computing systems. 2271--2282.Google ScholarDigital Library
- Brian McInnis, Gilly Leshed, and Dan Cosley. 2018. Crafting Policy Discussion Prompts as a Task for Newcomers. Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (2018), 121.Google ScholarDigital Library
- Microsoft. 2020. Microsoft Office Online. Online. https://web.archive.org/web/20200415074732/https://products.office.com/en-us/free-office-online-for-the-webGoogle Scholar
- Michael Nebeling, Alexandra To, Anhong Guo, Adrian A de Freitas, Jaime Teevan, Steven P Dow, and Jeffrey P Bigham. 2016. WearWrite: Crowd-assisted writing from smartwatches. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3834--3846.Google ScholarDigital Library
- Jonas Oppenlaender, Kristy Milland, Aku Visuri, Panos Ipeirotis, and Simo Hosio. 2020. Creativity on Paid Crowdsourcing Platforms. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--14.Google ScholarDigital Library
- Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on intelligence analysis, Vol. 5. McLean, VA, USA, 2--4.Google Scholar
- Jérémie Rappaz, Michele Catasta, Robert West, and Karl Aberer. 2018. Latent structure in collaboration: the case of Reddit R/place. In Twelfth International AAAI Conference on Web and Social Media.Google ScholarCross Ref
- Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S Bernstein. 2014. Expert crowdsourcing with flash teams. In Proceedings of the 27th annual ACM symposium on User interface software and technology. 75--85.Google ScholarDigital Library
- Niloufar Salehi, Jaime Teevan, Shamsi Iqbal, and Ece Kamar. 2017. Communicating context to the crowd for complex writing tasks. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 1890--1901.Google ScholarDigital Library
- Matthew J Salganik and Duncan J Watts. 2009. Web-based experiments for the study of collective social dynamics in cultural markets. Topics in cognitive science, Vol. 1, 3 (2009), 439--468.Google Scholar
- Gordon B Schmidt and William M Jettinghoff. 2016. Using Amazon Mechanical Turk and other compensated crowdsourcing sites. Business Horizons, Vol. 59, 4 (2016), 391--400.Google ScholarCross Ref
- Ben Shneiderman. 1996. The eyes have it: A task by data type taxonomy for information visualizations. In Proceedings 1996 IEEE symposium on visual languages. IEEE, 336--343.Google ScholarCross Ref
- Anselm L Strauss. 1987. Qualitative analysis for social scientists. Cambridge university press.Google Scholar
- Saiganesh Swaminathan, Raymond Fok, Fanglin Chen, Ting-Hao Huang, Irene Lin, Rohan Jadvani, Walter S Lasecki, and Jeffrey P Bigham. 2017. Wearmail: On-the-go access to information in your email with a privacy-preserving human computation workflow. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 807--815.Google ScholarDigital Library
- The Etherpad Foundation. 2020. Etherpad. https://web.archive.org/web/20200803185137/https://etherpad.org/.Google Scholar
- Richard C Thomas. 2012. Long term human-computer interaction: An exploratory perspective. Springer Science & Business Media.Google Scholar
- Melissa A Valentine, Daniela Retelny, Alexandra To, Negar Rahmati, Tulsee Doshi, and Michael S Bernstein. 2017. Flash organizations: Crowdsourcing complex work by structuring crowds as organizations. In Proceedings of the 2017 CHI conference on human factors in computing systems. 3523--3537.Google ScholarDigital Library
- Vasilis Verroios and Michael S Bernstein. 2014. Context trees: Crowdsourcing global understanding from local views. In Second AAAI Conference on Human Computation and Crowdsourcing.Google ScholarCross Ref
- Luis Von Ahn and Laura Dabbish. 2004. Labeling images with a computer game. In Proceedings of the SIGCHI conference on Human factors in computing systems. 319--326.Google ScholarDigital Library
- Nai-Ching Wang, David Hicks, and Kurt Luther. 2018. Exploring Trade-Offs Between Learning and Productivity in Crowdsourced History. Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (2018), 178.Google ScholarDigital Library
- Mark E Whiting, Dilrukshi Gamage, Snehalkumar (Neil) S Gaikwad, Aaron Gilbee, Shirish Goyal, Alipta Ballav, Dinesh Majeti, Nalin Chhibber, Angela Richmond-Fuller, Freddie Vargus, et almbox. 2017. Crowd guilds: Worker-led reputation and feedback on crowdsourcing platforms. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. 1902--1913.Google ScholarDigital Library
- Mark E Whiting, Grant Hugh, and Michael S Bernstein. 2019. Fair Work: Crowd Work Minimum Wage with One Line of Code. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, Vol. 7. 197--206.Google ScholarCross Ref
- Wikimedia Foundation, Inc. 2020. Wikipedia: The free encyclopedia. Online. https://www.wikipedia.orgGoogle Scholar
- Wesley Willett, Jeffrey Heer, and Maneesh Agrawala. 2012. Strategies for crowdsourcing social data analysis. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 227--236.Google ScholarDigital Library
- Anbang Xu, Huaming Rao, Steven P Dow, and Brian P Bailey. 2015. A classroom study of using crowd feedback in the iterative design process. In Proceedings of the 18th ACM conference on computer supported cooperative work & social computing. ACM, 1637--1648.Google ScholarDigital Library
- Amy X Zhang, Lea Verou, and David Karger. 2017. Wikum: Bridging discussion forums and wikis using recursive summarization. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, 2082--2096.Google ScholarDigital Library
- Haiyi Zhu, Steven P Dow, Robert E Kraut, and Aniket Kittur. 2014. Reviewing versus doing: Learning and performance in crowd assessment. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 1445--1455.Google ScholarDigital Library
Index Terms
- How We Write with Crowds
Recommendations
A study of the usability of handwriting recognition for text entry by children
This paper describes a pilot study that investigated the usability of handwriting recognition for text entry in a free writing activity. The study was carried out with eighteen children aged 7 and 8; each used three different writing methods to ...
Watch Me Write: Exploring the Effects of Revealing Creative Writing Process through Writing Replay
C&C '22: Proceedings of the 14th Conference on Creativity and CognitionThe written word is an asynchronous form of communication through which static texts are exchanged. However, the act of writing is a dynamic cognitive process in which evolving ideas are organized into a coherent narrative. This process is not shared ...
Why Users Do Not Want to Write Together When They Are Writing Together: Users' Rationales for Today's Collaborative Writing Practices
This study builds upon the 30-years HCI research of collaborative writing and focuses on users' experience of writing together in today's context. By interviewing 30 participants from both academia and industry, the paper examines how people write ...
Comments