skip to main content
10.1145/2858036.2858364acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Public Access

The Knowledge Accelerator: Big Picture Thinking in Small Pieces

Published: 07 May 2016 Publication History

Abstract

Crowdsourcing offers a powerful new paradigm for online work. However, real world tasks are often interdependent, requiring a big picture view of the difference pieces involved. Existing crowdsourcing approaches that support such tasks -- ranging from Wikipedia to flash teams -- are bottlenecked by relying on a small number of individuals to maintain the big picture. In this paper, we explore the idea that a computational system can scaffold an emerging interdependent, big picture view entirely through the small contributions of individuals, each of whom sees only a part of the whole. To investigate the viability, strengths, and weaknesses of this approach we instantiate the idea in a prototype system for accomplishing distributed information synthesis and evaluate its output across a variety of topics. We also contribute a set of design patterns that may be informative for other systems aimed at supporting big picture thinking in small pieces.

References

[1]
2015. Etherpad Lite. https://github.com/ether/etherpad-lite. (2015).
[2]
Salman Ahmad, Alexis Battle, Zahan Malkani, and Sepander Kamvar. 2011. The jabberwocky programming environment for structured social computing. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 53--64.
[3]
Paul Andre, Aniket Kittur, and Steven P Dow. 2014a. Crowd synthesis: Extracting categories and clusters from complex data. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. ACM, 989--998.
[4]
Paul Andre, Robert E Kraut, and Aniket Kittur. 2014b. Effects of simultaneous and sequential work structures on distributed collaborative interdependent tasks. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 139--148.
[5]
Anonymised. 2016. Alloy: Clustering with Crowds and Computation. In submission to CHI (2016).
[6]
Michelle Q Wang Baldonado and Terry Winograd. 1997. SenseMaker: an information-exploration interface supporting the contextual evolution of a user's interests. In Proceedings of the ACM SIGCHI Conference on Human factors in computing systems. ACM, 11--18.
[7]
Regina Barzilay, Kathleen R McKeown, and Michael Elhadad. 1999. Information fusion in the context of multi-document summarization. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics. Association for Computational Linguistics, 550--557.
[8]
Michael S. Bernstein, Joel Brandt, Robert C. Miller, and David R. Karger. 2011. Crowds in Two Seconds: Enabling Realtime Crowd-powered Interfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, NY, NY, USA, 33--42.
[9]
Michael S Bernstein, Greg Little, Robert C Miller, Björn Hartmann, Mark S Ackerman, David R Karger, David Crowell, and Katrina Panovich. 2010a. Soylent: a word processor with a crowd inside. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 313--322.
[10]
Michael S Bernstein, Desney Tan, Greg Smith, Mary Czerwinski, and Eric Horvitz. 2010b. Personalization via friendsourcing. ACM Transactions on Computer-Human Interaction (TOCHI) 17, 2 (2010), 6.
[11]
Michael S Bernstein, Jaime Teevan, Susan Dumais, Daniel Liebling, and Eric Horvitz. 2012. Direct answers for search queries in the long tail. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 237--246.
[12]
Jeffrey P Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and others. 2010. VizWiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 333--342.
[13]
Jaime Carbonell and Jade Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 335--336.
[14]
Pew Research Center. 2015. Generational differences in online activities. Report. (25 July 2015). http://www.pewinternet.org/2009/01/28/generational-differences-in-online-activities/.
[15]
Dana Chandler and Adam Kapelner. 2013. Breaking monotony with meaning: Motivation in crowdsourcing markets. Journal of Economic Behavior & Organization 90 (2013), 123--133.
[16]
Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) 2, 3 (2011), 27.
[17]
Lydia B. Chilton, Greg Little, Darren Edge, Daniel S. Weld, and James A. Landay. 2013. Cascade: Crowdsourcing Taxonomy Creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, NY, NY, USA, 1999--2008.
[18]
Richard L Daft and Karl E Weick. 1984. Toward a model of organizations as interpretation systems. Academy of management review 9, 2 (1984), 284--295.
[19]
Hoa Trang Dang, Diane Kelly, and Jimmy J Lin. 2007. Overview of the TREC 2007 Question Answering Track. In TREC, Vol. 7. 63.
[20]
Brenda Dervin. 1983. An overview of sense-making research: Concepts, methods, and results to date. The Author.
[21]
Brenda Dervin. 1992. From the minds eye of the user: The sense-making qualitative-quantitative methodology. Qualitative research in information management 9 (1992), 61--84.
[22]
Brenda Dervin. 1998. Sense-making theory and practice: an overview of user interests in knowledge seeking and use. Journal of knowledge management 2, 2 (1998), 36--46.
[23]
Gunes Erkan and Dragomir R Radev. 2004. LexRank: graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research (2004), 457--479.
[24]
Dennis A Gioia and Kumar Chittipeddi. 1991. Sensemaking and sensegiving in strategic change initiation. Strategic management journal 12, 6 (1991), 433--448.
[25]
Jade Goldstein, Vibhu Mittal, Jaime Carbonell, and Mark Kantrowitz. 2000. Multi-document summarization by sentence extraction. In Proceedings of the 2000 NAACL-ANLP Workshop on Automatic summarization-Volume 4. Association for Computational Linguistics, 40--48.
[26]
Vishal Gupta and Gurpreet Singh Lehal. 2010. A survey of text summarization extractive techniques. Journal of Emerging Technologies in Web Intelligence 2, 3 (2010), 258--268.
[27]
Udo Hahn and Ulrich Reimer. 1999. Knowledge-based text summarization: Salience and generalization operators for knowledge base abstraction. Advances in Automatic Text Summarization (1999), 215--232.
[28]
Anil K Jain, M Narasimha Murty, and Patrick J Flynn. 1999. Data clustering: a review. ACM computing surveys (CSUR) 31, 3 (1999), 264--323.
[29]
Jim Jones. 2013. Turkee Ruby Gem. https://github.com/aantix/turkee. (2013).
[30]
Ece Kamar, Severin Hacker, and Eric Horvitz. 2012. Combining Human and Machine Intelligence in Large-scale Crowdsourcing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems - Volume 1 (AAMAS '12). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 467--474. http://dl.acm.org/citation.cfm?id=2343576.2343643
[31]
Ece Kamar and Eric Horvitz. 2013. Light at the End of the Tunnel: A Monte Carlo Approach to Computing Value of Information. In Proceedings of the 2013 International Conference on Autonomous Agents and Multi-agent Systems (AAMAS '13). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 571--578. http://dl.acm.org/citation.cfm?id=2484920.2485011
[32]
Juho Kim, Phu Tran Nguyen, Sarah Weir, Philip J Guo, Robert C Miller, and Krzysztof Z Gajos. 2014. Crowdsourcing step-by-step information extraction to enhance existing how-to videos. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems. ACM, 4017--4026.
[33]
Aniket Kittur, Ed H Chi, and Bongwon Suh. 2008. Crowdsourcing user studies with Mechanical Turk. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 453--456.
[34]
Aniket Kittur, Susheel Khamkar, Paul Andre, and Robert Kraut. 2012. CrowdWeaver: Visually Managing Complex Crowd Work. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW '12). ACM, NY, NY, USA, 1033--1036.
[35]
Aniket Kittur and Robert E Kraut. 2008. Harnessing the wisdom of crowds in wikipedia: quality through coordination. In Proceedings of the 2008 ACM conference on Computer supported cooperative work. ACM, 37--46.
[36]
Aniket Kittur, Jeffrey V Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013a. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 1301--1318.
[37]
Aniket Kittur, Andrew M Peters, Abdigani Diriye, Trupti Telang, and Michael R Bove. 2013b. Costs and benefits of structured information foraging. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2989--2998.
[38]
Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E Kraut. 2011. Crowdforge: Crowdsourcing complex work. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 43--52.
[39]
Aniket Kittur, Bongwon Suh, Bryan A Pendleton, and Ed H Chi. 2007. He says, she says: conflict and coordination in Wikipedia. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 453--462.
[40]
Gary Klein, Brian Moon, and Robert R Hoffman. 2006. Making sense of sensemaking 2: A macrocognitive model. Intelligent Systems, IEEE 21, 5 (2006), 88--92.
[41]
Anand Kulkarni, Matthew Can, and Björn Hartmann. 2012. Collaboratively crowdsourcing workflows with turkomatic. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 1003--1012.
[42]
Anand P Kulkarni, Matthew Can, and Bjoern Hartmann. 2011. Turkomatic: automatic recursive task and workflow design for mechanical turk. In CHI'11 Extended Abstracts on Human Factors in Computing Systems. ACM, 2053--2058.
[43]
Walter Lasecki, Christopher Miller, Adam Sadilek, Andrew Abumoussa, Donato Borrello, Raja Kushalnagar, and Jeffrey Bigham. 2012. Real-time captioning by groups of non-experts. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 23--34.
[44]
Walter S Lasecki, Christopher D Miller, and Jeffrey P Bigham. 2013a. Warping time for more effective real-time crowdsourcing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2033--2036.
[45]
Walter S Lasecki, Christopher D Miller, Raja Kushalnagar, and Jeffrey P Bigham. 2013b. Legion scribe: real-time captioning by the non-experts. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility. ACM, 22.
[46]
Edith Law and Haoqi Zhang. 2011. Towards Large-Scale Collaborative Planning: Answering High-Level Search Queries Using Human Computation. In AAAI.
[47]
Greg Little, Lydia B Chilton, Max Goldman, and Robert C Miller. 2010. Turkit: human computation algorithms on mechanical turk. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM, 57--66.
[48]
Kurt Luther, Casey Fiesler, and Amy Bruckman. 2013. Redistributing Leadership in Online Creative Collaboration. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW '13). ACM, NY, NY, USA, 1007--1022.
[49]
Inderjeet Mani and Eric Bloedorn. 1997. Multi-document summarization by graph search and matching. arXiv preprint cmp-lg/9712004 (1997).
[50]
Gary Marchionini. 2006. Exploratory Search: From Finding to Understanding. Commun. ACM 49, 4 (April 2006), 41--46.
[51]
Kathleen McKeown, Judith Klavans, Vasileios Hatzivassiloglou, Regina Barzilay, and Eleazar Eskin. 1999. Towards multidocument summarization by reformulation: Progress and prospects. In AAAI/IAAI. 453--460.
[52]
James McKinney. 2015. TfIdf Similarity Ruby Gem. https://github.com/jpmckinney/tf-idf-similarity. (2015).
[53]
Frances J Milliken. 1990. Perceiving and interpreting environmental change: An examination of college administrators? interpretation of changing demographics. Academy of management Journal 33, 1 (1990), 42--63.
[54]
Meredith Ringel Morris, Jarrod Lombardo, and Daniel Wigdor. 2010. WeSearch: supporting collaborative search and sensemaking on a tabletop display. In Proceedings of the 2010 ACM conference on Computer supported cooperative work. ACM, 401--410.
[55]
Aditya Parameswaran, Ming Han Teh, Hector Garcia-Molina, and Jennifer Widom. 2013. Datasift: An expressive and accurate crowd-powered search toolkit. In First AAAI Conference on Human Computation and Crowdsourcing.
[56]
Sharoda A Paul and Meredith Ringel Morris. 2009. CoSense: enhancing sensemaking for collaborative web search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1771--1780.
[57]
Sharoda A Paul and Madhu C Reddy. 2010. Understanding together: sensemaking in collaborative information seeking. In Proceedings of the 2010 ACM conference on Computer supported cooperative work. ACM, 321--330.
[58]
Peter Pirolli and Stuart Card. 1999. Information foraging. Psychological review 106, 4 (1999), 643.
[59]
Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S Bernstein. 2014. Expert crowdsourcing with flash teams. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 75--85.
[60]
Jakob Rogstadius, Vassilis Kostakos, Aniket Kittur, Boris Smus, Jim Laredo, and Maja Vukovic. 2011. An Assessment of Intrinsic and Extrinsic Motivation on Task Performance in Crowdsourcing Markets. In ICWSM.
[61]
Daniel M. Russell, Mark J. Stefik, Peter Pirolli, and Stuart K. Card. 1993. The Cost Structure of Sensemaking. In Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems (CHI '93). ACM, NY, NY, USA, 269--276.
[62]
Bongwon Suh, Gregorio Convertino, Ed H Chi, and Peter Pirolli. 2009. The singularity is not near: slowing growth of Wikipedia. In Proceedings of the 5th International Symposium on Wikis and Open Collaboration. ACM, 8.
[63]
Jaime Teevan, Daniel J Liebling, and Walter S Lasecki. 2014. Selfsourcing personal tasks. In CHI'14 Extended Abstracts on Human Factors in Computing Systems. ACM, 2527--2532.
[64]
Karl E Weick. 1964. Reduction of cognitive dissonance through task enhancement and effort expenditure. The Journal of Abnormal and Social Psychology 68, 5 (1964), 533.
[65]
Karl E. Weick. 1995. Sensemaking in organizations. Vol. 3. Sage.
[66]
Ryen W White, Mikhail Bilenko, and Silviu Cucerzan. 2007. Studying the use of popular destinations to enhance web search interaction. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 159--166.
[67]
Haoqi Zhang, Edith Law, Rob Miller, Krzysztof Gajos, David Parkes, and Eric Horvitz. 2012. Human computation tasks with global constraints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 217--226.

Cited By

View all
  • (2024)OSINT Research Studios: A Flexible Crowdsourcing Framework to Scale Up Open Source Intelligence InvestigationsProceedings of the ACM on Human-Computer Interaction10.1145/36373828:CSCW1(1-38)Online publication date: 26-Apr-2024
  • (2024)Exploring the Future of Informed Consent: Applying a Service Design ApproachProceedings of the ACM on Human-Computer Interaction10.1145/36373308:CSCW1(1-31)Online publication date: 26-Apr-2024
  • (2024)"How fancy you are to make us use your fancy tool": Coordinating Individuals' Tool Preference over Group BoundariesProceedings of the ACM on Human-Computer Interaction10.1145/36330698:GROUP(1-31)Online publication date: 16-Feb-2024
  • Show More Cited By

Index Terms

  1. The Knowledge Accelerator: Big Picture Thinking in Small Pieces

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
    May 2016
    6108 pages
    ISBN:9781450333627
    DOI:10.1145/2858036
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 May 2016

    Permissions

    Request permissions for this article.

    Check for updates

    Badges

    • Honorable Mention

    Author Tags

    1. complex workflow
    2. crowd work
    3. crowdsourcing
    4. design patterns
    5. information synthesis

    Qualifiers

    • Research-article

    Funding Sources

    • NSF

    Conference

    CHI'16
    Sponsor:
    CHI'16: CHI Conference on Human Factors in Computing Systems
    May 7 - 12, 2016
    California, San Jose, USA

    Acceptance Rates

    CHI '16 Paper Acceptance Rate 565 of 2,435 submissions, 23%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)236
    • Downloads (Last 6 weeks)21
    Reflects downloads up to 24 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)OSINT Research Studios: A Flexible Crowdsourcing Framework to Scale Up Open Source Intelligence InvestigationsProceedings of the ACM on Human-Computer Interaction10.1145/36373828:CSCW1(1-38)Online publication date: 26-Apr-2024
    • (2024)Exploring the Future of Informed Consent: Applying a Service Design ApproachProceedings of the ACM on Human-Computer Interaction10.1145/36373308:CSCW1(1-31)Online publication date: 26-Apr-2024
    • (2024)"How fancy you are to make us use your fancy tool": Coordinating Individuals' Tool Preference over Group BoundariesProceedings of the ACM on Human-Computer Interaction10.1145/36330698:GROUP(1-31)Online publication date: 16-Feb-2024
    • (2024)Sensemaking: What is it today?Extended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3636322(1-5)Online publication date: 11-May-2024
    • (2024)Mitigating Barriers to Public Social Interaction with Meronymous CommunicationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642241(1-26)Online publication date: 11-May-2024
    • (2024)Selenite: Scaffolding Online Sensemaking with Comprehensive Overviews Elicited from Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642149(1-26)Online publication date: 11-May-2024
    • (2024)Facilitating Interdisciplinary Collaboration in Large-Scale Research Networks: Tackling Uncertainties in Knowledge Building and the Designing of Robotic Systems in HealthcareInteracting with Computers10.1093/iwc/iwae051Online publication date: 23-Dec-2024
    • (2023)Retrospector: Rapid collaborative reflection to improve collaborative practicesProceedings of the ACM on Human-Computer Interaction10.1145/36100847:CSCW2(1-20)Online publication date: 4-Oct-2023
    • (2023)The Synthesis Lab: Empowering Collaborative Learning in Higher Education through Knowledge SynthesisCompanion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing10.1145/3584931.3606996(245-248)Online publication date: 14-Oct-2023
    • (2023)You Know What I Meme: Enhancing People's Understanding and Awareness of Hateful Memes Using Crowdsourced ExplanationsProceedings of the ACM on Human-Computer Interaction10.1145/35795937:CSCW1(1-27)Online publication date: 16-Apr-2023
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media