research-article

Exploring Trade-Offs Between Learning and Productivity in Crowdsourced History

Authors:

Nai-Ching Wang,

Kurt LutherAuthors Info & Claims

Proceedings of the ACM on Human-Computer Interaction, Volume 2, Issue CSCW

Article No.: 178, Pages 1 - 24

https://doi.org/10.1145/3274447

Published: 01 November 2018 Publication History

Abstract

Crowdsourcing more complex and creative tasks is seen as a desirable goal for both employers and workers, but these tasks traditionally require domain expertise. Employers can recruit only expert workers, but this approach does not scale well. Alternatively, employers can decompose complex tasks into simpler micro-tasks, but some domains, such as historical analysis, cannot be easily modularized in this way. A third approach is to train workers to learn the domain expertise. This approach offers clear benefits to workers, but is perceived as costly or infeasible for employers. In this paper, we explore the trade-offs between learning and productivity in training crowd workers to analyze historical documents. We compare CrowdSCIM, a novel approach that teaches historical thinking skills to crowd workers, with two crowd learning techniques from prior work and a baseline. Our evaluation (n=360) shows that CrowdSCIM allows workers to learn domain expertise while producing work of equal or higher quality versus other conditions, but efficiency is slightly lower.

References

[1]

{n. d.}. DIYHistory | Transcribe. https://diyhistory.lib.uiowa.edu/

[2]

{n. d.}. Feature Article - Historical Thinking, Winter 2010- Teaching with Primary Sources | Teacher Resources - Library of Congress. http://www.loc.gov/teachers/tps/quarterly/historical_thinking/article.html

[3]

{n. d.}. Incite | Crowdsourced Analysis of Omeka Documents | Virginia Tech. http://incite.cs.vt.edu/

[4]

{n. d.}. Smithsonian Digital Volunteers. https://transcription.si.edu/

[5]

Keith C. Barton and Linda S. Levstik. 2004. Teaching history for the common good. Routledge.

[6]

Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich. 2010. Soylent: A Word Processor with a Crowd Inside. In Proceedings of the 23rd Annual ACM Symposium on User Interface Software and Technology (UIST '10). ACM, New York, NY, USA, 313--322.

Digital Library

[7]

Lydia B. Chilton, Greg Little, Darren Edge, Daniel S. Weld, and James A. Landay. 2013. Cascade: Crowdsourcing Taxonomy Creation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 1999--2008.

Digital Library

[8]

D. Coetzee, Seongtaek Lim, Armando Fox, Björn Hartmann, and Marti A. Hearst. 2015. Structuring Interactions for Large-Scale Synchronous Peer Learning. In Proceedings of the 18th ACMConference on Computer Supported Cooperative Work & Social Computing (CSCW '15). ACM, NewYork, NY, USA, 1139--1152.

Digital Library

[9]

Robin George Collingwood and Willem J. van der Dussen. 1993. The idea of history. Oxford University Press on Demand.

[10]

Allan Collins, John Seely Brown, and Susan E. Newman. 1988. Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics. Thinking: The Journal of Philosophy for Children 8, 1 (1988), 2--10.

[11]

Mira Dontcheva, Robert R. Morris, Joel R. Brandt, and Elizabeth M. Gerber. 2014. Combining Crowdsourcing and Learning to Improve Engagement and Performance. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 3379--3388.

Digital Library

[12]

Shayan Doroudi, Ece Kamar, Emma Brunskill, and Eric Horvitz. 2016. Toward a Learning Science for Complex Crowdsourcing Tasks. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 2623--2634.

Digital Library

[13]

Steven Dow, Anand Kulkarni, Scott Klemmer, and Björn Hartmann. 2012. Shepherding the Crowd Yields BetterWork. In Proceedings of the ACM 2012 Conference on Computer Supported CooperativeWork (CSCW '12). ACM, New York, NY, USA, 1013--1022.

Digital Library

[14]

B. Farrimond, S. Presland, J. Bonar-Law, and F. Pogson. 2008. Making History Happen: Spatiotemporal Data Visualization for Historians. In Second UKSIM European Symposium on Computer Modeling and Simulation, 2008. EMS '08. 424--429.

Digital Library

[15]

Elena L. Glassman, Juho Kim, Andrés Monroy-Hernández, and Meredith Ringel Morris. 2015. Mudslide: A Spatially Anchored Census of Student Confusion for Online Lecture Videos. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1555--1564.

Digital Library

[16]

Elena L. Glassman, Aaron Lin, Carrie J. Cai, and Robert C. Miller. 2016. Learnersourcing Personalized Hints. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York, NY, USA, 1626--1636.

Digital Library

[17]

Nitesh Goyal and Susan R. Fussell. 2015. Designing for Collaborative Sensemaking: Leveraging Human Cognition For Complex Tasks. arXiv preprint arXiv:1511.05737 (2015).

[18]

Nitesh Goyal and Susan R. Fussell. 2016. Effects of Sensemaking Translucence on Distributed Collaborative Analysis. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW '16). ACM, New York, NY, USA, 288--302.

Digital Library

[19]

Michael D. Greenberg, Matthew W. Easterday, and Elizabeth M. Gerber. 2015. Critiki: A Scaffolded Approach to Gathering Design Feedback from Paid Crowdworkers. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition (C&C '15). ACM, New York, NY, USA, 235--244.

Digital Library

[20]

Stuart Greene. 1994. Students as authors in the study of history. In Teaching and Learning in History. Routledge, 137--170.

[21]

David Hicks and Peter E. Doolittle. 2008. Fostering Analysis in Historical Inquiry Through Multimedia Embedded Scaffolding. Theory & Research in Social Education 36, 3 (July 2008), 206--232.

[22]

David Hicks, Peter E. Doolittle, and E. Thomas Ewing. 2004. The SCIM-C strategy: expert historians, historical inquiry, and multimedia. Social Education 68, 3 (April 2004), 221--226.

[23]

Cynthia Hynd, Jodi Patrick Holschuh, and Betty P. Hubbard. 2004. Thinking like a historian: College students' reading of multiple historical documents. Journal of Literacy Research 36, 2 (2004), 141--176. http://jlr.sagepub.com/content/36/2/141.short

[24]

Michael J. Jacobson, Chrystalla Maouri, Punyashloke Mishra, and Christopher Kolar. 1995. Learning with Hypertext Learning Environments: Theory, Design, and Research. J. Educ. Multimedia Hypermedia 4, 4 (Dec. 1995), 321--364. http://dl.acm.org/citation.cfm?id=227170.227173

Digital Library

[25]

Michael J. Jacobson and Rand J. Spiro. 1995. Hypertext Learning Environments, Cognitive Flexibility, and the Transfer of Complex Knowledge: An Empirical Investigation. Journal of Educational Computing Research 12, 4 (June 1995), 301--333.

[26]

Charlene Jennett, Laure Kloetzer, Daniel Schneider, Ioanna Iacovides, Anna Cox, Margaret Gold, Brian Fuchs, Alexandra Eveleigh, Kathleen Methieu, Zoya Ajani, and others. 2016. Motivations, learning and creativity in online citizen science. Journal of Science Communication 15, 3 (2016). http://oro.open.ac.uk/47008/

[27]

Juho Kim, Robert C. Miller, and Krzysztof Z. Gajos. 2013. Learnersourcing Subgoal Labeling to Support Learning from How-to Videos. In CHI '13 Extended Abstracts on Human Factors in Computing Systems (CHI EA '13). ACM, New York, NY, USA, 685--690.

Digital Library

[28]

Juho Kim and others. 2015. Learnersourcing: improving learning with collective learner activity. Ph.D. Dissertation. Massachusetts Institute of Technology. http://dspace.mit.edu/handle/1721.1/101464

[29]

Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The Future of Crowd Work. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (CSCW '13). ACM, New York, NY, USA, 1301--1318.

Digital Library

[30]

Aniket Kittur, Boris Smus, Susheel Khamkar, and Robert E. Kraut. 2011. CrowdForge: Crowdsourcing ComplexWork. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 43--52.

Digital Library

[31]

Travis Kriplean, Caitlin Bonnar, Alan Borning, Bo Kinney, and Brian Gill. 2014. Integrating On-demand Fact-checking with Public Dialogue. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '14). ACM, New York, NY, USA, 1188--1199.

Digital Library

[32]

J. Richard Landis and Gary G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33, 1 (1977), 159--174.

[33]

Edith Law, Krzysztof Z. Gajos, Andrea Wiggins, Mary L. Gray, and Alex Williams. 2017. Crowdsourcing As a Tool for Research: Implications of Uncertainty. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). ACM, New York, NY, USA, 1544--1561.

Digital Library

[34]

Doris Jung-Lin Lee, Joanne Lo, Moonhyok Kim, and Eric Paulos. 2016. Crowdclass: Designing Classification-Based Citizen Science Learning Modules. In Fourth AAAI Conference on Human Computation and Crowdsourcing. https://aaai.org/ocs/index.php/HCOMP/HCOMP16/paper/view/14027

[35]

Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. 2010. Exploring Iterative and Parallel Human Computation Processes. In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '10). ACM, New York, NY, USA, 68--76.

Digital Library

[36]

Kurt Luther, Nathan Hahn, Steven P. Dow, and Aniket Kittur. 2015. Crowdlines: Supporting Synthesis of Diverse Information Sources through Crowdsourced Outlines. In Third AAAI Conference on Human Computation and Crowdsourcing. http://www.aaai.org/ocs/index.php/HCOMP/HCOMP15/paper/view/11603

[37]

Nikki Mandell. 2008. Thinking like a Historian: A Framework for Teaching and Learning. OAH Magazine of History 22, 2 (April 2008), 55--59.

[38]

Ville J. E. Manninen. 2017. Sourcing practices in online journalism: an ethnographic study of the formation of trust in and the use of journalistic sources. Journal of Media Practice 18, 2--3 (Sept. 2017), 212--228.

[39]

Andrea L. McNeill, Peter E. Doolittle, and David Hicks. 2009. The effects of training, modality, and redundancy on the development of a historical inquiry strategy in a multimedia learning environment. Journal of Interactive Online Learning 8, 3 (2009), 255--269.

[40]

Piotr Mitros. 2015. Learnersourcing of Complex Assessments. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale (L@S '15). ACM, New York, NY, USA, 317--320.

Digital Library

[41]

Vineet Pandey, Amnon Amir, Justine Debelius, Embriette R. Hyde, Tomasz Kosciolek, Rob Knight, and Scott Klemmer. 2017. Gut Instinct: Creating Scientific Theories with Online Learners. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 6825--6836.

Digital Library

[42]

Jennifer Rutner and Roger Schonfeld. 2012. Supporting the Changing Research Practices of Historians. Technical Report. Ithaka S+R, New York. http://sr.ithaka.org/?p=22532

[43]

John W. Saye and Thomas Brush. 2002. Scaffolding critical reasoning about history and social issues in multimedia-supported learning environments. Educational Technology Research and Development 50, 3 (Sept. 2002), 77--96.

[44]

Ivor Shapiro, Colette Brin, Isabelle Bédard-Brûlé, and Kasia Mychajlowycz. 2013. Verification as a Strategic Ritual. Journalism Practice 7, 6 (Dec. 2013), 657--673.

[45]

Jakub Simko, Marián Simko, Mária Bieliková, Jakub Sevcech, and Roman Burger. 2013. Classsourcing: Crowd-Based Validation of Question-Answer Learning Objects. In International Conference on Computational Collective Intelligence. Springer, 62--71. http://link.springer.com/chapter/10.1007/978--3--642--40495--5_7

Digital Library

[46]

Peter N. Stearns, Peter C. Seixas, and Sam Wineburg. 2000. Knowing, teaching, and learning history: National and international perspectives. NYU Press.

[47]

Bill Tally and Lauren B. Goldenberg. 2005. Fostering historical thinking with digitized primary sources. Journal of Research on Technology in Education 38, 1 (2005), 1--21.

[48]

Peter Tolmie, Rob Procter, David William Randall, Mark Rouncefield, Christian Burger, Geraldine Wong Sak Hoi, Arkaitz Zubiaga, and Maria Liakata. 2017. Supporting the Use of User Generated Content in Journalistic Practice. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3632--3644.

Digital Library

[49]

Sarah Weir, Juho Kim, Krzysztof Z. Gajos, and Robert C. Miller. 2015. Learnersourcing Subgoal Labels for How-to Videos. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15). ACM, New York, NY, USA, 405--416.

Digital Library

[50]

Joseph Jay Williams, Juho Kim, Anna Rafferty, Samuel Maldonado, Krzysztof Z. Gajos, Walter S. Lasecki, and Neil Heffernan. 2016. AXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale (L@S '16). ACM, New York, NY, USA, 379--388.

Digital Library

[51]

Sam Wineburg. 1999. Historical thinking and other unnatural acts. The Phi Delta Kappan 80, 7 (1999), 488--499. http://www.jstor.org/stable/20439490

[52]

Sam Wineburg. 2010. Thinking like a historian. Teaching with primary sources quarterly 3, 1 (2010), 2--4. https://www.weteachnyc.org/media/filer_public/24/4a/244ab1eb-4540--4ca1-a60e-5ef96326b365/research_history.pdf

[53]

Sam Wineburg. 2010. Thinking Like a Historian. Library of Congress (2010). http://www.loc.gov/teachers/tps/quarterly/historical_thinking/article.html

[54]

Samuel S. Wineburg. 1991. On the Reading of Historical Texts: Notes on the Breach Between School and Academy.American Educational Research Journal 28, 3 (Sept. 1991), 495--519.

[55]

Samuel S. Wineburg and Suzanne M. Wilson. 1991. Subject matter knowledge in the teaching of history. Advances in research on teaching 2 (1991), 305--347.

[56]

Anbang Xu, Huaming Rao, Steven P. Dow, and Brian P. Bailey. 2015. A Classroom Study of Using Crowd Feedback in the Iterative Design Process. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15). ACM, New York, NY, USA, 1637--1648.

Digital Library

[57]

Haiyi Zhu, Steven P. Dow, Robert E. Kraut, and Aniket Kittur. 2014. Reviewing Versus Doing: Learning and Performance in Crowd Assessment. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '14). ACM, New York, NY, USA, 1445--1455.

Digital Library

Cited By

Wu JWang CHuang DNing JXie L(2025)Heart Rate Variability Estimation Based on RFID Tag-Pair in Dynamic EnvironmentsACM Transactions on Computing for Healthcare10.1145/36913556:1(1-26)Online publication date: 23-Jan-2025
https://dl.acm.org/doi/10.1145/3691355
Wu JLuo XChen SLyu YJi XMeng YWang D(2025)Continuous Authentication via Wrist Photoplethysmogram: An Extensive StudyIEEE Transactions on Mobile Computing10.1109/TMC.2024.344320824:2(525-538)Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1109/TMC.2024.3443208
Xu ZJin WYao CLiu XMa SLiu YQin ZVakilinia ILiu D(2024)EM-Rhythm: An Authentication Method for Heterogeneous IoT DevicesACM Transactions on Sensor Networks10.1145/370044120:6(1-21)Online publication date: 22-Nov-2024
https://dl.acm.org/doi/10.1145/3700441
Show More Cited By

Index Terms

Exploring Trade-Offs Between Learning and Productivity in Crowdsourced History
1. Applied computing
  1. Arts and humanities
  2. Education
    1. Interactive learning environments
2. Human-centered computing
  1. Collaborative and social computing

Recommendations

Job Hopping, Information Technology Spillovers, and Productivity Growth

The movement of information technology IT workers among firms is believed to be an important mechanism by which IT-related innovations diffuse throughout the economy. We use a newly developed source of employee microdata---an online resume database---to ...
Quantifying the Invisible Labor in Crowd Work
CSCW2

Crowdsourcing markets provide workers with a centralized place to find paid work. What may not be obvious at first glance is that, in addition to the work they do for pay, crowd workers also have to shoulder a variety of unpaid invisible labor in these ...
Individual, social and situational determinants of telecommuter productivity

Productivity of remote workers is of critical concern to organizations and managers contemplating telecommuting arrangements. Here we suggest a general theoretical framework for understanding telecommuter productivity, and then report on a two-phased ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Human-Computer Interaction

Proceedings of the ACM on Human-Computer Interaction Volume 2, Issue CSCW

November 2018

4104 pages

EISSN:2573-0142

DOI:10.1145/3290265

Editors:
Karrie Karahalios
University of Illinois & Adobe
,
Andrés Monroy-Hernández
Snap Inc.
,
Airi Lampinen
Stockholm University
,
Geraldine Fitzpatrick
Vienna University of Technology

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2018

Published in PACMHCI Volume 2, Issue CSCW

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

US National Historical Publications and Records Commission

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

108
Total Citations
View Citations
280
Total Downloads

Downloads (Last 12 months)22
Downloads (Last 6 weeks)1

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wu JWang CHuang DNing JXie L(2025)Heart Rate Variability Estimation Based on RFID Tag-Pair in Dynamic EnvironmentsACM Transactions on Computing for Healthcare10.1145/36913556:1(1-26)Online publication date: 23-Jan-2025
https://dl.acm.org/doi/10.1145/3691355
Wu JLuo XChen SLyu YJi XMeng YWang D(2025)Continuous Authentication via Wrist Photoplethysmogram: An Extensive StudyIEEE Transactions on Mobile Computing10.1109/TMC.2024.344320824:2(525-538)Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1109/TMC.2024.3443208
Xu ZJin WYao CLiu XMa SLiu YQin ZVakilinia ILiu D(2024)EM-Rhythm: An Authentication Method for Heterogeneous IoT DevicesACM Transactions on Sensor Networks10.1145/370044120:6(1-21)Online publication date: 22-Nov-2024
https://dl.acm.org/doi/10.1145/3700441
Adler DYang YViranda TXu XMohr DVan Meter ATartaglia JJacobson NWang FEstrin DChoudhury T(2024)Beyond Detection: Towards Actionable Sensing Research in Clinical Mental HealthcareProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997558:4(1-33)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699755
Zhang JWang LLi YJiang JWang TTao X(2024)Measuring and Mining Community Evolution in Developer Social Networks with Entropy-Based IndicesACM Transactions on Software Engineering and Methodology10.1145/368883234:1(1-43)Online publication date: 30-Dec-2024
https://dl.acm.org/doi/10.1145/3688832
Xu XLiu XZhang HWang WNepal SSefidgar YSeo WKuehn KHuckins JMorris MNurius PRiskin EPatel SAlthoff TCampbell ADey AMankoff J(2024)GLOBEM: Cross-Dataset Generalization of Longitudinal Human Behavior ModelingGetMobile: Mobile Computing and Communications10.1145/3686138.368614728:2(23-30)Online publication date: 31-Jul-2024
https://dl.acm.org/doi/10.1145/3686138.3686147
Bedmutha MKarumudi SPatrick KRataj HWeibel NKostakos VKay JHoang T(2024)Exploring User Willingness towards Mobile Sensing and Intervention: A Case Study on Mental Health of Undergraduate College StudentsCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678422(721-728)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678422
Adler DChoudhury TKostakos VKay JHoang T(2024)Ubiquitous Computing in Action: Infrastructure to Support Sensing and Mental Health Research in PracticeCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678372(301-306)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678372
Bhat AKulkarni NHusain SYadavalli AKaur JShukla AShelar MSeshadri V(2024)Speaking in Terms of Money: Financial Knowledge Acquisition via Speech Data GenerationACM Journal on Computing and Sustainable Societies10.1145/36637752:3(1-35)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1145/3663775
Englhardt ZMa CMorris MChang CXu XQin LMcDuff DLiu XPatel SIyer V(2024)From Classification to Clinical InsightsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596048:2(1-25)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659604
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents