extended-abstract

REGROW: Reimagining Global Crowdsourcing for Better Human-AI Collaboration

Authors:

Niels van Berkel,

Dmitry Ustalov,

Jonas Oppenlaender,

Danula Hettiachchi,

Ujwal Gadiraju,

Jorge Goncalves,

Simo HosioAuthors Info & Claims

CHI EA '22: CHI Conference on Human Factors in Computing Systems Extended Abstracts

Article No.: 88, Pages 1 - 7

https://doi.org/10.1145/3491101.3503725

Published: 28 April 2022 Publication History

Abstract

Crowdworkers silently enable much of today’s AI-based products, with several online platforms offering a myriad of data labelling and content moderation tasks through convenient labour marketplaces. The HCI community has been increasingly interested in investigating the worker-centric issues inherent in the current model and seeking for potential improvements that could be implemented in the future. This workshop explores how a reimagined perspective on crowdsourcing platforms could provide a more equitable, fair, and rewarding experience. This includes not only the workers but also the platforms, who could benefit e.g. from better processes for worker onboarding, skills-development, and growth. We invite visionary takes in various formats on this topic to spread awareness of worker-centric research and developments to the CHI community. As a result of interactive ideation work in the workshop, we articulate a future direction roadmap for research centred around crowdsourcing platforms. Finally, as a specific interest area, the workshop seeks to study crowdwork from the context of the Global South, which has been arising as an important but critically understudied crowdsourcing market in recent years.

References

[1]

Andy Alorwu, Saba Kheirinejad, Niels van Berkel, Marianne Kinnula, Denzil Ferreira, Aku Visuri, and Simo Hosio. 2021. Assessing MyData Scenarios: Ethics, Concerns, and the Promise. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–11.

Digital Library

[2]

Andy Alorwu, Aku Visuri, Niels van Berkel, and Simo Johannes Hosio. 2021. (Re) using Crowdsourced Health Data: Perceptions of Data Contributors. IEEE Software (2021).

[3]

Jeffrey Bardzell, Shaowen Bardzell, and Mark A Blythe. 2018. Critical theory and interaction design. MIT Press.

[4]

Oliver Bates, Christian Remy, Callum Nash, and Ben Kirman. 2019. The future of techno-disruption in gig economy workforces: challenging the dialogue with fictional abstracts. In Proceedings of the Halfway to the Future Symposium 2019. 1–4.

Digital Library

[5]

Michael S. Bernstein, Greg Little, Robert C. Miller, Björn Hartmann, Mark S. Ackerman, David R. Karger, David Crowell, and Katrina Panovich. 2010. Soylent: A Word Processor with a Crowd Inside. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (New York, New York, USA) (UIST ’10). Association for Computing Machinery, New York, NY, USA, 313–322. https://doi.org/10.1145/1866029.1866078

Digital Library

[6]

Jeffrey P. Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert C. Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, and Tom Yeh. 2010. VizWiz: Nearly Real-Time Answers to Visual Questions. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (New York, New York, USA) (UIST ’10). Association for Computing Machinery, New York, NY, USA, 333–342. https://doi.org/10.1145/1866029.1866080

Digital Library

[7]

Benedetta Catanzariti, Srrayva Chandhiramowuli, Suha Mohamed, Sarayu Natarajan, Shantanu Prabhat, Noopur Raval, Alex Taylor, and Ding Wang. 2021. The Global Labours of AI and Data Intensive Systems. In Companion of the 2021 ACM Conference on Computer Supported Cooperative Work and Social Computing(CSCW ’21 Companion). Association for Computing Machinery, New York, NY, USA, to appear.

[8]

Alan Chan, Chinasa T Okolo, Zachary Terner, and Angelina Wang. 2021. The Limits of Global Inclusion in AI Development. arXiv preprint arXiv:2102.01265(2021).

[9]

Angela Chen. 2021. Desperate Venezuelans are making money by training AI for self-driving cars. Mit Tech Review.

[10]

Chun-Wei Chiang, Eber Betanzos, and Saiph Savage. 2018. Exploring blockchain for trustful collaborations between immigrants and governments. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. 1–6.

Digital Library

[11]

Ben Chodor. 2020. Transitioning to Virtual and Hybrid Events: How to Create, Adapt, and Market an Engaging Online Experience. John Wiley & Sons.

[12]

Andy Crabtree, Terry Hemmings, Tom Rodden, Keith Cheverst, Karen Clarke, Guy Dewsbury, John Hughes, and Mark Rouncefield. 2003. Designing with care: Adapting cultural probes to inform design in sensitive settings. In Proceedings of the 2004 Australasian Conference on Computer-Human Interaction (OZCHI2004). 4–13.

[13]

Hettiachchi Danula, Sanderson Mark, Goncalves Jorge, Hosio Simo, Kazai Gabriella, Lease Matthew, Schaekermann Mike, and Emine Yilmaz. 2021. Investigating and Mitigating Biases in Crowdsourced Data. In 2021 Conference on Computer Supported Cooperative Work and Social Computing(CSCW ’21 Companion). Association for Computing Machinery, New York, NY, USA, 4 pages. https://doi.org/10.1145/3462204.3481729

[14]

Xuefei Deng, Kshiti D Joshi, and Robert D Galliers. 2016. The duality of empowerment and marginalization in microtask crowdsourcing: Giving voice to the less powerful through value sensitive design. Mis Quarterly 40, 2 (2016), 279–302.

[15]

Alexey Drutsa, Valentina Fedorova, Dmitry Ustalov, Olga Megorskaya, Evfrosiniya Zerminova, and Daria Baidakova. 2020. Crowdsourcing Practice for Efficient Data Labeling: Aggregation, Incremental Relabeling, and Pricing. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data(SIGMOD ’20). Association for Computing Machinery, Portland, OR, USA, 2623–2627. https://doi.org/10.1145/3318464.3383127

Digital Library

[16]

Alexey Drutsa, Valentina Fedorova, Dmitry Ustalov, Olga Megorskaya, Evfrosiniya Zerminova, and Daria Baidakova. 2020. Practice of Efficient Data Collection via Crowdsourcing: Aggregation, Incremental Relabelling, and Pricing. In Proceedings of the 13th International Conference on Web Search and Data Mining(WSDM ’20). Association for Computing Machinery, Houston, TX, USA, 873–876. https://doi.org/10.1145/3336191.3371875

Digital Library

[17]

Alexey Drutsa, Dmitry Ustalov, Valentina Fedorova, Olga Megorskaya, and Daria Baidakova. 2021. Crowdsourcing Natural Language Data at Scale: A Hands-On Tutorial. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorials(NAACL-HLT 2021). Association for Computational Linguistics, Online, 25–30. https://doi.org/10.18653/v1/2021.naacl-tutorials.6

[18]

Paul Du Gay, Stuart Hall, Linda Janes, Anders Koed Madsen, Hugh Mackay, and Keith Negus. 2013. Doing cultural studies: The story of the Sony Walkman. Sage.

[19]

Rosta Farzan, Saiph Savage, and Claudia Flores Saviaga. 2016. Bring on board new enthusiasts! A case study of impact of Wikipedia art+ feminism edit-a-thon events on newcomers. In International Conference on Social Informatics. Springer, 24–40.

Digital Library

[20]

Casey Fiesler. 2018. Owning the Servers: A Design Fiction Exploring the Transformation of Fandom into ‘Our Own’. Transformative Works and Cultures 28 (2018).

[21]

Claudia Flores-Saviaga and Saiph Savage. 2021. Fighting disaster misinformation in Latin America: the# 19S Mexican earthquake case study. Personal and Ubiquitous Computing 25, 2 (2021), 353–373.

Digital Library

[22]

Michelle Forelle, Philip N Howard, Andrés Monroy-Hernández, and Saiph Savage. 2015. Political bots and the manipulation of public opinion in Venezuela. Available at SSRN 2635800(2015).

[23]

Batya Friedman. 1996. Value-sensitive design. interactions 3, 6 (1996), 16–23.

[24]

Ujwal Gadiraju and Gianluca Demartini. 2019. Understanding Worker Moods and Reactions to Rejection in Crowdsourcing. In Proceedings of the 30th ACM Conference on Hypertext and Social Media (Hof, Germany) (HT ’19). Association for Computing Machinery, New York, NY, USA, 211–220. https://doi.org/10.1145/3342220.3343644

Digital Library

[25]

Frank W Geels. 2004. From sectoral systems of innovation to socio-technical systems: Insights about dynamics and change from sociology and institutional theory. Research policy 33, 6-7 (2004), 897–920.

[26]

Mary L Gray and Siddharth Suri. 2019. Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass. Eamon Dolan Books.

[27]

Rafael Grohmann and Willian F ARAÚJO. 2021. Beyond Mechanical Turk: the work of Brazilians on global AI platforms. VERDEGEM, Pieter. AI for everyone(2021), 247–266.

[28]

Simo Johannes Hosio, Niels van Berkel, Jonas Oppenlaender, and Jorge Goncalves. 2020. Crowdsourcing personalized weight loss diets. Computer 53, 1 (2020), 63–71. https://doi.org/10.1109/MC.2019.2902542

[29]

Lilly C. Irani and M. Six Silberman. 2013. Turkopticon: Interrupting Worker Invisibility in Amazon Mechanical Turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 611–620. https://doi.org/10.1145/2470654.2470742

Digital Library

[30]

Martin Jay. 1996. The dialectical imagination: A history of the Frankfurt School and the Institute of Social Research, 1923-1950. Vol. 10. Univ of California Press.

[31]

Phil Jones. 2021. Work Without the Worker: Labour in the Age of Platform Capitalism. Verso Books.

[32]

Zachary Kilhoffer. 2020. Platform Work and Participation: Disentangling the Rhetoric. In Using New Media for Citizen Engagement and Participation. IGI Global, 1–15.

[33]

Stephen Kline, Nick Dyer-Witheford, Greig De Peuter, 2003. Digital play: The interaction of technology, culture, and marketing. McGill-Queen’s Press-MQUP.

[34]

Joseph Lindley and Paul Coulton. 2015. Back to the future: 10 years of design fiction. In Proceedings of the 2015 British HCI Conference. 210–211.

Digital Library

[35]

Conor Linehan, Ben J Kirman, Stuart Reeves, Mark A Blythe, Theresa Jean Tanenbaum, Audrey Desjardins, and Ron Wakkary. 2014. Alternate endings: using fiction to explore design futures. In CHI’14 Extended Abstracts on Human Factors in Computing Systems. 45–48.

[36]

Herbert Marcuse. 1964. One-dimensional man: Studies in the ideology of advanced industrial society. Beacon Press.

[37]

Herbert Marcuse. 1966. The individual in the great society. Collected Papers of Herbert Marcuse Vol. 2: Towards a Critical Theory of Society (1966), 59–80.

[38]

Brian McInnis, Alissa Centivany, Juho Kim, Marta Poblet, Karen Levy, and Gilly Leshed. 2017. Crowdsourcing Law and Policy: A Design-Thinking Approach to Crowd-Civic Systems. In Companion of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW ’17 Companion). Association for Computing Machinery, New York, NY, USA, 355–361. https://doi.org/10.1145/3022198.3022656

Digital Library

[39]

Vishwajeet Narwal, Mohamed Hashim Salih, Jose Angel Lopez, Angel Ortega, John O’Donovan, Tobias Höllerer, and Saiph Savage. 2017. Automated assistants to identify and prompt action on visual news bias. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 2796–2801.

Digital Library

[40]

Linda Nilsson. 2020. Hybrid Events Breaking the Borders: Transferring your hybrid event into an engaging and inclusive experience for different audiences and stakeholders. (2020).

[41]

Jonas Oppenlaender, Maximilian Mackeprang, Abderrahmane Khiat, Maja Vuković, Jorge Goncalves, and Simo Hosio. 2019. DC2S2: Designing Crowd-Powered Creativity Support Systems. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–8. https://doi.org/10.1145/3290607.3299027

Digital Library

[42]

Jonas Oppenlaender, Kristy Milland, Aku Visuri, Panos Ipeirotis, and Simo Hosio. 2020. Creativity on paid crowdsourcing platforms. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems(CHI ’20). ACM, New York, NY, USA, Article 548, 14 pages. https://doi.org/10.1145/3313831.3376677

Digital Library

[43]

Jonas Oppenlaender, Naghmi Shireen, Maximilian Mackeprang, Halil Erhan, Jorge Goncalves, and Simo Hosio. 2019. Crowd-Powered Interfaces for Creative Design Thinking. In Proceedings of the 2019 on Creativity and Cognition (San Diego, CA, USA) (C&C ’19). Association for Computing Machinery, New York, NY, USA, 722–729. https://doi.org/10.1145/3325480.3326553

Digital Library

[44]

Julian Posada. 2021. The Coloniality of Data Work in Latin America. arXiv preprint arXiv:2105.06262(2021).

[45]

Uma Rani and Marianne Furrer. 2021. Digital labour platforms and new forms of flexible work in developing countries: Algorithmic management of work and workers. Competition & Change 25, 2 (2021), 212–236.

[46]

Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Walter S. Lasecki, Jay Patel, Negar Rahmati, Tulsee Doshi, Melissa Valentine, and Michael S. Bernstein. 2014. Expert Crowdsourcing with Flash Teams. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 75–85. https://doi.org/10.1145/2642918.2647409

Digital Library

[47]

Shruti Sannon and Dan Cosley. 2019. Privacy, Power, and Invisible Labor on Amazon Mechanical Turk. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300512

Digital Library

[48]

Saiph Savage, Andres Monroy-Hernandez, and Tobias Höllerer. 2016. Botivist: Calling volunteers to action using online bots. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. 813–822.

Digital Library

[49]

Eric Schatzberg. 2018. Technology: critical history of a concept. University of Chicago Press.

[50]

Florian Alexander Schmidt. 2019. Crowdsourced production of AI training data: how human workers teach self-driving cars how to see. Technical Report 155. Hans-Böckler-Stiftung, Düsseldorf, Germany. http://nbn-resolving.de/urn:nbn:de:101:1-2019102414513364122668

[51]

John Storey. 2018. Cultural theory and popular culture: An introduction. Routledge.

[52]

Dmitry Ustalov, Fabio Casati, Alexey Drutsa, and Daria Baidakova (Eds.). 2020. Crowd Science Workshop: Remoteness, Fairness, and Mechanisms as Challenges of Data Supply by Humans for Automation. Vancouver, BC, Canada (Online). http://ceur-ws.org/Vol-2736/

[53]

Dmitry Ustalov, Fabio Casati, Alexey Drutsa, Ivan Stelmakh, Nikita Pavlichenko, and Daria Baidakova (Eds.). 2021. Crowd Science Workshop: Trust, Ethics, and Excellence in Crowdsourced Data Management at Scale. Copenhagen, Denmark. http://ceur-ws.org/Vol-2932/

[54]

Niels van Berkel, Simo Hosio, Jorge Goncalves, Katarzyna Wac, Vassilis Kostakos, and Anna Cox. 2018. MHC ’18: International Workshop on Mobile Human Contributions: Opportunities and Challenges. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers (Singapore, Singapore) (UbiComp ’18). Association for Computing Machinery, New York, NY, USA, 590–593. https://doi.org/10.1145/3267305.3274134

Digital Library

[55]

PJ Wall, Deepak Saxena, and Suzana Brown. 2021. Artificial Intelligence in the Global South (AI4D): Potential and Risks. arXiv preprint arXiv:2108.10093(2021).

[56]

Robert West 2021. Summary of Tutorials at The Web Conference 2021. In Companion Proceedings of The Web Conference 2021(WWW ’21). Association for Computing Machinery, Ljubljana, Slovenia, 727–733. https://doi.org/10.1145/3442442.3453701

Digital Library

[57]

Richmond Y Wong, Ellen Van Wyk, and James Pierce. 2017. Real-fictional entanglements: Using science fiction and design fiction to interrogate sensing technologies. In Proceedings of the 2017 Conference on Designing Interactive Systems. 567–579.

Digital Library

Cited By

Boscovic DKhullar KOguri PPandey PEghbali MOlsen MDordevic M(2023)Tokenization of Digital Assets for Ethical and Equitable Model Training in an AI-Based Proctoring Application2023 International Conference on Computational Science and Computational Intelligence (CSCI)10.1109/CSCI62032.2023.00279(1691-1699)Online publication date: 13-Dec-2023
https://doi.org/10.1109/CSCI62032.2023.00279
Barbour JJensen JCall SSharma N(2023)Substance, discourse, and practice: a review of communication research on automationAnnals of the International Communication Association10.1080/23808985.2023.218323247:3(261-291)Online publication date: 2-Mar-2023
https://doi.org/10.1080/23808985.2023.2183232

Index Terms

REGROW: Reimagining Global Crowdsourcing for Better Human-AI Collaboration
1. Human-centered computing
  1. Human computer interaction (HCI)

Index terms have been assigned to the content through auto-classification.

Recommendations

Human Collaboration on Crowdsourcing Platforms – a Content Analysis
HCI in Business, Government and Organizations. Information Systems and Analytics
Abstract
The crowdsourcing phenomenon offers the opportunity to address an open call to the crowd. Crowd workers may work together to find a solution that satisfies the open call. One of the major benefits for a crowdsourcer is the pool of crowd workers ...
Crowdsourcing for information retrieval

The 2nd SIGIR Workshop on Crowdsourcing for Information Retrieval (CIR 2011) was held on July 28, 2011 in Beijing, China, in conjunction with the 34th Annual ACM SIGIR Conference1. The workshop brought together researchers and practitioners to ...
Crowdsourcing 101: putting the WSDM of crowds to work for you
WSDM '11: Proceedings of the fourth ACM international conference on Web search and data mining

Crowdsourcing has emerged in recent years as an exciting new avenue for leveraging the tremendous potential and resources of today's digitally-connected, diverse, distributed workforce. Generally speaking, crowdsourcing describes outsourcing of tasks to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems

April 2022

3066 pages

ISBN:9781450391566

DOI:10.1145/3491101

Editors:
Simone Barbosa
PUC-Rio, Brazil
,
Cliff Lampe
University of Michigan, USA
,
Caroline Appert
Université Paris-Saclay, France
,
David A. Shamma
Toyota Research Institute, USA

Copyright © 2022 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 April 2022

Check for updates

Qualifiers

Extended-abstract
Research
Refereed limited

Conference

CHI '22

Sponsor:

SIGCHI

CHI '22: CHI Conference on Human Factors in Computing Systems

April 29 - May 5, 2022

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
368
Total Downloads

Downloads (Last 12 months)83
Downloads (Last 6 weeks)11

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Boscovic DKhullar KOguri PPandey PEghbali MOlsen MDordevic M(2023)Tokenization of Digital Assets for Ethical and Equitable Model Training in an AI-Based Proctoring Application2023 International Conference on Computational Science and Computational Intelligence (CSCI)10.1109/CSCI62032.2023.00279(1691-1699)Online publication date: 13-Dec-2023
https://doi.org/10.1109/CSCI62032.2023.00279
Barbour JJensen JCall SSharma N(2023)Substance, discourse, and practice: a review of communication research on automationAnnals of the International Communication Association10.1080/23808985.2023.218323247:3(261-291)Online publication date: 2-Mar-2023
https://doi.org/10.1080/23808985.2023.2183232

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten