skip to main content
10.1145/3643691.3648583acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
research-article

Responsible AI Engineering: The Case of an Inclusive Image Annotation Team in a Global Technology Company

Published: 29 July 2024 Publication History

Abstract

Artificial Intelligence (AI) models require a high quantity of high-quality data to learn from. For tolling applications, traffic cameras can collect massive amounts of images. However, annotating them is expensive, time-consuming, and requires distinct skills. People on the autism spectrum, on the other hand, typically have a hard time finding work, especially fulfilling work that benefits from their skills. Based on six interviews, we present the case of an inclusive traffic image annotation team in a global technology company that bridges the gap between the growing demand for high-quality image annotation and the difficult employment situation of people with autism. In particular, we explore relevant use cases in the pilot project, what requirements can be identified to improve software tool support, and what the enabling factors are in general. The results indicate that this initiative has a positive impact along many dimensions. The overall goal of this paper is to provide researchers and practitioners with a blueprint for creating inclusive AI annotation workplaces-with modifications where necessary.

References

[1]
DSMTF American Psychiatric Association, American Psychiatric Association, et al. 2013. Diagnostic and statistical manual of mental disorders: DSM-5. Vol. 5. American psychiatric association Washington, DC.
[2]
Hala Annabi, Karthika Sundaresan, and Annuska Zolyomi. 2017. It's Not Just About Attention to Details: Redefining the Talents Autistic Software Developers Bring to Software Development.
[3]
Andrew Begel, John Tang, Sean Andrist, Michael Barnett, Tony Carbary, Piali Choudhury, Edward Cutrell, Alberto Fung, Sasa Junuzovic, Daniel McDuff, Kael Rowan, Shibashankar Sahoo, Jennifer Frances Waldern, Jessica Wolk, Hui Zheng, and Annuska Zolyomi. 2020. Lessons Learned in Designing AI for Autistic Adults. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility. ACM, Virtual Event Greece, 1--6.
[4]
Elin Björling and Emma Rose. 2019. Participatory Research Principles in Human-Centered Design: Engaging Teens in the Co-Design of a Social Robot. Multimodal Technologies and Interaction 3, 1 (Feb. 2019), 8.
[5]
Gordon Calleja. 2011. In-Game: From Immersion to Incorporation. The MIT Press.
[6]
Flavia Chiarotti and Aldina Venerosi. 2020. Epidemiology of autism spectrum disorders: a review of worldwide prevalence estimates since 2014. Brain sciences 10, 5 (2020), 274.
[7]
Marc Fabri and Penny C.S. Andrews. 2016. Human-Centered Design with Autistic University Students: Interface, Interaction and Information Preferences. Lecture Notes in Computer Science, Vol. 9747. Springer International Publishing, Cham, 157--166.
[8]
Kiev Gama and Aline Lacerda. 2023. Understanding and Supporting Neurodiverse Software Developers in Agile Teams. In Proceedings of the XXXVII Brazilian Symposium on Software Engineering. ACM, Campo Grande Brazil, 497--502.
[9]
Elizabeth Garrison, Dalvir Singh, Donald Hantula, Matt Tincani, John Nosek, Sungsoo Ray Hong, Eduard Dragut, and Slobodan Vucetic. 2023. Understanding the experience of neurodivergent workers in image and text data annotation. Computers in Human Behavior Reports 11 (Aug. 2023), 100318.
[10]
Mary L Gray and Siddharth Suri. 2019. Ghost work: How to stop Silicon Valley from building a new global underclass. Eamon Dolan Books.
[11]
Kotaro Hara and Jeffrey P Bigham. 2017. Introducing people with ASD to crowd work. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility. 42--51.
[12]
Terrell D. Holmes and Hala Annabi. 2020. The Dark Side of Software Development: Job Stress amongst Autistic Software Developers. In Hawaii International Conference on System Sciences. https://api.semanticscholar.org/CorpusID:211533622
[13]
Nicola Martin, Christopher Barnham, and Joanna Krupa. 2019. Identifying and addressing barriers to employment of autistic adults. Journal of Inclusive Practice in Further and Higher Education 10, 1 (2019), 56--77.
[14]
Tolou Maslahati, Christian J Bachmann, Juliana Höfer, Charlotte Küpper, Sanna Stroth, Nicole Wolff, Luise Poustka, Veit Roessner, Inge Kamp-Becker, Falk Hoffmann, et al. 2022. How do adults with autism spectrum disorder participate in the labor market? A German multi-center survey. Journal of autism and developmental disorders (2022), 1--11.
[15]
Alisha Ohl, Mira Grice Sheff, Sarah Small, Jamie Nguyen, Kelly Paskor, and Aliza Zanjirian. 2017. Predictors of employment status among adults with Autism Spectrum Disorder. Work 56, 2 (2017), 345--355.
[16]
Billy Perrigo. 2023. Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic. Last accessed 19 (2023).
[17]
Per Runeson and Martin Höst. 2009. Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering 14, 2 (Apr 2009), 131--164.
[18]
Mohammad Tahaei, Marios Constantinides, Daniele Quercia, Sean Kennedy, Michael Muller, Simone Stumpf, Q Vera Liao, Ricardo Baeza-Yates, Lora Aroyo, Jess Holbrook, et al. 2023. Human-Centered Responsible Artificial Intelligence: Current & Future Trends. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems. 1--4.
[19]
Rebecca Tan and Regine Cabato. 2023. Behind the AI boom, an army of overseas workers in'digital sweatshops'. The Washington Post (2023), NA-NA.
[20]
Taylor T.L. 2003. Power games just want to have fun?: instrumental play in a MMOG. In DiGRA &#3903 - Proceedings of the 2003 DiGRA International Conference: Level Up. http://www.digra.org/wp-content/uploads/digital-library/05163.32071.pdf
[21]
Video Games Europe. 2023. All about video games. https://www.videogameseurope.eu/wp-content/uploads/2023/08/Video-Games-Europe_Key-Facts-2022_FINAL.pdf accessed: 2023-12-10.
[22]
Mary Wagner, Lynn Newman, Renée Cameto, Phyllis Levine, and Nicolle Garza. 2006. An Overview of Findings from Wave 2 of the National Longitudinal Transition Study-2 (NLTS2). NCSER 2006-3004. National Center for Special Education Research (2006).
[23]
Ding Wang, Shantanu Prabhat, and Nithya Sambasivan. 2022. Whose AI Dream? In search of the aspiration in data annotation. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1--16.
[24]
R.K. Yin. 2012. Applications of Case Study Research. SAGE Publications. https://books.google.at/books?id=-1Y2J0sFaWgC

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
RAIE '24: Proceedings of the 2nd International Workshop on Responsible AI Engineering
April 2024
62 pages
ISBN:9798400705724
DOI:10.1145/3643691
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

  • Faculty of Engineering of University of Porto

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 July 2024

Check for updates

Author Tags

  1. autism
  2. image annotation
  3. artificial intelligence
  4. case study

Qualifiers

  • Research-article

Funding Sources

  • FFG

Conference

RAIE '24
Sponsor:

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 88
    Total Downloads
  • Downloads (Last 12 months)88
  • Downloads (Last 6 weeks)19
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media