David Gray Widder
James Herbsleb
ABSTRACT
How do software engineers identify and act on their ethical concerns? Past work examines how software practitioners navigate specific ethical principles such as “fairness”, but this narrows the scope of concerns to implementing pre-specified principles. In contrast, we report self-identified ethical concerns of 115 survey respondents and 21 interviewees across five continents and in non-profit, contractor, and non-tech firms. We enumerate their concerns – military, privacy, advertising, surveillance, and the scope of their concerns – from simple bugs to questioning their industry’s entire existence. We illustrate how attempts to resolve concerns are limited by factors such as personal precarity and organizational incentives. We discuss how even relatively powerful software engineers often lacked the power to resolve their ethical concerns. Our results suggest that ethics interventions must expand from helping practitioners merely identify issues to instead helping them build their (collective) power to resolve them, and that tech ethics discussions may consider broadening beyond foci on AI or Big Tech.
- 2020. A Tech Workers’ Bill of Rights. https://techworkerscoalition.org/bill-of-rights/. Tech Workers Coalition (2020). Accessed: 2021-010-1.Google Scholar
- Mike Allen. 2020. Palantir’s CEO Said He’s Suffered Because of His Contract with ICE. https://www.axios.com/2020/05/26/palantir-ceo-ice-immigration.Google Scholar
- Anat Alon-Beck. 2020. Times They Are a-Changin’: When Tech Employees Revolt!Md. L. Rev. 80 (2020), 120.Google Scholar
- Guilherme Avelino, Leonardo Passos, Andre Hora, and Marco Tulio Valente. 2016. A novel approach for estimating truck factors. In 2016 IEEE 24th International Conference on Program Comprehension (ICPC). IEEE, 1–10.Google ScholarCross Ref
- Haydn Belfield. 2020. Activism by the AI Community: Analysing Recent Achievements and Future Prospects. In AAAI/ACM Conference on AI, Ethics, and Society. ACM, New York NY USA, 15–21. https://doi.org/10.1145/3375627.3375814Google ScholarDigital Library
- Garfield Benjamin. 2022. #FuckTheAlgorithm: Algorithmic Imaginaries and Political Resistance. In 2022 ACM Conference on Fairness, Accountability, and Transparency. ACM, Seoul Republic of Korea, 46–57. https://doi.org/10.1145/3531146.3533072Google ScholarDigital Library
- Sam Biddle and Ryan Devereaux. 2019. Peter Thiel’s Palantir Used to Bust Migrant Children’s Relatives. https://theintercept.com/2019/05/02/peter-thiels-palantir-was-used-to-bust-hundreds-of-relatives-of-migrant-children-new-documents-show/.Google Scholar
- Abeba Birhane, Elayne Ruane, Thomas Laurent, Matthew S. Brown, Johnathan Flowers, Anthony Ventresque, and Christopher L. Dancy. 2022. The Forgotten Margins of AI Ethics. In 2022 ACM Conference on Fairness, Accountability, and Transparency. ACM, Seoul Republic of Korea, 948–958. https://doi.org/10.1145/3531146.3533157Google ScholarDigital Library
- Jason Borenstein and Ayanna Howard. 2021. Emerging Challenges in AI and the Need for AI Ethics Education. AI and Ethics 1, 1 (Feb. 2021), 61–65. https://doi.org/10.1007/s43681-020-00002-7Google ScholarCross Ref
- Virginia Braun and Victoria Clarke. 2012. Thematic analysis. (2012).Google Scholar
- Enda Brophy. 2006. System Error: Labour Precarity and Collective Organizing at Microsoft. Canadian Journal of Communication 31, 3 (Oct. 2006), 619–638. https://doi.org/10.22230/cjc.2006v31n3a1767Google ScholarCross Ref
- Margaret Burnett, Simone Stumpf, Jamie Macbeth, Stephann Makri, Laura Beckwith, Irwin Kwan, Anicia Peters, and William Jernigan. 2016. GenderMag: A Method for Evaluating Software’s Gender Inclusiveness. Interacting with Computers 28, 6 (Nov. 2016), 760–787. https://doi.org/10.1093/iwc/iwv046Google ScholarCross Ref
- Kelley Changfong-Hagen. 2020. " Don’t Be Evil": Collective Action and Employee Prosocial Activism. HRLR Online 5 (2020), 188.Google Scholar
- Peter Cihon, Jonas Schuett, and Seth D. Baum. 2021. Corporate Governance of Artificial Intelligence in the Public Interest. Information 12, 7 (July 2021), 275. https://doi.org/10.3390/info12070275Google ScholarCross Ref
- Jenny L Davis. 2020. How artifacts afford: The power and politics of everyday things. MIT Press.Google Scholar
- Julia Dressel and Hany Farid. 2018. The Accuracy, Fairness, and Limits of Predicting Recidivism. Science Advances 4, 1 (Jan. 2018), eaao5580. https://doi.org/10.1126/sciadv.aao5580Google ScholarCross Ref
- Avi Feller, Emma Pierson, Sam Corbett-Davies, and Sharad Goel. 2016. A computer program used for bail and sentencing decisions was labeled biased against blacks. It’s actually not that clear. The Washington Post 17 (2016).Google Scholar
- Casey Fiesler, Natalie Garrett, and Nathan Beard. 2020. What Do We Teach When We Teach Tech Ethics?: A Syllabi Analysis. In 51st ACM Technical Symposium on Computer Science Education. ACM, Portland OR USA, 289–295. https://doi.org/10.1145/3328778.3366825Google ScholarDigital Library
- Ben Gansky and Sean McDonald. 2022. CounterFAccTual: How FAccT undermines its organizing principles. In 2022 ACM Conference on Fairness, Accountability, and Transparency. 1982–1992.Google ScholarDigital Library
- Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021. Datasheets for datasets. Commun. ACM 64, 12 (2021), 86–92.Google ScholarDigital Library
- Marcela F González. 2022. Precarity for the global talent: The impact of visa policies on high-skilled immigrants’ work in the United States. International Migration 60, 2 (2022), 193–207.Google ScholarCross Ref
- DW Gotterbarn, Bo Brinkman, Catherine Flick, Michael S Kirkpatrick, Keith Miller, Kate Vazansky, and Marty J Wolf. 2018. ACM code of ethics and professional conduct. (2018).Google Scholar
- Colin M Gray, Yubo Kou, Bryan Battles, Joseph Hoggatt, and Austin L Toombs. 2018. The dark (patterns) side of UX design. In 2018 CHI conference on human factors in computing systems. 1–14.Google ScholarDigital Library
- Daniel Greene, Anna Lauren Hoffmann, and Luke Stark. 2019. Better, nicer, clearer, fairer: A critical assessment of the movement for ethical artificial intelligence and machine learning. In 52nd Hawaii international conference on system sciences.Google ScholarCross Ref
- Philip J Guo, Thomas Zimmermann, Nachiappan Nagappan, and Brendan Murphy. 2010. Characterizing and predicting which bugs get fixed: an empirical study of microsoft windows. In 32Nd ACM/IEEE International Conference on Software Engineering-Volume 1. 495–504.Google ScholarDigital Library
- Karen Hao. 2019. In 2020, let’s stop AI ethics-washing and actually do something. MIT Technology Review 27, December (2019), 2019.Google Scholar
- Thomas A. Hemphill. 2019. ‘Techlash’, Responsible Innovation, and the Self-Regulatory Organization. Journal of Responsible Innovation 6, 2 (May 2019), 240–247. https://doi.org/10.1080/23299460.2019.1602817Google ScholarCross Ref
- Ralph Hertwig, Carola Fanselow, and Ulrich Hoffrage. 2003. Hindsight bias: How knowledge and heuristics affect our reconstruction of the past. Memory (2003).Google Scholar
- Claudia Hilderbrand, Christopher Perdriau, Lara Letaw, Jillian Emard, Zoe Steine-Hanson, Margaret Burnett, and Anita Sarma. 2020. Engineering Gender-Inclusivity into Software: Ten Teams’ Tales from the Trenches. In ACM/IEEE 42nd International Conference on Software Engineering. ACM, Seoul South Korea, 433–444. https://doi.org/10.1145/3377811.3380371Google ScholarDigital Library
- Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. 2019. Improving fairness in machine learning systems: What do industry practitioners need?. In 2019 CHI conference on human factors in computing systems. 1–16.Google ScholarDigital Library
- Lilly Irani. 2019. Chasing Innovation: Making Entrepreneurial Citizens in Modern India. Princeton University Press. https://doi.org/10.1515/9780691189444Google Scholar
- Maurice Jakesch, Zana Buçinca, Saleema Amershi, and Alexandra Olteanu. 2022. How Different Groups Prioritize Ethical Values for Responsible AI. In 2022 ACM Conference on Fairness, Accountability, and Transparency(FAccT ’22). Association for Computing Machinery, New York, NY, USA, 310–323. https://doi.org/10.1145/3531146.3533097Google ScholarDigital Library
- Anna Jobin, Marcello Ienca, and Effy Vayena. 2019. The global landscape of AI ethics guidelines. Nature Machine Intelligence 1, 9 (2019), 389–399.Google ScholarCross Ref
- Jason Karaian and Lora Kelley. 2023. How Big Tech Layoffs Stack Up With the Rest of Their Work Forces. https://www.nytimes.com/2023/01/21/business/tech-layoffs.html.Google Scholar
- Os Keyes, Jevan Hutson, and Meredith Durbin. 2019. A mulching proposal: Analysing and improving an algorithmic system for turning the elderly into high-nutrient slurry. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1–11.Google ScholarDigital Library
- P. M. Krafft, Meg Young, Michael Katell, Karen Huang, and Ghislain Bugingo. 2020. Defining AI in Policy versus Practice. In AAAI/ACM Conference on AI, Ethics, and Society. ACM, New York NY USA, 72–78. https://doi.org/10.1145/3375627.3375835Google ScholarDigital Library
- Logan Kugler. 2021. The unionization of technology companies. Commun. ACM 64, 8 (2021), 18–20.Google ScholarDigital Library
- Min Kyung Lee, Daniel Kusbit, Evan Metsky, and Laura Dabbish. 2015. Working with machines: The impact of algorithmic and data-driven management on human workers. In 33rd annual ACM conference on human factors in computing systems. 1603–1612.Google Scholar
- Michelle Seng Ah Lee and Jat Singh. 2021. The landscape and gaps in open source fairness toolkits. In 2021 CHI conference on human factors in computing systems. 1–13.Google ScholarDigital Library
- Zachary C Lipton. 2018. The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery.Queue 16, 3 (2018), 31–57.Google ScholarDigital Library
- Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, and Hanna Wallach. 2022. Assessing the Fairness of AI Systems: AI Practitioners’ Processes, Challenges, and Needs for Support. ACM Conference on Human-Computer Interaction 6, CSCW1 (2022), 1–26.Google Scholar
- Michael A Madaio, Luke Stark, Jennifer Wortman Vaughan, and Hanna Wallach. 2020. Co-designing checklists to understand organizational challenges and opportunities around fairness in ai. In 2020 CHI Conference on Human Factors in Computing Systems. 1–14.Google ScholarDigital Library
- Andrew McNamara, Justin Smith, and Emerson Murphy-Hill. 2018. Does ACM’s code of ethics change ethical decision making in software development?. In 2018 26th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering. 729–733.Google Scholar
- Jacob Metcalf, Emanuel Moss, 2019. Owning ethics: Corporate logics, silicon valley, and the institutionalization of ethics. Social Research: An International Quarterly 86, 2 (2019), 449–476.Google ScholarCross Ref
- Debra E Meyerson. 2008. Rocking the boat: How tempered radicals effect change without making trouble. Harvard Business Review Press.Google Scholar
- Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In conference on fairness, accountability, and transparency. 220–229.Google ScholarDigital Library
- Brent Mittelstadt. 2019. Principles alone cannot guarantee ethical AI. Nature Machine Intelligence 1, 11 (2019), 501–507.Google ScholarCross Ref
- Anton J Nederhof. 1985. Methods of coping with social desirability bias: A review. European journal of social psychology (1985).Google ScholarCross Ref
- Nataliya Nedzhvetskaya, JS Tan, Hyatt Dirbas, and Wynnie Chan. 2022. Collective Action in Tech. https://data.collectiveaction.tech/.Google Scholar
- Nataliya Nedzhvetskaya and J. S. Tan. 2022. The Role of Workers in AI Ethics and Governance. In The Oxford Handbook of AI Governance (first ed.), Justin B. Bullock, Yu-Che Chen, Johannes Himmelreich, Valerie M. Hudson, Anton Korinek, Matthew M. Young, and Baobao Zhang (Eds.). Oxford University Press, C68.S1–C68.N14. https://doi.org/10.1093/oxfordhb/9780197579329.013.68Google Scholar
- Gina Neff. 2020. From Bad Users and Failed Uses to Responsible Technologies: A Call to Expand the AI Ethics Toolkit. In AAAI/ACM Conference on AI, Ethics, and Society. ACM, New York NY USA, 5–6. https://doi.org/10.1145/3375627.3377141Google ScholarDigital Library
- Rodrigo Ochigame. 2019. How Big Tech Manipulates Academia to Avoid Regulation. https://theintercept.com/2019/12/20/mit-ethical-ai-artificial-intelligence/.Google Scholar
- Will Orr and Jenny L Davis. 2020. Attributions of ethical responsibility by Artificial Intelligence practitioners. Information, Communication & Society 23, 5 (2020), 719–735.Google ScholarCross Ref
- Hyanghee Park, Daehwan Ahn, Kartik Hosanagar, and Joonhwan Lee. 2022. Designing Fair AI in Human Resource Management: Understanding Tensions Surrounding Algorithmic Evaluation and Envisioning Stakeholder-Centered Solutions. In CHI Conference on Human Factors in Computing Systems. 1–22.Google Scholar
- Hatim A. Rahman. 2021. The Invisible Cage: Workers’ Reactivity to Opaque Algorithmic Evaluations. Administrative Science Quarterly 66, 4 (Dec. 2021), 945–988. https://doi.org/10.1177/00018392211010118Google ScholarCross Ref
- Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury. 2020. Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices. In 24th ACM Conference on Computer-Supported Cooperative Work and Social Computing (2020).Google Scholar
- Brianna Richardson, Jean Garcia-Gathright, Samuel F. Way, Jennifer Thom, and Henriette Cramer. 2021. Towards Fairness in Practice: A Practitioner-Oriented Rubric for Evaluating Fair ML Toolkits. In 2021 CHI Conference on Human Factors in Computing Systems. ACM, Yokohama Japan, 1–13. https://doi.org/10.1145/3411764.3445604Google ScholarDigital Library
- Marc J Riemer. 2007. Communication skills for the 21st century engineer. Global J. of Engng. Educ 11, 1 (2007), 89–100.Google Scholar
- Henrik Skaug Sætra, Mark Coeckelbergh, and John Danaher. 2022. The AI Ethicist’s Dilemma: Fighting Big Tech by Supporting Big Tech. AI and Ethics 2, 1 (Feb. 2022), 15–27. https://doi.org/10.1007/s43681-021-00123-7Google ScholarCross Ref
- Niloufar Salehi, Lilly C Irani, Michael S Bernstein, Ali Alkhatib, Eva Ogbe, and Kristy Milland. 2015. We are dynamo: Overcoming stalling and friction in collective action for crowd workers. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. 1621–1630.Google ScholarDigital Library
- Reginald G Smart. 1966. Subject selection bias in psychological research.Canadian Psychologist/Psychologie canadienne (1966).Google Scholar
- Norman Makoto Su, Amanda Lazar, and Lilly Irani. 2021. Critical Affects: Tech Work Emotions Amidst the Techlash. ACM Conference on Human-Computer Interaction 5, CSCW1 (2021), 1–27.Google ScholarDigital Library
- Suzanne L Thomas. 2019. Migration versus management: the global distribution of computer vision engineering work. In 2019 ACM/IEEE 14th International Conference on Global Software Engineering (ICGSE). IEEE, 12–17.Google ScholarCross Ref
- Ville Vakkuri, Kai-Kristian Kemell, Marianna Jantunen, and Pekka Abrahamsson. 2020. “This is Just a Prototype”: How Ethics Are Ignored in Software Startup-Like Environments. In Agile Processes in Software Engineering and Extreme Programming, Viktoria Stray, Rashina Hoda, Maria Paasivaara, and Philippe Kruchten (Eds.). Vol. 383. Springer International Publishing, Cham, 195–210. https://doi.org/10.1007/978-3-030-49392-9_13 Series Title: Lecture Notes in Business Information Processing.Google Scholar
- Michael Veale, Max Van Kleek, and Reuben Binns. 2018. Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making. In 2018 chi conference on human factors in computing systems. 1–14.Google ScholarDigital Library
- Ben Wagner. 2018. Ethics as an escape from regulation. From “ethics-washing” to ethics-shopping? (2018).Google Scholar
- Robert Philip Weber. 1990. Basic content analysis. Number 49. Sage.Google Scholar
- Robert S Weiss. 1995. Learning from strangers: The art and method of qualitative interview studies. Simon and Schuster.Google Scholar
- Meredith Whittaker. 2021. The steep cost of capture. Interactions 28, 6 (2021), 50–55.Google ScholarDigital Library
- David Gray Widder, Dawn Nafus, Laura Dabbish, and James Herbsleb. 2022. Limits and Possibilities for “Ethical AI” in Open Source: A Study of Deepfakes. In conference on fairness, accountability, and transparency.Google Scholar
- Richmond Y Wong. 2021. Tactics of Soft Resistance in User Experience Professionals’ Values Work. ACM Conference on Human-Computer Interaction 5, CSCW2 (2021), 1–28.Google Scholar
- Richmond Y. Wong, Michael A. Madaio, and Nick Merrill. 2023. Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethics. ACM Conference on Human-Computer Interaction 7, CSCW1 (April 2023), 1–27. https://doi.org/10.1145/3579621Google ScholarDigital Library
- Karen Yeung, Andrew Howes, and Ganna Pogrebna. 2019. AI governance by human rights-centred design, deliberation and oversight: An end to ethics washing. The Oxford Handbook of AI Ethics, Oxford University Press (2019) (2019).Google Scholar
- Meg Young, Michael Katell, and P.M. Krafft. 2022. Confronting Power and Corporate Capture at the FAccT Conference. In 2022 ACM Conference on Fairness, Accountability, and Transparency. ACM, Seoul Republic of Korea, 1375–1386. https://doi.org/10.1145/3531146.3533194Google ScholarDigital Library
- Thomas Zimmermann. 2016. Card-sorting: From text to themes. In Perspectives on data science for software engineering. Elsevier, 137–141.Google ScholarCross Ref
- J Zunger. 2018. Computer science faces an ethics crisis. The Cambridge Analytica scandal proves it. Boston Globe 22 (2018).Google Scholar
Recommendations
Managing emergent ethical concerns for software engineering in society
ICSE '15: Proceedings of the 37th International Conference on Software Engineering - Volume 2This paper presents an initial framework for managing emergent ethical concerns during software engineering in society projects. We argue that such emergent considerations can neither be framed as absolute rules about how to act in relation to fixed and ...
Ethical responsibilities: the smart card engineer
ETHICS '14: Proceedings of the IEEE 2014 International Symposium on Ethics in Engineering, Science, and TechnologyAlbert Einstein in one of his philosophical notes stated: "I do not believe in immortality of the individual, and I consider ethics to be an exclusively human concern with no superhuman authority behind it". Arguably, ethics does not necessarily mean ...
An Ethical Can of Worms for Software Certifiers
Society and the software industry are well served if certifiers do their job ethically. Thus it behooves everyone to, as much as possible, create an environment that encourages ethical behavior while identifying and discouraging unethical behavior ...
Comments