research-article

Human intervention in automated decision-making: Toward the construction of contestable systems

Author:

Marco AlmadaAuthors Info & Claims

ICAIL '19: Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law

Pages 2 - 11

https://doi.org/10.1145/3322640.3326699

Published: 17 June 2019 Publication History

Abstract

Concerns about "black box" machine learning algorithms have influenced why modern data protection laws and regulations on their establishment of a right to human intervention on decision-making supported by artificial intelligence. Such interventions provide data subjects with means to protect their rights, freedoms, and legitimate interests, either as a bare minimum requirement for data processing or as a central norm governing decision-aiding artificial intelligence. In this paper, I present contestability by design as an approach to two kinds of issues with current legal implementations of the right to human intervention. The first kind is the uncertainty about what kind of decision should be covered by this right: should intervention be restricted to those decisions with no human involvement, or should it be interpreted in a broader sense, encompassing all decisions that are effectively shaped by automated processing? The second class of issues ensues from practical limitations of this right to intervention: even within a clear conceptual framework, data subjects might still lack the information they need to the concrete exercise of their right, or the human intervention itself might introduce biases and limitations that result in undesirable outcomes. After discussing how those effects can be identified and measured, I then advance the thesis that proper protection of the rights of data subjects is feasible only if there are means for contesting decisions based solely on automated processing is not an afterthought, but instead a requirement at each stage of an artificial intelligence system's lifecycle.

References

[1]

Reuben Binns, Max Van Kleek, Michael Veale, Ulrik Lyngs, Jun Zhao, and Nigel Shadbolt. 2018. 'it's reducing a human being to a percentage': perceptions of justice in algorithmic decisions. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 377.

Digital Library

[2]

Kiel Brennan-Marquez and Stephen E Henderson. 2019. Artificial intelligence and role-reversible judgment. Journal of Criminal Law and Criminology. Forthcoming. Available at SSRN:

[3]

Jason Brennan. 2017. Against Democracy. Princeton University Press.

[4]

Maja Brkan. 2018. Do algorithms rule the world? algorithmic decision-making in the framework of the GDPR and beyond. In Terminator or the Jetsons? The Economics and Policy Implications of Artificial Intelligence. Technology Policy Institute. Washington, (February 2018).

[5]

Joanna J Bryson. 2016. Patiency is not a virtue: AI and the design of ethical systems. In 2016 AAAI Spring Symposium Series.

[6]

Joanna J Bryson, Mihailis E Diamantis, and Thomas D Grant. 2017. Of, for, and by the people: the legal lacuna of synthetic persons. Artificial Intelligence and Law, 25, 3, 273--291.

Digital Library

[7]

Jenna Burrell. 2016. How the machine 'thinks': understanding opacity in machine learning algorithms. Big Data & Society, 3, 1.

[8]

Bryan H Choi. 2019. Crashworthy code. Washington Law Review, 94, 1.

[9]

Márcio Cots and Ricardo Oliveira. 2018. Lei Geral de Proteção de Dados Pessoais Comentada. Thomson Reuters Brasil, São Paulo.

[10]

Bo Cowgill and Catherine Tucker. 2017. Algorithmic Bias: A Counterfactual Perspective. Working Paper. NSF Trustworthy Algorithms.

[11]

Janet Davis. 2009. Design methods for ethical persuasive computing. In Proceedings of the 4th International Conference on Persuasive Technology. ACM, 6.

Digital Library

[12]

Lilian Edwards and Michael Veale. 2018. Enslaving the algorithm: from a "right to an explanation" to a "right to better decisions"? IEEE Security & Privacy, 16, 3, 46--54.

[13]

Marion Fourcade and Kieran Healy. 2016. Seeing like a market. Socio-Economic Review, 15, 1, 9--29.

[14]

Bryce W Goodman. 2016. Economic models of (algorithmic) discrimination. In 29th Conference on Neural Information Processing Systems. Volume 6.

[15]

Mireille Hildebrandt. 2019. Privacy as protection of the incomputable self: from agnostic to agonistic machine learning. Theoretical Inquiries of Law, 20, 1.

[16]

Tad Hirsch, Kritzia Merced, Shrikanth Narayanan, Zac E Imel, and David C Atkins. 2017. Designing contestability: interaction design, machine learning, and mental health. In Proceedings of the 2017 Conference on Designing Interactive Systems. ACM, 95--99.

Digital Library

[17]

J-H Hoepman. 2018. Making Privacy By Design Concrete. Technical report. KPN CISO Office, The Hague.

[18]

Lu Hong and Scott E Page. 2004. Groups of diverse problem solvers can outperform groups of high-ability problem solvers. PNAS, 101, 46, 16385--16389.

[19]

Dimitra Kamarinou, Christopher Millard, and Jatinder Singh. 2016. Machine Learning with Personal Data. Queen Mary School of Law Legal Studies Research Paper 247. Queen Mary, University of London, United Kingdom.

[20]

Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. 2017. Human decisions and machine predictions. The quarterly journal of economics, 133, 1, 237--293.

[21]

Joshua A Kroll, Solon Barocas, Edward W Felten, Joel R Reidenberg, David G Robinson, and Harlan Yu. 2016. Accountable algorithms. University of Pennsylvania Law Review., 165, 633.

[22]

David Lehr and Paul Ohm. 2017. Playing with the data: what legal scholars should learn about machine learning. UC Davis Law Review, 51, 653.

[23]

Isak Mendoza and Lee A Bygrave. 2017. The right not to be subject to automated decisions based on profiling. In EU Internet Law. Springer, 77--98.

[24]

Brent Mittelstadt, Chris Russell, and Sandra Wachter. 2019. Explaining explanations in AI. In FAT* '19: Conference on Fairness, Accountability, and Transparency. Atlanta, GA, USA.

Digital Library

[25]

Nathalie Nevejans. 2016. European civil law rules in robotics. Technical report. European Parliament, Brussels.

[26]

Michael Polanyi. 2009. The tacit dimension. University of Chicago Press.

[27]

Antoni Roig. 2018. Safeguards for the right not to be subject to a decision based solely on automated processing (article 22 GDPR). European Journal of Law and Technology, 8, 3.

[28]

Andrew D Selbst and Julia Powles. 2017. Meaningful information and the right to explanation. International Data Privacy Law, 7, 4, 233--242.

[29]

Ian Sommerville. 2011. Software Engineering. (9th edition). Pearson Education Ltd, London.

Digital Library

[30]

Christian B. Sundquist. 2019. Uncovering juror racial bias. Denver University Law Review, 96, 1.

[31]

Cass R Sunstein. 2019. Algorithms, correcting biases. Social Research, 86, 2. Forthcoming. SSRN draft.

[32]

Michaël Trazzi and Roman V Yampolskiy. 2018. Building safer AGI by introducing artificial stupidity. arXiv preprint arXiv:1808.03644.

[33]

WP21. 2018. Guidelines on Automated individual decision-making and Profiling for the purposes of Regulation 2016/679. Technical report. Article 29 Data Protection Working Party, (February 2018).

[34]

Tal Zarsky. 2016. The trouble with algorithmic decisions: an analytic road map to examine efficiency and fairness in automated and opaque decision making. Science, Technology, & Human Values, 41, 1, 118--132.

[35]

John Zerilli, Alistair Knott, James Maclaurin, and Colin Gavaghan. 2018. Transparency in algorithmic and human decision-making: is there a double standard? Philosophy & Technology, 1--23.

[36]

Baobao Zhang and Allan Dafoe. 2019. Artificial Intelligence: American Attitudes and Trends. Technical report. Center for the Governance of AI, Future of Humanity Institute, University of Oxford, (January 2019).

Cited By

Stephanidis CSalvendy GAntona MDuffy VGao QKarwowski WKonomi SNah FNtoa SRau PSiau KZhou J(2025)Seven HCI Grand Challenges Revisited: Five-Year ProgressInternational Journal of Human–Computer Interaction10.1080/10447318.2025.2450411(1-49)Online publication date: 4-Feb-2025
https://doi.org/10.1080/10447318.2025.2450411
Veluwenkamp HBuijsman S(2025)Design for operator contestability: control over autonomous systems by introducing defeatersAI and Ethics10.1007/s43681-025-00657-0Online publication date: 7-Feb-2025
https://doi.org/10.1007/s43681-025-00657-0
Lim JLim J(2025)Digital Sociocracy: A Decolonial Participatory GovernanceDigital Media Interventions in Southeast Asia10.1007/978-3-031-78211-4_5(193-256)Online publication date: 5-Mar-2025
https://doi.org/10.1007/978-3-031-78211-4_5
Show More Cited By

Index Terms

Human intervention in automated decision-making: Toward the construction of contestable systems

Recommendations

Reviewable Automated Decision-Making: A Framework for Accountable Algorithmic Systems
FAccT '21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency

This paper introduces reviewability as a framework for improving the accountability of automated and algorithmic decisionmaking (ADM) involving machine learning. We draw on an understanding of ADM as a socio-technical process involving both human and ...
From Human Intervention to Human Involvement: A Critical Examination of the Role of Humans in (Semi-)Automated Administrative Decision-Making
In view of recent scandals, evolving regulatory demands, and changing perceptions of human involvement in automated systems, this study examines how the role of humans in semi-automated administrative decision-making (AADM) is conceptualized in academic ...
AI-supported decision-making under the general data protection regulation
ICAIL '17: Proceedings of the 16th edition of the International Conference on Articial Intelligence and Law

The purpose of this paper is to analyse the rules of the General Data Protection Regulation on automated decision making in the age of Big Data and to explore how to ensure transparency of such decisions, in particular those taken with the help of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICAIL '19: Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law

June 2019

312 pages

ISBN:9781450367547

DOI:10.1145/3322640

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence

In-Cooperation

Univ. of Montreal: University of Montreal
AAAI
IAAIL: Intl Asso for Artifical Intel & Law

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICAIL '19

Sponsor:

SIGAI

ICAIL '19: Seventeenth International Conference on Artificial Intelligence and Law

June 17 - 21, 2019

QC, Montreal, Canada

Acceptance Rates

Overall Acceptance Rate 69 of 169 submissions, 41%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

45
Total Citations
View Citations
1,412
Total Downloads

Downloads (Last 12 months)271
Downloads (Last 6 weeks)39

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Stephanidis CSalvendy GAntona MDuffy VGao QKarwowski WKonomi SNah FNtoa SRau PSiau KZhou J(2025)Seven HCI Grand Challenges Revisited: Five-Year ProgressInternational Journal of Human–Computer Interaction10.1080/10447318.2025.2450411(1-49)Online publication date: 4-Feb-2025
https://doi.org/10.1080/10447318.2025.2450411
Veluwenkamp HBuijsman S(2025)Design for operator contestability: control over autonomous systems by introducing defeatersAI and Ethics10.1007/s43681-025-00657-0Online publication date: 7-Feb-2025
https://doi.org/10.1007/s43681-025-00657-0
Lim JLim J(2025)Digital Sociocracy: A Decolonial Participatory GovernanceDigital Media Interventions in Southeast Asia10.1007/978-3-031-78211-4_5(193-256)Online publication date: 5-Mar-2025
https://doi.org/10.1007/978-3-031-78211-4_5
Hai-Jew S(2024)Exploring Generative AI as Personally Effective Decision-Making ToolsEnhancing Automated Decision-Making Through AI10.4018/979-8-3693-6230-3.ch014(451-492)Online publication date: 29-Nov-2024
https://doi.org/10.4018/979-8-3693-6230-3.ch014
Sieber RBrandusescu ASangiambut SAdu-Daako A(2024)What is civic participation in artificial intelligence?Environment and Planning B: Urban Analytics and City Science10.1177/23998083241296200Online publication date: 1-Nov-2024
https://doi.org/10.1177/23998083241296200
Sieber RBrandusescu AAdu-Daako ASangiambut S(2024)Who are the publics engaging in AI?Public Understanding of Science10.1177/0963662523121985333:5(634-653)Online publication date: 28-Jan-2024
https://doi.org/10.1177/09636625231219853
Balayn APi YWidder DAlfrink KYurrita MUpadhyay SKarusala NLyons HTurkay CTessono CAttard-Frost BGadiraju UFarzan RLópez CCardoso Llach DQuercia DMustafa MNiu SWong-Villacrés M(2024)From Stem to Stern: Contestability Along AI Value ChainsCompanion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing10.1145/3678884.3681831(720-723)Online publication date: 11-Nov-2024
https://dl.acm.org/doi/10.1145/3678884.3681831
Alharbi RLor PHerskovitz JSchoenebeck SBrewer R(2024)Misfitting With AI: How Blind People Verify and Contest AI ErrorsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675659(1-17)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3663548.3675659
Atreja SIm JResnick PHemphill L(2024)AppealMod: Inducing Friction to Reduce Moderator Workload of Handling User AppealsProceedings of the ACM on Human-Computer Interaction10.1145/36372968:CSCW1(1-35)Online publication date: 26-Apr-2024
https://dl.acm.org/doi/10.1145/3637296
Corti LOltmans RJung JBalayn AWijsenbeek MYang J(2024)``It Is a Moving Process": Understanding the Evolution of Explainability Needs of Clinicians in Pulmonary MedicineProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642551(1-21)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642551
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten