The Opacity of Automated Decision-Making Systems (ADMS) and its Challenges for Political Legitimacy in a Democracy

This paper focuses specifically on Automated Decision-Making Systems (ADMS) based on Artificial Intelligence (AI). Since the last decades, AI systems are increasingly deployed by governments across the planet to manage public infrastructures and resources, as well as to engage with citizens for the provision of public services. Their introduction is advertised as a cost-cutting tool, as well as an instrument to combat traditional institutional disfunctions such as inefficiency, understaffing, corruption and human bias.

While AI offers an incredible potential for progress, an emerging body of literature highlights the challenges that AI-driven decision-making may raise for a public sector ethics. A common trait of these challenges is their being related to some form of "epistemological opacity" that undermines the capacity of humans to explain and justify decisions based on AI systems, detect errors or unfairness and adopt corrective actions. The situation may entail public officers and citizens taking the outcomes of AI systems at face value, thus basing their actions (wholly or in part) on pieces of information that cannot be scrutinized and/or corrected if necessary.

This paper intends to contribute to an emerging but still underdeveloped trend in normative political theory that study how AI-driven decision-making is reshaping the conceptualization and assessment of interactions between citizens and public officials. The overall goal of the paper is to analyze how various sources of "epistemological opacity" (algorithmic/legal/illiteracy/discursive) affecting AI systems, may undermine the democratic legitimacy of public decisions based on them.

Broadly speaking, legitimacy is the property that grounds the exercise of political authority, where authority standardly means the right to rule [1]. In this paper, democratic legitimacy is understood as a distinctive form of political authority grounded in the recognition of citizens as joint legislators. The paper offers a conception of democratic legitimacy conditional on the capacity of decision-making procedures and outcomes to realize the principle of public equality, which requires citizens' control over public decision-making, as well as respect for their equal status as political decision-makers.

Specifically, the paper argues that the "epistemological opacity" affecting AI-driven decision-making systems, brings about a mistreatment of citizens as coauthors of public decisions, which is a premise of the idea of democratic citizenship.

The main conjecture is that different sources of "epistemological opacity" (algorithmic/legal/illiteracy/discursive) are causing the disengagement of citizens and public officers from public decision-making, either because they directly undermine necessary conditions for the realization of public equality (co-authorship/accountability/publicity), or because they hide from the public eye instances of illegitimate automation and privatization of decisional power.

The paper offers a normative conception of democratic legitimacy that may contribute to efforts in various fields, including "AI fairness" and "Explainable AI", to better adapt technological tools to equality requirements distinctive of public decision-making within democratic societies.