skip to main content
10.1145/3514094.3534163acmconferencesArticle/Chapter ViewAbstractPublication PagesaiesConference Proceedingsconference-collections
research-article
Public Access

Generating Deontic Obligations From Utility-Maximizing Systems

Published: 27 July 2022 Publication History

Abstract

This work gives a logical characterization of the (ethical and social) obligations of an agent trained with Reinforcement Learning (RL). An RL agent takes actions by following a utility-maximizing policy. We maintain that the choice of utility function embeds ethical and social values implicitly, and that it is necessary to make these values explicit. This work provides a basis for doing so. First, we propose a probabilistic deontic logic that is suited for formally specifying the obligations of a stochastic system, including its ethical obligations. We prove some useful validities about this logic, and how its semantics are compatible with those of Markov Decision Processes (MDPs). Second, we show that model checking allows us to prove that an agent has a given obligation to bring about some state of affairs - meaning that by acting optimally, it is seeking to reach that state of affairs. We develop a model checker for our logic against MDPs. Third, we observe that it is useful for a system designer to obtain a logical characterization of her system's obligations, which is potentially more interpretable and helpful in debugging than the expression of a utility function. Enumerating all the obligations of an agent is impractical, so we propose a Bayesian optimization routine that learns to generate a system's obligations that the system designer deems interesting. We implement the model checking and Bayesian optimization routines, and demonstrate their effectiveness with an initial pilot study. This work provides a rigorous method to characterize utility-maximizing agents in terms of the (ethical and social) obligations that they implicitly seek to satisfy.

Supplementary Material

MP4 File (AIES22-fp116.mp4)
If you had to evaluate the ethics embedded in the decision process of a self driving car, how would you do it? The car has to go from point A to point B, but what obligations arise from the controllers that drive it? Does the car have an obligation to sacrifice pedestrians to save its passengers? And how could we express that obligation in a human-legible form? Autonomous vehicles and robots like those that work in hospitals and nursing homes are faced with ethical decisions, but their ethics are hidden by their complexity. How can we extract the obligations of a robot and communicate those obligations to host communities? To answer these questions, we introduce a deontic logic for expressing the social norms of reinforcement learning systems, and an algorithm to check that a system has a given obligation. We also introduce an optimization procedure to generate obligations that are interesting to an evaluator; increasing the coverage of a system beyond those norms the evaluator thought to check.

References

[1]
Natasha Alechina, Nick Bassiliades, Mehdi Dastani, Marina De Vos, Brian Logan, Sergio Mera, Andreasa Morris-Martin, and Fernando Schapachnik. 2013. Computational models for normative multi-agent systems. In Dagstuhl Follow-Ups, Vol. 4. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.
[2]
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 (2016).
[3]
Konstantine Arkoudas, Selmer Bringsjord, and Paul Bello. 2005. Toward ethical robots via mechanized deontic logic. In AAAI fall symposium on machine ethics. The AAAI Press Menlo Park, CA, 17--23.
[4]
Christel Baier and Joost-Pieter Katoen. 2008. Principles of model checking. MIT press.
[5]
Richard Bellman. 1957. A Markovian Decision Process. Indiana Univ. Math. J. 6 (1957), 679--684. Issue 4.
[6]
Jan Broersen, Stephen Cranefield, Yehia Elrakaiby, Dov Gabbay, Davide Grossi, Emiliano Lorini, Xavier Parent, Leendert WN van der Torre, Luca Tummolini, Paolo Turrini, et al. 2013. Normative reasoning and consequence. In Dagstuhl Follow-Ups, Vol. 4. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.
[7]
B.F. Chellas. 1968. The Logical Form of Imperatives. Department of Philosophy, Stanford University.
[8]
Peter I Frazier. 2018. A tutorial on Bayesian optimization. arXiv preprint arXiv:1807.02811 (2018).
[9]
Dov Gabbay, John Horty, and Xavier Parent (Eds.). 2013. Handbook of deontic logic and normative systems. College Publications.
[10]
J Christian Gerdes and Sarah M Thornton. 2015. Implementable ethics for autonomous vehicles. In Autonomes fahren. Springer, 87--102.
[11]
Risto Hilpinen and Paul McNamara. 2013. Deontic Logic: A historical survey and introduction. College Publications, 3--136.
[12]
John Horty. 2001. Agency and Deontic Logic. Cambridge University Press.
[13]
M. Kwiatkowska, G. Norman, and D. Parker. 2011. PRISM 4.0: Verification of Probabilistic Real-time Systems. In Proc. 23rd International Conference on Computer Aided Verification (CAV'11) (LNCS, Vol. 6806), G. Gopalakrishnan and S. Qadeer (Eds.). Springer, 585--591.
[14]
Benjamin Letham, Brian Karrer, Guilherme Ottoni, and Eytan Bakshy. 2019. Constrained Bayesian optimization with noisy experiments. Bayesian Analysis 14, 2 (2019), 495--519.
[15]
Henry Moss, David Leslie, Daniel Beck, Javier Gonzalez, and Paul Rayson. 2020. Boss: Bayesian optimization over string spaces. Advances in neural information processing systems 33 (2020), 15476--15486.
[16]
Emery Neufeld, Ezio Bartocci, Agata Ciabattoni, and Guido Governatori. 2021. A Normative Supervisor for Reinforcement Learning Agents. Automated Deduction-- CADE 28 (2021), 565.
[17]
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. 2015. Taking the human out of the loop: A review of Bayesian optimization. Proc. IEEE 104, 1 (2015), 148--175.
[18]
Colin Shea-Blymyer and Houssam Abbas. 2021. Algorithmic Ethics: Formalization and Verification of Autonomous Vehicle Obligations. ACM Transactions on Cyber-Physical Systems (TCPS) 5, 4 (2021), 1--25.
[19]
Colin Shea-Blymyer and Houssam Abbas. 2021. Learning a Robot's Social Obligations from Comparisons of Observed Behavior. In 2021 IEEE International Conference on Advanced Robotics and Its Social Impacts (ARSO). IEEE, 15--20.
[20]
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. 2012. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems 25 (2012).
[21]
Milica Todorovi?, Michael U Gutmann, Jukka Corander, and Patrick Rinke. 2019. Bayesian inference of atomistic structure in functional materials. Npj computational materials 5, 1 (2019), 1--7.
[22]
Georg H. von Wright. 1951. Deontic Logic. Mind 60, 237 (January 1951).

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
AIES '22: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society
July 2022
939 pages
ISBN:9781450392471
DOI:10.1145/3514094
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 July 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. deontic logic
  2. explainability
  3. machine ethics
  4. model checking
  5. normative systems

Qualifiers

  • Research-article

Funding Sources

Conference

AIES '22
Sponsor:
AIES '22: AAAI/ACM Conference on AI, Ethics, and Society
May 19 - 21, 2021
Oxford, United Kingdom

Acceptance Rates

Overall Acceptance Rate 61 of 162 submissions, 38%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 260
    Total Downloads
  • Downloads (Last 12 months)134
  • Downloads (Last 6 weeks)20
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media