Abstract
This paper addresses how to act towards digital agents while uncertain about their moral status. It focuses specifically on the problem of how to act towards simulated minds operated by an artificial superintelligence (ASI). This problem can be treated as a sub-set of the larger problems of AI-safety (how to ensure a desirable outcome after the emergence of ASI) and also invokes debates about the grounds of moral status. The paper presents a formal structure for solving the problem by first constraining it as a sub-problem to the AI-safety problem, and then suggesting a decision-theoretic approach to how this problem can be solved under uncertainty about what the true grounds of moral status are, and whether such simulations do possess these relevant grounds. The paper ends by briefly suggesting a way to generalize the approach.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
There has also been discussion on how we should handle artificial suffering more generally, even in non-ASI circumstances (See for instance Metzinger (2013)). However, for the purposes of this paper, I want to focus on ASI-run simulations as (1) they might be significantly more extensively deployed given that the cost to do so would be lower for a superintelligent agent, and (2) they are relevant for the larger ASI problem (as will be laid out in the Sect. 2.1), which is my larger focus here.
- 3.
While I’m using the phrase ‘AI-safety’ here as specific to ASIs, the term has been used more broadly in the literature. See for instance Amodei et al. (2016).
- 4.
One point often raised by individuals is whether such an intelligence will start to develop its own goals. While I will not go into this issue here, there is an extensive discussion of this topic in Chap. 7 of Bostrom (2014), where he argues that there need be no necessary relationship between intelligence and fundamental goals.
- 5.
To align the terminology used with the existing literature (Bostrom 2014), the AI-safety problem is the problem of articulating the correct motivation for the ASI, which is one part of what is known as the motivational selection problem.
- 6.
Here, the ATC constraint seems to be too vaguely formulated – for instance, we need to be able to say not just that solutions run afoul of it, but also assess the extent to which they do so. The ATC constraint thus needs to be further developed to provide a metric that allows us to make this assessment. Such a metric would need to track, for any given solution, the probability of incurring other costs (apart from morally impermissible treatment of simulations) along with the magnitude of the cost incurred. We shall look at one way of defining such a metric in Sect. 4.
- 7.
Jaworska and Tannenbaum (2013). An understanding of moral status as being concerned with moral patienthood is not universally accepted in the literature – sometimes, moral status is associated both with moral agency and moral patienthood. For the purposes of this paper, however, we should understand it as just referring to moral patienthood.
- 8.
There is a further distinction that can be drawn here, between distinct types of non-instrumental moral status. Kamm notes that we can speak of the status an entity has “in its own right” compared to the status that it has “for its own sake”, where violations against the latter allow us to speak of the entity being morally wronged whereas violations against the former do not permit it. Here, I will limit my scope to only considering an account of moral status in the traditional sense, where an entity counts “for its own sake”. Kamm (2007).
- 9.
Coherent Extrapolated Volition as a solution to the AI-safety problem recommends that we ask the ASI to implement the coherent extrapolated volition of humankind, where this CEV is an agent’s wish for himself if he were idealized in certain ways – if he were smarter, more the person he wished he were, etc. This proposal looks for coherence among such idealized wishes of humans and recommends that the ASI implements the wishes where there is such a strong coherence. Yudkowsky (2004).
- 10.
This is a decision problem that is relevant to more than one type of ASI. Among other types, ASIs are classified as oracles (question-answering ASIs), genies (intermediate-level task-oriented autonomously acting ASIs) and sovereigns (autonomously acting ASIs with a single open goal). However, all three types may run simulations to understand the psychological profiles of individuals. See Bostrom (2014) for further details on the classification of the various types of ASIs.
- 11.
More generally, such a possibility could obtain if the ASI’s architecture involves ‘black-box’ methods. See Armstrong et al. (2012), pp. 15–19.
- 12.
I follow, with some modifications, the categorization put forward by Jaworska and Tannenbaum (2013).
- 13.
Bostrom and Yudkowsky introduce this notion of sapience as “a set of capacities associated with higher intelligence, such as self-awareness and being a reason-responsive agent”. Bostrom and Yudkowsky (2011).
- 14.
Two important accounts, while not being potentiality or membership accounts, still resemble them. The first account is Shelly Kagan’s ‘modal personhood’ account – to have moral status, an entity must either have the capacity of personhood (“a being that is rational and self-conscious, aware of itself as one being among others, extended through time”), or have the property of being a ‘modal person’, such that even if it is not currently a person, it could have been a person. The second account is S. Mathew Liao’s genetic basis for moral agency account, where for an entity to have moral status it is sufficient that it possesses the genetic basis for moral agency, as it occurs in those human beings that we normally take to exercise moral agency. See Kagan (2016) and Liao (2010) respectively.
- 15.
Apart from these four major categories of accounts, there are a few other possibilities that we need to cover for the sake of comprehensively mapping out the logical space. Firstly, there is a possibility that the correct account of the grounds of moral status invokes a property that is not covered under any of the above four categories. Secondly, there is also the possibility that the correct account of the grounds of moral status could be a combination of two or more of the above four accounts – the correct account could thus feature a conjunction of two or more accounts (for instance, sapience as well as sentience) or a disjunction of two or more accounts (for instance, with both sapience and sentience being sufficient conditions for moral status without either being a necessary condition).
- 16.
Once we have confirmed that the relevant property is computationally realizable, there is still further work to be done depending on which account is the correct account of moral status. If, for instance, the rationality account is the right one, then all we need to check is if simulations are rational. However, if the potential for rationality account is the right one, then we need to ask whether simulations are rational, and if not, whether they have the potential to be rational. This would require the further step of figuring out what would it take for something to have the potential to be rational. However, in either case, the computational realizability of rationality still needs to be checked.
- 17.
Here, we shall simplify the scenario such that it is not the case that multiple properties can individually be sufficient for grounding moral status – thus one and only one property grounds moral status.
- 18.
An alternative scenario could be if we’re not certain about which properties are computationally realizable, but hold credences about the computational realizability of the various properties. In this case, DM2 would look more or less like DM1, except with credences associated with each column (each situation).
- 19.
I have assumed in Sect. 3.2 that we won’t be able to solve the moral question (or even MQ(a)) in time. However, if we can solve MQ(a) in time and thus have full information about which property it is that grounds moral status, this would just be reflected with a credence of 1 associated with the scenario that that property grounds moral status, and a credence of 0 for all other scenarios.
- 20.
We can use our credences for these states in a way that is consistent with recent work on normative uncertainty. See MacAskill (2014). The only difference here is the further problem of deciding whose credences should be taken into account.
- 21.
- 22.
MacAskill puts forward a solution to the problem of intertheoretic comparability by drawing parallels with social choice theory, specifically with how social welfare functionals have been axiomatized under alternative assumptions about informational comparability. See MacAskill (2014).
- 23.
See Peterson (2009) Chap. 4 for some paradoxes raised that undermine the utilization of expected utility maximisation as a decision rule for decisions under risk.
References
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., Mane, D.: Concrete problems in AI safety. arXiv:1606.06565v2 [cs.AI] (2016)
Armstrong, S., Sandberg, A., Bostrom, N.: Thinking inside the box: controlling and using an oracle AI. Mind. Mach. 22, 299–324 (2012)
Bernstein, M.H.: On Moral Considerability. An Essay on Who Morally Matters. Oxford University Press, Oxford (1998)
Boonin, D.: In Defense of Abortion. Cambridge University Press, New York (2003)
Bostrom, N.: Pascal’s mugging. Analysis 69, 443–445 (2009)
Bostrom, N.: Superintelligence: Paths, Dangers, Strategies. Oxford University Press, Oxford (2014)
Bostrom, N., Cirkovic, M.: Introduction. In: Bostrom, N., Cirkovic, M. (eds.) Global Catastrophic Risks. Oxford University Press, Oxford (2008)
Bostrom, N., Yuskowsky, E.: The ethics of artificial intelligence. In: Frankish, K., Ramsey, W.M. (eds.) The Cambridge Handbook of Artificial Intelligence. Cambridge University Press, Cambridge (2011)
Chalmers, D.: The singularity: a philosophical analysis. J. Conscious. Stud. 17, 7–65 (2010)
Cohen, C.: The case for the use of animals in biomedical research. N. Engl. J. Med. 315, 865–870 (1986)
Dainton, B.: On singularities and simulations. J. Conscious. Stud. 19, 42–85 (2012)
Dworkin, R.: Life’s Dominion: An Argument about Abortion, Euthanasia, and Individual Freedom. Vintage Books, New York (1993)
Gruen, L.: The moral status of animals. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy (Fall 2014 edition) (2014). https://plato.stanford.edu/archives/fall2014/entries/moral-animal/
Feinberg, J.: Abortion. In: Reagan, T. (ed.) Matters of Life and Death, pp. 183–217. Temple University Press, Philadelphia (1980)
Finnis, J.: The fragile case for euthanasia: a reply to John Harris. In: Keown, J. (ed.) Euthanasia Examined. Cambridge University Press, Cambridge (1995)
Jaworska, A., Tannenbaum, J.: The grounds of moral status. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy (Summer 2013 Edition) (2013). https://plato.stanford.edu/archives/sum2013/entries/grounds-moral-status/
Kagan, S.: What’s wrong with speciesism? J. Appl. Philos. 33, 1–21 (2016)
Kamm, F.M.: Intricate Ethics. Rights, Responsibilities and Permissible Harm. Oxford University Press, Oxford (2007)
Kant, I.: Groundwork of the Metaphysics of Morals (1785). Gregor, M. (trans. and ed.) Cambridge University Press (1988)
Liao, M.S.: The basis of human moral status. J. Moral Philos. 7(2), 159–179 (2010)
List, C.: Social choice theory. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy (Winter 2013 Edition) (2013). https://plato.stanford.edu/archives/win2013/entries/social-choice/
MacAskill, W.: Normative uncertainty. DPhil Thesis (2014)
Marquis, D.: Why abortion is immoral. J. Philos. 86, 183–202 (1989)
McMahan, J.: The Ethics of Killing: Problems at the Margins of Life. Oxford University Press, Oxford (2002)
Metzinger, T.: Two principles for robot ethics. In: Hilgendorf, E., Gunther, J.-P. (eds.) Robotik und Gesetzgebung, pp. 263–302. Nomos, Baden-Baden (2013)
Norcross, A.: Puppies, pigs and people: eating meat and marginal cases. Philos. Perspect. 18, Ethics, 229–245 (2004)
Nozick, R.: Do animals have rights? In: Socratic Puzzles, pp. 303–310. Harvard University Press (1997)
Peterson, M.: An Introduction to Decision Theory. Cambridge University Press, Cambridge (2009)
Sepielli, A.: Moral uncertainty. In: Routledge Encyclopaedia of Philosophy Online (Forthcoming)
Shulman, C., Bostrom, N.: How hard is artificial intelligence? – Evolutionary arguments and selection effects. J. Conscious. Stud. 19, 103–130 (2012)
Singer, P.: Practical Ethics, 2nd edn. Cambridge University Press, Cambridge (1993)
Steele, K., Stefánsson, O.H.: Decision theory. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy (Winter 2016 Edition) (2016). https://plato.stanford.edu/archives/win2016/entries/decision-theory/
Steinbock, B.: Life Before Birth: The Moral and Legal Status of Embryos and Fetuses. Oxford University Press, Oxford (1992)
Steinhart, E.: The singularity: beyond philosophy of mind. J. Conscious. Stud. 19, 131–137 (2012)
Stone, J.: Why potentiality matters. Can. J. Philos. 17, 815–829 (1987)
Yudkowsky, E.: Creating friendly AI 1.0: the analysis and design of benevolent goal architectures. The Singularity Institute, San Francisco, CA (2001)
Yudkowsky, E.: Coherent extrapolated volition. The Singularity Institute, San Francisco, CA (2004)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Mishra, A. (2018). Moral Status of Digital Agents: Acting Under Uncertainty. In: Müller, V. (eds) Philosophy and Theory of Artificial Intelligence 2017. PT-AI 2017. Studies in Applied Philosophy, Epistemology and Rational Ethics, vol 44. Springer, Cham. https://doi.org/10.1007/978-3-319-96448-5_30
Download citation
DOI: https://doi.org/10.1007/978-3-319-96448-5_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-96447-8
Online ISBN: 978-3-319-96448-5
eBook Packages: Computer ScienceComputer Science (R0)