Abstract
Autonomous vehicles, and the larger socio-technical systems that they are a part of are likely to have a deep and lasting impact on our societies. Trust is a key value that will play a role in the development of autonomous driving systems. This paper suggests that trust of autonomous driving systems will impact the ways that these systems are taken up, the norms and laws that guide them and the design of the systems themselves. Further to this, in order to have autonomous driving systems that are worthy of our trust, we need a superstructure of oversight and a process that designs trust into these systems from the outset. Rather than banning or avoiding all autonomous vehicles should a tragedy occur, despite these systems having some level of risk, we want resilient systems that can survive tragedies, and indeed, improve from them. I will argue that trust plays a role in developing these resilient systems.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
By autonomous vehicle here, I mean a complex system that necessarily involves sensors, analytics, actuators and decision making elements, in which the default setting means humans are not directly involved in decision making—a vehicle that ‘drives itself’: “These refer primarily to future vehicles that may have the ability to operate without human intervention for extended periods of time and to perform a broad range of actions” (Lin 2016, p. 69).
There is an interesting discussion about social amplification of risk that certainly applies here, in which “risk events interact with psychological, social, and cultural processes in ways that can heighten or attenuate public perceptions of risk and related risk behavior. Behavioral patterns, in turn, generate secondary social or economic consequences but may act also to increase or decrease the physical risk itself” (Kasperson et al. 1988, pp. 178–179). Social amplification of risk likely underpins the general claims made in this paper. However, there is no space here to explore social amplification of risk further.
Autonomous driving systems (ADSs) refer not just to autonomous vehicles but the larger socio-technical system that autonomous vehicles operate in. ADSs will be explained more in Sect. 2.
Wagner and Koopman’s (2015) paper is one example that looks at trust and autonomous vehicles (Wagner and Koopman 2015). However, it is focussed on the reliance and predictability elements of trust—it does not cover the goodwill, affective or public elements, moreover, it is focused on autonomous vehicles and only implicitly concerns the autonomous driving systems that the vehicles will operate in.
It is unlikely that an autonomous vehicle meets a strict definition of ‘decide’ that we apply to conscious agents, where the agent has the “ability to understand criticism of one’s agency, along with the ability to defend or alter one’s actions based on one’s principles or principled criticism of one’s agency” (Nyholm 2018a, p. 1208). In line with the account of agent being used, decide is used more loosely here to refer to an agent that pursues “a goal on the basis of representations in a way that is regulated by certain rules or principles, while being supervised by some authority who can stop us or to whom control can be ceded, at least within certain limited domains” (Nyholm 2018a, p. 1208).
Certain approaches to autonomous vehicles and safety see a value in retaining a human on the loop, where control can be returned to the driver. “[I]f a driver is present in the vehicle, driver assistance systems can attain a safe state by handing control to the driver or by braking to a standstill” (Reschka 2016, p. 476). Moreover, as some argue, there are ethical issues about handing back control to the driver (Lin 2016, pp. 71–72).
For those who have driven in regions where there is limited law and order around driving, it is descriptively true that driving is a different experience than driving in regions with tight regulations and effective law and order. Though we do adapt. “Anyone who has visited another country as a tourist, whether as a driver or a pedestrian, has experienced for themselves how drastically informal rules, culturally specific modes of behavior, and communication between road users affect traffic. Human drivers who encounter such divergent “canons of rules” initially react with a phase of irritation. In the second phase, the adaptation phase, they unconsciously adapt over time to these new imprecise and unwritten rules. In the consolidation phase these rules seem to be obvious although they are not set in stone and in some cases cannot even be precisely described” (Färber 2016, p. 126).
For instance, Michael Wagner and Philip Koopman recognize that driving in “an “unstructured environment” such as a real-world road network includes plenty of unforeseen conditions. This lack of predictive capability demands new verification techniques to allow us to justify trusting self-driving cars in our everyday lives” (Wagner and Koopman 2015, p. 164).
Which is to recognize that some people do in fact suffer significant trauma from road accidents (Clapp et al. 2014), or have existing anxieties that prevent them from driving etc., in the first place. However, these people are in the minority.
This is taken from Francis Fukuyama (Fukuyama 1995).
Rather than seeking a definition trust, I will use a purposive approach which explains trust by describing the relation between the trustor and the trustee. Here, I am drawing from Mark Coeckelbergh’s relational account of responsibility. “responsibility is always relational, it is responsibility to someone… We are responsible for what we do to others who might be affected by what we do” (Coeckelbergh 2016, pp. 750, 751 Emphasis original). On this, trust is always relational; a trustor is always trusting someone to do something.
I credit Shannon Vallor for introducing me to this distinction between affective trust and public trust.
Another approach might include trust that the public has in the system. However, a more relevant form for this discussion is the trust that individuals have in the public system.
Computing disciplines, for instance, cover concepts of ‘trusted networks’ etc. But these, I have suggested elsewhere, run the risk of overlooking the human elements of trust (Henschke and Ford 2016).
In line with the description of autonomous given earlier, I want to avoid discussion about decisions and consciousness. That is, questions of the sort ‘can a non-conscious system make decision?’ are not of interest here. Particularly when looking at the different elements of trust, what’s more important is if and when drivers, other road users and other humans in the autonomous driving system act as if the autonomous vehicles decide. That is, whether a vehicle decides or not is not the point of enquiry here, instead, it is if and when we treat the vehicles like they make decisions.
I thank one of the anonymous reviewers for pressing this point.
References
Allhoff, F., & Henschke, A. (2018). The Internet of things: Foundational ethical issues. Internet of Things,1–2, 55–66. https://doi.org/10.1016/j.iot.2018.08.005.
Borenstein, J., Herkert, J. R., & Miller, K. W. (2017). Self-driving cars and engineering ethics: The need for a system level analysis. Science and Engineering Ethics. https://doi.org/10.1007/s11948-017-0006-0.
Clapp, J. D., Baker, A. S., Litwack, S. D., Sloan, D. M., & Beck, J. G. (2014). Properties of the driving behavior survey among individuals with motor vehicle accident-related posttraumatic stress disorder. Journal of Anxiety Disorders,28(1), 1–7.
Clark, D. D., & Blumenthal, M. S. (2011). The end-to-end argument and application design: The role of trust. Federal Communications Law Journal,63(2), 357.
Coeckelbergh, M. (2012). Can we trust robots? Ethics and Information Technology,14(1), 53–60. https://doi.org/10.1007/s10676-011-9279-1.
Coeckelbergh, M. (2016). Responsibility and the moral phenomenology of using self-driving cars. Applied Artificial Intelligence,30(8), 748–757. https://doi.org/10.1080/08839514.2016.1229759.
Cummings, D. M. (2016). Embedded software under the courtroom microscope: A case study of the Toyota unintended acceleration trial. IEEE Technology and Society Magazine,35(4), 76–84. https://doi.org/10.1109/MTS.2016.2618681.
Darling, K. (2017). Who’s Johnny? Anthropomorphic framing in human–robot interaction, integration, and policy. In P. Lin, K. Abney, & R. Jenkins (Eds.), Robot ethics 2.0: From autonomous cars to artificial intelligence. New York: Oxford University Press.
Darling, K., Nandy, P., & Breazeal, C. (2015). Empathic concern and the effect of stories in human–robot interaction. In Proceedings of the IEEE international workshop on robot and human communication (ROMAN).
Färber, B. (2016). Communication and communication problems between autonomous vehicles and human drivers. In M. Maurer, J. C. Gerdes, B. Lenz, & H. Winner (Eds.), Autonomous driving: Technical, legal and social aspects (pp. 125–147). Berlin: Springer.
Frewer, L. J., Miles, S., & Marsh, R. (2002). The media and genetically modified foods: Evidence in support of social amplification of risk. Risk Analysis,22(4), 701–711. https://doi.org/10.1111/0272-4332.00062.
Fukuyama, F. (1995). Trust: The social virtues and the creation of prosperity. Glencoe, IL: Free Press Paperbacks.
Henschke, A., & Ford, S. B. (2016). Cybersecurity, trustworthiness and resilient systems: Guiding values for policy. Journal of Cyber Policy. https://doi.org/10.1080/23738871.2016.1243721.
Hevelke, A., & Nida-rümelin, J. (2015). Responsibility for crashes of autonomous vehicles: An ethical analysis. Science and Engineering Ethics,21(3), 619–630. https://doi.org/10.1007/s11948-014-9565-5.
Himmelreich, J. (2018). Never mind the trolley: The ethics of autonomous vehicles in mundane situations. Ethical Theory and Moral Practice. https://doi.org/10.1007/s10677-018-9896-4.
Jones, K. (1996). Trust as an affective attitude. Ethics,107(1), 4–25.
Kasperson, R. E., Renn, O., Slovic, P., Brown, H. S., Emel, J., Goble, R., et al. (1988). The social amplification of risk: A conceptual framework. Risk Analysis,8(2), 177–187. https://doi.org/10.1111/j.1539-6924.1988.tb01168.x.
Lang, J. T., & Hallman, W. K. (2005). Who does the public trust? The case of genetically modified food in the United States. Risk Analysis,25(5), 1241–1252. https://doi.org/10.1111/j.1539-6924.2005.00668.x.
Levin, S. (2018). ‘Uber should be shut down’: Friends of self driving car crash victim seek justice. The Guardian, 20 March.
Lin, P. (2016). Why ethics matters for autonomous cars. In M. Maurer, J. C. Gerdes, B. Lenz, & H. Winner (Eds.), Autonomous driving: Technical, legal and social aspects (pp. 69–85). Berlin: Springer.
Maurer, M. (2016). Introduction. In M. Maurer, J. C. Gerdes, B. Lenz, & H. Winner (Eds.), Autonomous driving: Technical, legal and social aspects (pp. 1–8). Berlin: Springer.
McLeod, C. (2015). Trust. Retrieved April 15, 2016 from http://plato.stanford.edu/entries/trust/.
Nyholm, S. (2018a). Attributing agency to automated systems: Reflections on human–robot collaborations and responsibility-loci. Science and Engineering Ethics,24(4), 1201–1219. https://doi.org/10.1007/s11948-017-9943-x.
Nyholm, S. (2018b). The ethics of crashes with self-driving cars: A roadmap, I. Philosophy Compass,13(7), e12507. https://doi.org/10.1111/phc3.12507.
Nyholm, S. (2018c). The ethics of crashes with self-driving cars: A roadmap, II. Philosophy Compass,13(7), e12506. https://doi.org/10.1111/phc3.12506.
Nyholm, S. (2018d). Teaching & learning guide for: The ethics of crashes with self-driving cars: A roadmap, I–II. Philosophy Compass,13(7), e12508. https://doi.org/10.1111/phc3.12508.
Nyholm, S., & Smids, J. (2018). Automated cars meet human drivers: Responsible human-robot coordination and the ethics of mixed traffic. Ethics and Information Technology. https://doi.org/10.1007/s10676-018-9445-9.
Poppo, L., & Schepker, D. J. (2010). Repairing public trust in organizations. Corporate Reputation Review,13(2), 124–141. https://doi.org/10.1057/crr.2010.12.
Poppo, L., & Schepker, D. J. (2014). The repair of public trust following controllable or uncontrollable organizational failures: A conceptual framework. In J. D. Harris, B. Moriarty, & A. C. Wicks (Eds.), Public trust in business (pp. 326–360). Cambridge: Cambridge University Press.
Purves, D., Jenkins, R., & Strawser, B. J. (2015). Autonomous machines, moral judgment, and acting for the right reasons. Ethical Theory and Moral Practice,18(4), 851–872. https://doi.org/10.1007/s10677-015-9563-y.
Reschka, A. (2016). Safety concept for autonomous vehicles. In M. Maurer, J. C. Gerdes, B. Lenz, & H. Winner (Eds.), Autonomous driving: Technical, legal and social aspects (pp. 473–496). Berlin: Springer.
Reuters. (2018). Uber settles with family of woman killed by self-driving car. The Guardian, 29 March. https://www.theguardian.com/technology/2018/mar/29/uber-settles-with-family-of-woman-killed-by-self-driving-car.
Rhodes, C. (2016). Democratic business ethics: Volkswagen’s emissions scandal and the disruption of corporate sovereignty. Organization Studies,37(10), 1501–1518. https://doi.org/10.1177/0170840616641984.
Roff, H. M. (2013). Killing in war: Responsibility, liability, and lethal autonomous robots. In F. Allhoff, N. G. Evans, & A. Henschke (Eds.), Routledge handbook of ethics and war: Just war in the 21st century. New York: Routledge.
Said, C. (2018). Uber puts the brakes on testing robot cars in california after arizona fatality. San Francisco Chronicle, 27 March. https://www.sfchronicle.com/business/article/Uber-pulls-out-of-all-self-driving-car-testing-in-12785490.php.
Simpson, T. W. (2012). What is trust? Pacific Philosophical Quarterly,93(4), 550–569. https://doi.org/10.1111/j.1468-0114.2012.01438.x.
T.S. (2018). Why uber’s self-driving car killed a pedestrian. The Economist, 29 March. https://www.economist.com/the-economist-explains/2018/05/29/why-ubers-self-driving-car-killed-a-pedestrian.
UNIDR. (2014). Framing discussions on the weaponization of increasingly autonomous technologies. Geneva: United Nations Institute For Disarmament Research.
Uslaner, P. (2002). The moral foundations of trust. Cambridge: Cambridge University Press.
Vallor, S. (2017). Artificial intelligence and public trust. Ethics of technology: The future agenda (4TU Ethics), University of Twente, Enschede, 12 June.
Verbeek, P.-P. (2011). Moralizing technology: Understanding and designing the morality of things. Chicago: University of Chicago Press.
Vlasic, B., & Apuzzo, M. (2014). Toyota is fined $1.2 billion for concealing safety defects. The New York Times, 19 March. https://www.nytimes.com/2014/03/20/business/toyota-reaches-1-2-billion-settlement-in-criminal-inquiry.html?_r=0.
Wagner, M., & Koopman, P. (2015). A philosophy for developing trust in self-driving cars. In G. Meyer & S. Beiker (Eds.), Road vehicle automation (Vol. 2, pp. 163–171). Cham: Springer.
Whitfield, S. C., Rosa, E. A., Dan, A., & Dietz, T. (2009). The future of nuclear power: Value orientations and risk perception. Risk Analysis,29(3), 425–437. https://doi.org/10.1111/j.1539-6924.2008.01155.x.
Winkle, T. (2016). Safety benefits of automated vehicles: Extended findings from accident research for development, validation and testing. In M. Maurer, J. C. Gerdes, B. Lenz, & H. Winner (Eds.), Autonomous driving: Technical, legal and social aspects (pp. 335–364). Berlin: Springer.
Wolf, I. (2016). The interaction between humans and autonomous agents. In M. Maurer, J. C. Gerdes, B. Lenz, & H. Winner (Eds.), Autonomous driving: Technical, legal and social aspects (pp. 103–124). Berlin: Springer.
World Health Organization. (2015). Global status report on road safety 2015. Geneva: World Health Organization.
Yadron, D., & Tynan, D. (2016). Tesla driver dies in first fatal crash while using autopilot mode. The Guardian, 1 July. https://www.theguardian.com/technology/2016/jun/30/tesla-autopilot-death-self-driving-car-elon-musk.
Zhang, X., Zhao, X., Du, H., & Rong, J. (2014). A study on the effects of fatigue driving and drunk driving on drivers’ physical characteristics. Traffic Injury Prevention,15(8), 801–808. https://doi.org/10.1080/15389588.2014.881996.
Acknowledgements
This material is based upon work supported by the National Science Foundation under Grant No. 1522240. Preliminary research on this topic was conducted as a Visiting Researcher at the Brocher Foundation (Switzerland) He thanks the European Research Council and the Australian National University ‘Grand Challenge’, which have provided generous support under awards Advanced Grant project on Collective Responsibility and Counterterrorism and the ‘Our Health In Our Hands’ grant, respectively. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation, the Brocher Foundation, the European Research Council or the Australian National University.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Henschke, A. Trust and resilient autonomous driving systems. Ethics Inf Technol 22, 81–92 (2020). https://doi.org/10.1007/s10676-019-09517-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10676-019-09517-y