Abstract
There is already evidence of several mishaps, accidents, and even major catastrophes in which AI technologies have played a part. Among them are traditional types of organizational accidents but also complex misadventures where the effects of the use of AI technology intertwine with human action in the context of intricate socio-technical settings. The origins of problems range from making errors during a planned course of action to accidents due to unintended and unpredictable outcomes of the use of AI. A variety of ethical principles for the use and design of AI have been formulated to remedy the ills. The article sums up the main concepts of the AI ethical codes and discusses their role in attempts to prevent AI-induced problems. According to recent empirical studies, the impact of ethics principles is weak in terms of real-life AI design and use. As an alternative, the article suggests the findings of safety research as a source for a practice-oriented approach to tackle the problems. A century-long study of safety provides both conceptual models and practical tools for the prevention of accidents and creating the means to improve safety. The article introduces the main findings of this line of research and suggests how the Normal Accident Theory, High Reliability Organizations studies, and Just Culture approach could contribute to a safer AI design.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Inkster [43], 1–17.
- 2.
Coeckelbergh [15], 75–76.
- 3.
Hagendorff [30], 101.
- 4.
Hagendorff [30], 102.
- 5.
Hagendorff [30], 102–103.
- 6.
Hagendorff [30], 102–103; The European Commission’s High Level Expert Group.
- 7.
Hagendorff [30], 103–104.
- 8.
Hagendorff [30], 105.
- 9.
Hagendorff [30], 105.
- 10.
Hagendorff [30], 106.
- 11.
Hagendorff [30], 107.
- 12.
Hagendorff [30], 108.
- 13.
Hagendorff [30], 108–109.
- 14.
Boddington 2017, 56.
- 15.
Hagendorff [30], 108–109.
- 16.
Coeckelbergh [15], 157.
- 17.
Trustworthy AI [22], 20.
- 18.
Trustworthy AI [22], 25–31.
- 19.
Trustworthy AI [22], 31.
- 20.
Hagendorff [30], 112–113.
- 21.
Coeckelbergh [15], 145–165.
- 22.
Coeckelbergh [15], 162.
- 23.
Clarke [14], 7.
- 24.
Reiman & Oedewald [66], 39.
- 25.
Reiman & Oedewald [66], 39–48.
- 26.
See also Reiman & Oedewald [66], 39–47.
- 27.
Reiman & Oedewald [66], 40.
- 28.
Manuele [55], 236.
- 29.
Manuele [55], 234–256.
- 30.
- 31.
Reiman & Oedewald [66], 40.
- 32.
Reason [61], 9.
- 33.
Reason [61], 53.
- 34.
- 35.
Reason [61], 197.
- 36.
Perrow 1999 [58], 11.
- 37.
Maidment 1997 [53], 71.
- 38.
- 39.
Reason [63], 49–50.
- 40.
Reiman & Oedewald [66], 44.
- 41.
Perrow [58], 5.
- 42.
Perrow [58], 321–350.
- 43.
Hollnagel [40], 25, 28.
- 44.
Reiman & Oedewald [66], 241–242.
- 45.
Reiman & Oedewald [66], 246.
- 46.
Dekker [18], 45.
- 47.
Reason [62], 195–196.
References
A walk on the dark side. The Economist (2007)
Abusus non tollit usum (s.a.). https://www.merriam-webster.com/dictionary/abusus%20non%20tollit%20usum
AI for Good Global Summit 2017. ITU News (2017). https://www.itu.int/en/ITU-T/AI/Pages/201706-default.aspx
AI Now: 2019 Report. AI Now Institute, New York University (2019). https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwiix8_MmcvuAhVFl4sKHVuLDWsQFjAAegQIAxAC&url=https%3A%2F%2Fainowinstitute.org%2FAI_Now_2019_Report.pdf&usg=AOvVaw2hwdlsKFcce1B1wW0ucWRL
Alimahomed-Wilson, J., Reese, E.: The Cost of Free Shipping: Amazon in the Global Economy. Pluto Press, London (2020)
Auernhammer, J.: Human-centered AI: the role of human-centered design research in the development of AI. In: Boess, S., Cheung, M., Cain, R. (eds.) Synergy - DRS International Conference 2020, 11–14 August (2020). https://doi.org/10.21606/drs.2020.282
Bayern, M.: How AI is spreading everywhere with the rise of smart machines. TechRepublic, Innovation (2018). https://www.techrepublic.com/article/how-ai-is-spreading-everywhere-with-the-rise-of-smart-machines/
Bernstein, P.: Against the Gods: The Remarkable Story of Risk. Wiley, Hoboken (1998)
Blumberg, D.: What AI startups need to achieve before VCs will invest. Extra Crunch (2019). https://techcrunch.com/2019/11/13/what-ai-startups-need-to-achieve-before-vcs-will-invest/
Bradshaw, S., Howard, P.N.: Challenging truth and trust. A global inventory of organized social media manipulation. Computational Propaganda Research Project. Oxford Internet Institute. University of Oxford, Oxford (2018). http://comprop.oii.ox.ac.uk/wp-content/uploads/sites/93/2018/07/ct2018.pdf
Carback, J.T.: Cybersex trafficking: toward a more effective prosecutorial response. Crim. Law Bull. 54(1), 64–183 (2018)
Chaffey, D.: Global social media research summary August 2020. Smart Insights (2020). https://www.smartinsights.com/social-media-marketing/social-media-strategy/new-global-social-media-research/
Chui, M., Harrysson, M., Manyika, J., Roberts, R., Chung, R., Nel, P., et al.: Applying artificial intelligence for social good. Discussion Paper. McKinsey Global Institute (2018). https://www.mckinsey.com/featured-insights/artificial-intelligence/applying-artificial-intelligence-for-social-good
Clarke, M.: Insurance fraud. Br. J. Criminol. 29 (1), 1–20 (1989). https://www.jstor.org/stable/23638129?seq=8#metadata_info_tab_contents
Coeckelbergh, M.: AI Ethics. The MIT Press Essential Knowledge Series. The MIT Press, Cambridge (2020)
Combating cyber crime: Department of Homeland Security (2012)
Cyber crime: Federal Bureau of Investigation (2017)
Dekker, S.: The Field Guide to Understanding ‘Human Error’, 3rd edn. CRC Press, Boca Raton (2014)
Dekker, S.W.A., Breakey, H.: ‘Just culture:’ improving safety by achieving substantive, procedural and restorative justice. Saf. Sci. 85, 187–193 (2016)
Doris, J.: Lack of Character. Personality and Moral Behavior. Cambridge University Press, Cambridge (2002)
Enya, A., Dempsey, S., Pillay, M.: High reliability organisation (HRO) principles of collective mindfulness: an opportunity to improve construction safety management. In: Arezes, P.M.F.M. (ed.) AHFE 2018. AISC, vol. 791, pp. 3–13. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-94589-7_1
Ethics guidelines for trustworthy AI. High-Level Expert Group on Artificial Intelligence. European Commission, Brussels (2019). https://ec.europa.eu/digital-single-market/en/news/ethics-guidelines-trustworthy-ai
Feinberg, T.: Whether it happens at school or off-campus, cyberbullying disrupts and affects all aspects of students’ lives (2008). http://docplayer.net/17558112-Whether-it-happens-at-school-or-off-campus-cyberbullying-disrupts-and-affects-all-aspects-of-students-lives.html
Floridi, L.: Faultless responsibility: on the nature and allocation of moral responsibility for distributed moral actions. Philos. Trans. Ser. Math. Phys. Eng. Sci. 374(2083), 1–13 (2016)
Floridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., Dignum, V., et al.: AI4People: an ethical framework for a good AI society: opportunities, risks, principles, and recommendations. Minds Mach. 28(4), 689–707 (2018)
Fossheim, H.: Virtue ethics and everyday strategies. Revue Internationale de Philosophie 267(1), 65–82 (2014). https://doi.org/10.3917/rip.267.0065
Fresneau, V.: 10 promising European deeptech startups leading us into the future. EU-Startups (2020). https://www.eu-startups.com/2020/05/10-promising-deeptech-startups/
Gawande, A.: The Checklist Manifesto: How to Get Things Right. Profile Books, London (2011)
Grace, K., Salvatier, J., Dafoe, A., Zhang, B., Evans, O.: Viewpoint: when will AI exceed human performance? Evidence from AI experts. J. Artif. Intell. Res. 62, 729–754 (2018)
Hagendorff, T.: The ethics of AI ethics: an evaluation of guidelines. Minds Mach. 30, 99–120 (2020). https://doi.org/10.1007/s11023-020-09517-8
Halder, D., Jaishankar, K.: Cyber Crime and the Victimization of Women: Laws, Rights, and Regulations. IGI Global, Hershey (2011)
Hanington, B.: Methods in making: a perspective on the state of human research in design. Des. Issues 19(4) 9–18 (2003). Massachusetts Institute of Technology. http://scholar.google.fi/scholar_url?url=https://www.ida.liu.se/divisions/hcs/ixs/material/servicedesignGbg10/2%2520Additional%2520readings/humancentereddesign.pdf&hl=fi&sa=X&ei=iXYaYPTGCY-Ny9YP3o6D2AY&scisig=AAGBfm0uqeu6IdfPOI3kEXfH1CUmKFZKEA&nossl=1&oi=scholarr
Hao, K.: In 2020, let’s stop AI ethics-washing and actually do something. MIT Technol. Rev. (2019). https://www.technologyreview.com/2019/12/27/57/ai-ethics-washing-time-to-act/
Harman, G.: Moral philosophy meets social psychology: virtue ethics and the fundamental attribution error. In: Proceedings of the Aristotelian Society, vol. 99, pp. 315–356 (1998–1999)
Harper, J.: Tesla: Soaring share price creates army of ‘Teslanaires’. BBC News (2020). https://www.bbc.com/news/business-55391571
Heinrich, H.W.: Industrial Accident Prevention. McGraw-Hill, New York (1931)
Hibbard, B.: Ethical Artificial Intelligence (2015). https://arxiv.org/abs/1411.1373
Hogg, M.A., Vaughan, G.M.: Social Psychology, 5th edn. Pearson Education Limited, Harlow (2008)
Holland, J. H.: Complex adaptive systems. Daedalus 121(1), 17–30 (1992). A New Era in Computation. https://www.jstor.org.stable/20025416
Hollnagel, E.: The ETTO Principle: Efficiency-Thoroughness Trade-off: Why Things That Go Right Sometimes Go Wrong. Ashgate, Farnham (2009)
Hollnagel, E.: Safety-I and Safety-II: The Past and Future of Safety Management. CRC Press, Boca Raton (2014)
HRO Safety Culture Definition: United States Nuclear Regulatory Commission and United States Government Accountability Office (2010)
Inkster, I.: Science and Technology in History: An Approach to Industrial Development. Macmillan Educational, New Brunswick (1991)
Johnson, A.: Examining the foundation. Saf. Health Mag. (2011). https://www.safetyandhealthmagazine.com/articles/print/6368-examining-the-foundation
Just Culture. Skybrary. https://www.skybrary.aero/index.php/Just_Culture
Kibby, M.: The impact of AI on human intelligence: can we live without it? AmeInfo (2020). https://www.ameinfo.com/industry/tech-and-mobility/the-impact-of-ai-on-human-intelligence
Knapp, A.: Six billionaires who made their fortunes from videogames. Forbes (2017). https://www.forbes.com/sites/alexknapp/2017/03/21/six-billionaires-who-made-fortunes-from-videogames/
Kranzberg, M.: Technology and history: “Kranzberg’s laws”. Technol. Cult. 27(3), 544–560 (1986). https://doi.org/10.2307/3105385
Krupiv, T.: A vulnerability analysis: theorising the impact of artificial intelligence decision-making processes on individuals, society and human diversity from a social justice perspective. Comput. Law Secur. Rev. 38 (2020). https://doi.org/10.1016/j.clsr.2020.105429
Lin, P., Abney, K., Jenkins, R. (eds.): Robot ethics 2.0: From autonomous cars to artificial intelligence. Oxford Scholarship Online (2017)
MacIntyre, A.: After Virtue: A Study in Moral Theory. University of Notre Dame Press, Notre Dame (1981)
McNamara, A., Smith, J., Murphy-Hill, E.: Does ACM’s code of ethics change ethical decision making in software development? In: Leavens, G.T., Garcia, A., Păsăreanu C.S. (eds.) Proceedings of the 2018 26th ACM joint meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering—ESEC/FSE 2018, pp. 1–7. ACM Press, New York (2018)
Maidment, D.: The effect of the accident inquiry process within the railway industry. In: Hale, A., Wilpert, B., Freitag, M. (eds.) After the Event. From Accident to Organisational Learning, Pergamon, Oxford, pp. 61–76 (1997)
Mann, D., Sutton, M.: Netcrime. Br. J. Criminol. 38(2) 201–229 (2011). CiteSeerX 10.1.1.133.3861
Manuele, P.A.: On the Practice of Safety, 4th edn. Wiley, Hoboken (2013)
Marr, B.: Artificial Intelligence in Practice: How 50 Successful Companies used AI and Machine Learning to Solve Problems. Wiley, Chichester (2019)
Osoba, O.A., Welser IV, W.: The risks of artificial intelligence to security and the future of work. Perspective. Expert insights Timely Policy Issue. https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwigjOPD68ruAhXsiIsKHQ98CIUQFjALegQIJRAC&url=https%3A%2F%2Fwww.rand.org%2Fcontent%2Fdam%2Frand%2Fpubs%2Fperspectives%2FPE200%2FPE237%2FRAND_PE237.pdf&usg=AOvVaw2RDrXdeOZjnxJrVoXqV0pk
Perrow, C.: Normal Accidents: Living with High-Risk Technologies. Princeton University Press, Princeton (1999)
Perrow, C.: The Next Catastrophe: Reducing Our Vulnerabilities to Natural, Industrial, and Terrorist Disasters. Princeton University Press, Princeton (2007)
Railway Passengers Assurance Company Ltd. http://www.aviva.com/about-us/heritage/companies/railway-passengers-assurance-company/
Reason, J.: Human Error. Cambridge University Press, Cambridge (1990)
Reason, J.: Managing the Risks of Organizational Accidents. Routledge, London and New York (1997)
Reason, J.: The Human Contribution. Unsafe Acts, Accidents and Heroic Recoveries. Routledge, New York (2008)
Reason, J.: Organizational Accidents Revisited. Ashgate, Farnham (2016)
Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (2016). http://data.europa.eu/eli/reg/2016/679/oj
Reiman, T., Oedewald, P.: Turvallisuuskriittiset organisaatiot: Onnettomuudet, kulttuuri ja johtaminen. (Safety-Critical Organizations: Accidents, Culture, and Leadership) Edita, Helsinki (2008)
Resilience Engineering (2019). https://www.resilience-engineering-association.org/blog/2019/11/09/what-is-resilience-engineering/
Roberts, K.H.: New challenges in organizational research: high reliability organizations. Organ. Environ. 3(2), 111–125 (1989). https://doi.org/10.1177/108602668900300202
Robertson, K., Black, K., Grand-Clement, S., Hall, A.: Human and Organisational Factors in Major Accident Prevention: A Snapshot of the Academic Landscape. RAND Corporation, Santa Monica (2016). https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwiipdWKqMvuAhVwAxAIHaWZD00QFjATegQIChAC&url=https%3A%2F%2Fwww.rand.org%2Fcontent%2Fdam%2Frand%2Fpubs%2Fresearch_reports%2FRR1500%2FRR1512%2FRAND_RR1512.pdf&usg=AOvVaw27bcYG_JDYDOKlcAn5bSe1
Rosenberg, S.: Why AI is still waiting for its ethics transplant. Wired (2017). https://www.wired.com/story/why-ai-is-still-waiting-for-its-ethics-transplant/
Rossi, B.: Has AI become something we cannot live without? Information Age (2016). https://www.information-age.com/has-ai-become-something-we-cant-live-without-123461456/
Russell, C.: A case for not regulating the development of artificial intelligence. Towards Data Science (2019). https://towardsdatascience.com/a-case-for-not-regulating-the-development-of-artificial-intelligence-f3d23db2e8cd
Sagan, S.D.: The Limits of Safety. Organizations, Accidents, and Nuclear Weapons. Princeton University Press, Princeton (1993)
Sammarco, J.J.: Operationalizing normal accident theory for safety-related computer systems. Saf. Sci. 43(9), 697–714 (2005). https://doi.org/10.1016/j.ssci.2005.03.001
Saran, S., Natrajan, N., Srikumar, M.: In Pursuit of Autonomy: AI and National Strategies. Observer Research Foundation, New Delhi (2018)
Scherer, M.U.: Regulating artificial intelligence systems: risks, challenges, competencies, and strategies. Harv. J. Law Technol. 29(2) (2016). https://doi.org/10.2139/ssrn.2609777
Schmeltzer, R.: The complex nature of regulating AI. SearchEnterpriseAI. TechTarget (2019). https://searchenterpriseai.techtarget.com/feature/The-complex-nature-of-regulating-AI
Shrivastava, S., Sonpar, K., Pazzaglia, F.: Normal accident theory versus high reliability theory: a resolution and call for an open systems view of accidents. Hum. Relat. 62(9), 1357–1390 (2009). https://doi.org/10.1177/0018726709339117
Stametalatos, M.: Probabilistic Risk Assessment: What is It and Why Is It Worth Performing It? (2000). http://www.hq.nasa.gov/office/codecq/qnews/prs.pdf
Stultz, J.: High Reliability Organization Toolkit. Manage Operations (2018). https://web.mhanet.com/media-library/high-reliability-organization-toolkit/
The Montréal Declaration for a Responsible Development of Artificial Intelligence. Inven_T, University of Montreal’s Technosocial Innovation Centre (2018). https://www.montrealdeclaration-responsibleai.com/
Vanderhaegen, F., Hollnagel, E.: Safety-I and Safety-II, the past and future of safety management. Cognit. Technol. Work 17(3), 461–464 (2015). https://doi.org/10.1007/s10111-015-0345-z
Wagner, B.: Ethics as an escape from regulation. From “ethicswashing” to ethics-shopping? Repositorium für die Medienwissenschaft (2018). https://doi.org/10.25969/mediarep/13281
Why Air Travel is The Safest Mode of Transportation? Sheffield School of Aeronautics. https://www.sheffield.com/air-travel-safest-mode-transportation
Why is regulating artificial intelligence so tricky? The Startup (2020). https://medium.com/swlh/why-is-regulating-artificial-intelligence-so-tricky-f202c967c2b4
Weick, K.E., Sutcliffe, K.M.: Managing the Unexpected: Assuring High Performance in an Age of Complexity. Jossey Bass, San Francisco (2001)
Weick, K.E., Sutcliffe, K.M.: Managing the Unexpected: Sustained Performance in a Complex World, 3rd edn. Wiley, Hoboken (2015)
Winck, B.: 3 reasons why bitcoin has doubled in less than a month – and why experts think it won’t repeat its 2012 crash. Businessinsider (2021). https://markets.businessinsider.com/currencies/news/bitcoin-price-btc-performance-explanation-cryptocurrency-market-crash-token-investing-2021-1-1029945005
Winfield, A.F.T., Jirotka, M.: The case for an ethical black box. In: Gao, Y., Fallah, S., Jin, Y., Lekakou, C. (eds.) TAROS 2017. LNCS (LNAI), vol. 10454, pp. 262–273. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-64107-2_21
Wootton, D.: The Invention of Science: A New History of the Scientific Revolution. Penguin, Random House, London (2016)
Acknowledgment
This paper is part of the ETAIROS (https://etairos.fi/en/front-page/) funded by The Strategic Research Council (SRC) within the Academy of Finland.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Hallamaa, J. (2021). What Could Safety Research Contribute to Technology Design?. In: Rauterberg, M. (eds) Culture and Computing. Design Thinking and Cultural Computing. HCII 2021. Lecture Notes in Computer Science(), vol 12795. Springer, Cham. https://doi.org/10.1007/978-3-030-77431-8_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-77431-8_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77430-1
Online ISBN: 978-3-030-77431-8
eBook Packages: Computer ScienceComputer Science (R0)