Skip to main content

What Could Safety Research Contribute to Technology Design?

  • Conference paper
  • First Online:
Culture and Computing. Design Thinking and Cultural Computing (HCII 2021)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12795))

Included in the following conference series:

  • 1477 Accesses

Abstract

There is already evidence of several mishaps, accidents, and even major catastrophes in which AI technologies have played a part. Among them are traditional types of organizational accidents but also complex misadventures where the effects of the use of AI technology intertwine with human action in the context of intricate socio-technical settings. The origins of problems range from making errors during a planned course of action to accidents due to unintended and unpredictable outcomes of the use of AI. A variety of ethical principles for the use and design of AI have been formulated to remedy the ills. The article sums up the main concepts of the AI ethical codes and discusses their role in attempts to prevent AI-induced problems. According to recent empirical studies, the impact of ethics principles is weak in terms of real-life AI design and use. As an alternative, the article suggests the findings of safety research as a source for a practice-oriented approach to tackle the problems. A century-long study of safety provides both conceptual models and practical tools for the prevention of accidents and creating the means to improve safety. The article introduces the main findings of this line of research and suggests how the Normal Accident Theory, High Reliability Organizations studies, and Just Culture approach could contribute to a safer AI design.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Inkster [43], 1–17.

  2. 2.

    Coeckelbergh [15], 75–76.

  3. 3.

    Hagendorff [30], 101.

  4. 4.

    Hagendorff [30], 102.

  5. 5.

    Hagendorff [30], 102–103.

  6. 6.

    Hagendorff [30], 102–103; The European Commission’s High Level Expert Group.

  7. 7.

    Hagendorff [30], 103–104.

  8. 8.

    Hagendorff [30], 105.

  9. 9.

    Hagendorff [30], 105.

  10. 10.

    Hagendorff [30], 106.

  11. 11.

    Hagendorff [30], 107.

  12. 12.

    Hagendorff [30], 108.

  13. 13.

    Hagendorff [30], 108–109.

  14. 14.

    Boddington 2017, 56.

  15. 15.

    Hagendorff [30], 108–109.

  16. 16.

    Coeckelbergh [15], 157.

  17. 17.

    Trustworthy AI [22], 20.

  18. 18.

    Trustworthy AI [22], 25–31.

  19. 19.

    Trustworthy AI [22], 31.

  20. 20.

    Hagendorff [30], 112–113.

  21. 21.

    Coeckelbergh [15], 145–165.

  22. 22.

    Coeckelbergh [15], 162.

  23. 23.

    Clarke [14], 7.

  24. 24.

    Reiman & Oedewald [66], 39.

  25. 25.

    Reiman & Oedewald [66], 39–48.

  26. 26.

    See also Reiman & Oedewald [66], 39–47.

  27. 27.

    Reiman & Oedewald [66], 40.

  28. 28.

    Manuele [55], 236.

  29. 29.

    Manuele [55], 234–256.

  30. 30.

    Hogg & Vaughan [38], 641; Dekker [18], xxii–xxiii, 65.

  31. 31.

    Reiman & Oedewald [66], 40.

  32. 32.

    Reason [61], 9.

  33. 33.

    Reason [61], 53.

  34. 34.

    Reason [64], 2–3; Reason [61], 17.

  35. 35.

    Reason [61], 197.

  36. 36.

    Perrow 1999 [58], 11.

  37. 37.

    Maidment 1997 [53], 71.

  38. 38.

    Reason [63], 77; Reiman & Oedewald [66], 35.

  39. 39.

    Reason [63], 49–50.

  40. 40.

    Reiman & Oedewald [66], 44.

  41. 41.

    Perrow [58], 5.

  42. 42.

    Perrow [58], 321–350.

  43. 43.

    Hollnagel [40], 25, 28.

  44. 44.

    Reiman & Oedewald [66], 241–242.

  45. 45.

    Reiman & Oedewald [66], 246.

  46. 46.

    Dekker [18], 45.

  47. 47.

    Reason [62], 195–196.

References

  1. A walk on the dark side. The Economist (2007)

    Google Scholar 

  2. Abusus non tollit usum (s.a.). https://www.merriam-webster.com/dictionary/abusus%20non%20tollit%20usum

  3. AI for Good Global Summit 2017. ITU News (2017). https://www.itu.int/en/ITU-T/AI/Pages/201706-default.aspx

  4. AI Now: 2019 Report. AI Now Institute, New York University (2019). https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwiix8_MmcvuAhVFl4sKHVuLDWsQFjAAegQIAxAC&url=https%3A%2F%2Fainowinstitute.org%2FAI_Now_2019_Report.pdf&usg=AOvVaw2hwdlsKFcce1B1wW0ucWRL

  5. Alimahomed-Wilson, J., Reese, E.: The Cost of Free Shipping: Amazon in the Global Economy. Pluto Press, London (2020)

    Google Scholar 

  6. Auernhammer, J.: Human-centered AI: the role of human-centered design research in the development of AI. In: Boess, S., Cheung, M., Cain, R. (eds.) Synergy - DRS International Conference 2020, 11–14 August (2020). https://doi.org/10.21606/drs.2020.282

  7. Bayern, M.: How AI is spreading everywhere with the rise of smart machines. TechRepublic, Innovation (2018). https://www.techrepublic.com/article/how-ai-is-spreading-everywhere-with-the-rise-of-smart-machines/

  8. Bernstein, P.: Against the Gods: The Remarkable Story of Risk. Wiley, Hoboken (1998)

    Google Scholar 

  9. Blumberg, D.: What AI startups need to achieve before VCs will invest. Extra Crunch (2019). https://techcrunch.com/2019/11/13/what-ai-startups-need-to-achieve-before-vcs-will-invest/

  10. Bradshaw, S., Howard, P.N.: Challenging truth and trust. A global inventory of organized social media manipulation. Computational Propaganda Research Project. Oxford Internet Institute. University of Oxford, Oxford (2018). http://comprop.oii.ox.ac.uk/wp-content/uploads/sites/93/2018/07/ct2018.pdf

  11. Carback, J.T.: Cybersex trafficking: toward a more effective prosecutorial response. Crim. Law Bull. 54(1), 64–183 (2018)

    Google Scholar 

  12. Chaffey, D.: Global social media research summary August 2020. Smart Insights (2020). https://www.smartinsights.com/social-media-marketing/social-media-strategy/new-global-social-media-research/

  13. Chui, M., Harrysson, M., Manyika, J., Roberts, R., Chung, R., Nel, P., et al.: Applying artificial intelligence for social good. Discussion Paper. McKinsey Global Institute (2018). https://www.mckinsey.com/featured-insights/artificial-intelligence/applying-artificial-intelligence-for-social-good

  14. Clarke, M.: Insurance fraud. Br. J. Criminol. 29 (1), 1–20 (1989). https://www.jstor.org/stable/23638129?seq=8#metadata_info_tab_contents

  15. Coeckelbergh, M.: AI Ethics. The MIT Press Essential Knowledge Series. The MIT Press, Cambridge (2020)

    Google Scholar 

  16. Combating cyber crime: Department of Homeland Security (2012)

    Google Scholar 

  17. Cyber crime: Federal Bureau of Investigation (2017)

    Google Scholar 

  18. Dekker, S.: The Field Guide to Understanding ‘Human Error’, 3rd edn. CRC Press, Boca Raton (2014)

    Google Scholar 

  19. Dekker, S.W.A., Breakey, H.: ‘Just culture:’ improving safety by achieving substantive, procedural and restorative justice. Saf. Sci. 85, 187–193 (2016)

    Article  Google Scholar 

  20. Doris, J.: Lack of Character. Personality and Moral Behavior. Cambridge University Press, Cambridge (2002)

    Book  Google Scholar 

  21. Enya, A., Dempsey, S., Pillay, M.: High reliability organisation (HRO) principles of collective mindfulness: an opportunity to improve construction safety management. In: Arezes, P.M.F.M. (ed.) AHFE 2018. AISC, vol. 791, pp. 3–13. Springer, Cham (2019). https://doi.org/10.1007/978-3-319-94589-7_1

    Chapter  Google Scholar 

  22. Ethics guidelines for trustworthy AI. High-Level Expert Group on Artificial Intelligence. European Commission, Brussels (2019). https://ec.europa.eu/digital-single-market/en/news/ethics-guidelines-trustworthy-ai

  23. Feinberg, T.: Whether it happens at school or off-campus, cyberbullying disrupts and affects all aspects of students’ lives (2008). http://docplayer.net/17558112-Whether-it-happens-at-school-or-off-campus-cyberbullying-disrupts-and-affects-all-aspects-of-students-lives.html

  24. Floridi, L.: Faultless responsibility: on the nature and allocation of moral responsibility for distributed moral actions. Philos. Trans. Ser. Math. Phys. Eng. Sci. 374(2083), 1–13 (2016)

    Google Scholar 

  25. Floridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., Dignum, V., et al.: AI4People: an ethical framework for a good AI society: opportunities, risks, principles, and recommendations. Minds Mach. 28(4), 689–707 (2018)

    Article  Google Scholar 

  26. Fossheim, H.: Virtue ethics and everyday strategies. Revue Internationale de Philosophie 267(1), 65–82 (2014). https://doi.org/10.3917/rip.267.0065

    Article  Google Scholar 

  27. Fresneau, V.: 10 promising European deeptech startups leading us into the future. EU-Startups (2020). https://www.eu-startups.com/2020/05/10-promising-deeptech-startups/

  28. Gawande, A.: The Checklist Manifesto: How to Get Things Right. Profile Books, London (2011)

    Google Scholar 

  29. Grace, K., Salvatier, J., Dafoe, A., Zhang, B., Evans, O.: Viewpoint: when will AI exceed human performance? Evidence from AI experts. J. Artif. Intell. Res. 62, 729–754 (2018)

    Article  MathSciNet  Google Scholar 

  30. Hagendorff, T.: The ethics of AI ethics: an evaluation of guidelines. Minds Mach. 30, 99–120 (2020). https://doi.org/10.1007/s11023-020-09517-8

    Article  Google Scholar 

  31. Halder, D., Jaishankar, K.: Cyber Crime and the Victimization of Women: Laws, Rights, and Regulations. IGI Global, Hershey (2011)

    Google Scholar 

  32. Hanington, B.: Methods in making: a perspective on the state of human research in design. Des. Issues 19(4) 9–18 (2003). Massachusetts Institute of Technology. http://scholar.google.fi/scholar_url?url=https://www.ida.liu.se/divisions/hcs/ixs/material/servicedesignGbg10/2%2520Additional%2520readings/humancentereddesign.pdf&hl=fi&sa=X&ei=iXYaYPTGCY-Ny9YP3o6D2AY&scisig=AAGBfm0uqeu6IdfPOI3kEXfH1CUmKFZKEA&nossl=1&oi=scholarr

    Google Scholar 

  33. Hao, K.: In 2020, let’s stop AI ethics-washing and actually do something. MIT Technol. Rev. (2019). https://www.technologyreview.com/2019/12/27/57/ai-ethics-washing-time-to-act/

  34. Harman, G.: Moral philosophy meets social psychology: virtue ethics and the fundamental attribution error. In: Proceedings of the Aristotelian Society, vol. 99, pp. 315–356 (1998–1999)

    Google Scholar 

  35. Harper, J.: Tesla: Soaring share price creates army of ‘Teslanaires’. BBC News (2020). https://www.bbc.com/news/business-55391571

  36. Heinrich, H.W.: Industrial Accident Prevention. McGraw-Hill, New York (1931)

    Google Scholar 

  37. Hibbard, B.: Ethical Artificial Intelligence (2015). https://arxiv.org/abs/1411.1373

  38. Hogg, M.A., Vaughan, G.M.: Social Psychology, 5th edn. Pearson Education Limited, Harlow (2008)

    Google Scholar 

  39. Holland, J. H.: Complex adaptive systems. Daedalus 121(1), 17–30 (1992). A New Era in Computation. https://www.jstor.org.stable/20025416

  40. Hollnagel, E.: The ETTO Principle: Efficiency-Thoroughness Trade-off: Why Things That Go Right Sometimes Go Wrong. Ashgate, Farnham (2009)

    Google Scholar 

  41. Hollnagel, E.: Safety-I and Safety-II: The Past and Future of Safety Management. CRC Press, Boca Raton (2014)

    Google Scholar 

  42. HRO Safety Culture Definition: United States Nuclear Regulatory Commission and United States Government Accountability Office (2010)

    Google Scholar 

  43. Inkster, I.: Science and Technology in History: An Approach to Industrial Development. Macmillan Educational, New Brunswick (1991)

    Book  Google Scholar 

  44. Johnson, A.: Examining the foundation. Saf. Health Mag. (2011). https://www.safetyandhealthmagazine.com/articles/print/6368-examining-the-foundation

  45. Just Culture. Skybrary. https://www.skybrary.aero/index.php/Just_Culture

  46. Kibby, M.: The impact of AI on human intelligence: can we live without it? AmeInfo (2020). https://www.ameinfo.com/industry/tech-and-mobility/the-impact-of-ai-on-human-intelligence

  47. Knapp, A.: Six billionaires who made their fortunes from videogames. Forbes (2017). https://www.forbes.com/sites/alexknapp/2017/03/21/six-billionaires-who-made-fortunes-from-videogames/

  48. Kranzberg, M.: Technology and history: “Kranzberg’s laws”. Technol. Cult. 27(3), 544–560 (1986). https://doi.org/10.2307/3105385

    Article  Google Scholar 

  49. Krupiv, T.: A vulnerability analysis: theorising the impact of artificial intelligence decision-making processes on individuals, society and human diversity from a social justice perspective. Comput. Law Secur. Rev. 38 (2020). https://doi.org/10.1016/j.clsr.2020.105429

  50. Lin, P., Abney, K., Jenkins, R. (eds.): Robot ethics 2.0: From autonomous cars to artificial intelligence. Oxford Scholarship Online (2017)

    Google Scholar 

  51. MacIntyre, A.: After Virtue: A Study in Moral Theory. University of Notre Dame Press, Notre Dame (1981)

    Google Scholar 

  52. McNamara, A., Smith, J., Murphy-Hill, E.: Does ACM’s code of ethics change ethical decision making in software development? In: Leavens, G.T., Garcia, A., Păsăreanu C.S. (eds.) Proceedings of the 2018 26th ACM joint meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering—ESEC/FSE 2018, pp. 1–7. ACM Press, New York (2018)

    Google Scholar 

  53. Maidment, D.: The effect of the accident inquiry process within the railway industry. In: Hale, A., Wilpert, B., Freitag, M. (eds.) After the Event. From Accident to Organisational Learning, Pergamon, Oxford, pp. 61–76 (1997)

    Google Scholar 

  54. Mann, D., Sutton, M.: Netcrime. Br. J. Criminol. 38(2) 201–229 (2011). CiteSeerX 10.1.1.133.3861

    Google Scholar 

  55. Manuele, P.A.: On the Practice of Safety, 4th edn. Wiley, Hoboken (2013)

    Book  Google Scholar 

  56. Marr, B.: Artificial Intelligence in Practice: How 50 Successful Companies used AI and Machine Learning to Solve Problems. Wiley, Chichester (2019)

    Google Scholar 

  57. Osoba, O.A., Welser IV, W.: The risks of artificial intelligence to security and the future of work. Perspective. Expert insights Timely Policy Issue. https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwigjOPD68ruAhXsiIsKHQ98CIUQFjALegQIJRAC&url=https%3A%2F%2Fwww.rand.org%2Fcontent%2Fdam%2Frand%2Fpubs%2Fperspectives%2FPE200%2FPE237%2FRAND_PE237.pdf&usg=AOvVaw2RDrXdeOZjnxJrVoXqV0pk

  58. Perrow, C.: Normal Accidents: Living with High-Risk Technologies. Princeton University Press, Princeton (1999)

    Google Scholar 

  59. Perrow, C.: The Next Catastrophe: Reducing Our Vulnerabilities to Natural, Industrial, and Terrorist Disasters. Princeton University Press, Princeton (2007)

    Google Scholar 

  60. Railway Passengers Assurance Company Ltd. http://www.aviva.com/about-us/heritage/companies/railway-passengers-assurance-company/

  61. Reason, J.: Human Error. Cambridge University Press, Cambridge (1990)

    Book  Google Scholar 

  62. Reason, J.: Managing the Risks of Organizational Accidents. Routledge, London and New York (1997)

    Google Scholar 

  63. Reason, J.: The Human Contribution. Unsafe Acts, Accidents and Heroic Recoveries. Routledge, New York (2008)

    Google Scholar 

  64. Reason, J.: Organizational Accidents Revisited. Ashgate, Farnham (2016)

    Book  Google Scholar 

  65. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation) (2016). http://data.europa.eu/eli/reg/2016/679/oj

  66. Reiman, T., Oedewald, P.: Turvallisuuskriittiset organisaatiot: Onnettomuudet, kulttuuri ja johtaminen. (Safety-Critical Organizations: Accidents, Culture, and Leadership) Edita, Helsinki (2008)

    Google Scholar 

  67. Resilience Engineering (2019). https://www.resilience-engineering-association.org/blog/2019/11/09/what-is-resilience-engineering/

  68. Roberts, K.H.: New challenges in organizational research: high reliability organizations. Organ. Environ. 3(2), 111–125 (1989). https://doi.org/10.1177/108602668900300202

    Article  Google Scholar 

  69. Robertson, K., Black, K., Grand-Clement, S., Hall, A.: Human and Organisational Factors in Major Accident Prevention: A Snapshot of the Academic Landscape. RAND Corporation, Santa Monica (2016). https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwiipdWKqMvuAhVwAxAIHaWZD00QFjATegQIChAC&url=https%3A%2F%2Fwww.rand.org%2Fcontent%2Fdam%2Frand%2Fpubs%2Fresearch_reports%2FRR1500%2FRR1512%2FRAND_RR1512.pdf&usg=AOvVaw27bcYG_JDYDOKlcAn5bSe1

  70. Rosenberg, S.: Why AI is still waiting for its ethics transplant. Wired (2017). https://www.wired.com/story/why-ai-is-still-waiting-for-its-ethics-transplant/

  71. Rossi, B.: Has AI become something we cannot live without? Information Age (2016). https://www.information-age.com/has-ai-become-something-we-cant-live-without-123461456/

  72. Russell, C.: A case for not regulating the development of artificial intelligence. Towards Data Science (2019). https://towardsdatascience.com/a-case-for-not-regulating-the-development-of-artificial-intelligence-f3d23db2e8cd

  73. Sagan, S.D.: The Limits of Safety. Organizations, Accidents, and Nuclear Weapons. Princeton University Press, Princeton (1993)

    Google Scholar 

  74. Sammarco, J.J.: Operationalizing normal accident theory for safety-related computer systems. Saf. Sci. 43(9), 697–714 (2005). https://doi.org/10.1016/j.ssci.2005.03.001

    Article  Google Scholar 

  75. Saran, S., Natrajan, N., Srikumar, M.: In Pursuit of Autonomy: AI and National Strategies. Observer Research Foundation, New Delhi (2018)

    Google Scholar 

  76. Scherer, M.U.: Regulating artificial intelligence systems: risks, challenges, competencies, and strategies. Harv. J. Law Technol. 29(2) (2016). https://doi.org/10.2139/ssrn.2609777

  77. Schmeltzer, R.: The complex nature of regulating AI. SearchEnterpriseAI. TechTarget (2019). https://searchenterpriseai.techtarget.com/feature/The-complex-nature-of-regulating-AI

  78. Shrivastava, S., Sonpar, K., Pazzaglia, F.: Normal accident theory versus high reliability theory: a resolution and call for an open systems view of accidents. Hum. Relat. 62(9), 1357–1390 (2009). https://doi.org/10.1177/0018726709339117

    Article  Google Scholar 

  79. Stametalatos, M.: Probabilistic Risk Assessment: What is It and Why Is It Worth Performing It? (2000). http://www.hq.nasa.gov/office/codecq/qnews/prs.pdf

  80. Stultz, J.: High Reliability Organization Toolkit. Manage Operations (2018). https://web.mhanet.com/media-library/high-reliability-organization-toolkit/

  81. The Montréal Declaration for a Responsible Development of Artificial Intelligence. Inven_T, University of Montreal’s Technosocial Innovation Centre (2018). https://www.montrealdeclaration-responsibleai.com/

  82. Vanderhaegen, F., Hollnagel, E.: Safety-I and Safety-II, the past and future of safety management. Cognit. Technol. Work 17(3), 461–464 (2015). https://doi.org/10.1007/s10111-015-0345-z

  83. Wagner, B.: Ethics as an escape from regulation. From “ethicswashing” to ethics-shopping? Repositorium für die Medienwissenschaft (2018). https://doi.org/10.25969/mediarep/13281

  84. Why Air Travel is The Safest Mode of Transportation? Sheffield School of Aeronautics. https://www.sheffield.com/air-travel-safest-mode-transportation

  85. Why is regulating artificial intelligence so tricky? The Startup (2020). https://medium.com/swlh/why-is-regulating-artificial-intelligence-so-tricky-f202c967c2b4

  86. Weick, K.E., Sutcliffe, K.M.: Managing the Unexpected: Assuring High Performance in an Age of Complexity. Jossey Bass, San Francisco (2001)

    Google Scholar 

  87. Weick, K.E., Sutcliffe, K.M.: Managing the Unexpected: Sustained Performance in a Complex World, 3rd edn. Wiley, Hoboken (2015)

    Google Scholar 

  88. Winck, B.: 3 reasons why bitcoin has doubled in less than a month – and why experts think it won’t repeat its 2012 crash. Businessinsider (2021). https://markets.businessinsider.com/currencies/news/bitcoin-price-btc-performance-explanation-cryptocurrency-market-crash-token-investing-2021-1-1029945005

  89. Winfield, A.F.T., Jirotka, M.: The case for an ethical black box. In: Gao, Y., Fallah, S., Jin, Y., Lekakou, C. (eds.) TAROS 2017. LNCS (LNAI), vol. 10454, pp. 262–273. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-64107-2_21

    Chapter  Google Scholar 

  90. Wootton, D.: The Invention of Science: A New History of the Scientific Revolution. Penguin, Random House, London (2016)

    Google Scholar 

Download references

Acknowledgment

This paper is part of the ETAIROS (https://etairos.fi/en/front-page/) funded by The Strategic Research Council (SRC) within the Academy of Finland.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jaana Hallamaa .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hallamaa, J. (2021). What Could Safety Research Contribute to Technology Design?. In: Rauterberg, M. (eds) Culture and Computing. Design Thinking and Cultural Computing. HCII 2021. Lecture Notes in Computer Science(), vol 12795. Springer, Cham. https://doi.org/10.1007/978-3-030-77431-8_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-77431-8_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-77430-1

  • Online ISBN: 978-3-030-77431-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics