Abstract
Artificial intelligence (AI) is a key technology that is utilized in autonomous systems. However, using AI in such systems introduces also several safety challenges, many of which are related to complex human-AI interactions. In general, these challenges relate to the changing roles of the people who interact with the increasingly autonomous systems in various ways. In this paper, we consider a set of practical safety challenges related to applying advanced AI to autonomous machine systems and present solutions from the perspective of human factors. We apply the novel concept of AI awareness (AIA) to discuss the challenges in detail, and based on this, provide suggestions and guidelines for mitigating the AI safety risks with autonomous systems. In addition, we briefly consider the system design process to identify the actions that should be taken to ensure AIA throughout the different systems engineering design phases. The theoretical research presented in this paper aims to provide the first steps towards considering AIA in autonomous systems and understanding the human factors perspective viewpoint on AI safety for autonomous systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
National Aeronautics and Space Administration.: NASA technology roadmaps – TA 4: robotics and autonomous systems (2015). https://www.nasa.gov/sites/default/files/atoms/files/2015_nasa_technology_roadmaps_ta_4_robotics_and_autonomous_systems_final.pdf. Accessed 23 Feb 2020
Fong, T.: Autonomous systems – NASA capability overview (2018). https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20180007804.pdf. Accessed 23 Feb 2020
Miranda, D.: 2020 NASA technology taxonomy – TX10: autonomous systems (2020). https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20200000399.pdf. Accessed 23 Feb 2020
Pendleton, S., et al.: Perception, planning, control, and coordination for autonomous vehicles. Machines 5(1), 6 (2017)
SAE International: SAE-J3016 Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles (2014)
Papernot, N., Mcdaniel, P., Jha, S., Fredrikson, M., Celik, Z.B., Swami, A.: The limitations of deep learning in adversarial settings. In: IEEE European Symposium on Security and Privacy (EuroS&P), pp. 372–387. IEEE (2016)
Moore, A.: Carnegie Mellon Dean Of Computer Science On The Future Of AI. Forbes (2017). https://www.forbes.com/sites/peterhigh/2017/10/30/carnegie-mellon-dean-of-computer-science-on-the-future-of-ai/#3d4124e72197. Accessed 23 Feb 2020
Endsley, M.R.: Situation awareness global assessment technique (SAGAT). In: Proceedings of the National Aerospace and Electronics Conference (NAECON), pp. 789–795. IEEE, New York (1988)
Endsley, M.R.: Measurement of situation awareness in dynamic systems. Hum. Factors 37(1), 65–84 (1995)
Karvonen, H., Liinasuo, M., Lappalainen, J.: Assessment of automation awareness. In: Automaatio XXI Conference, vol. 44 (2015)
Karvonen, H., Liinasuo, M., Lappalainen, J.: Assessment of situation and automation awareness, SAFIR2014 research report, VTT Research Report VTT-R-05997-14 (2014)
Karvonen, H., Heikkilä, E., Wahlström, M.: Artificial intelligence awareness in work environments. In: Barricelli, B.R., et al. (eds.) HWID 2018. IAICT, vol. 544, pp. 175–185. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05297-3_12
Theodorou, A., Wortham, R.H., Bryson, J.J.: Designing and implementing transparency for real time inspection of autonomous robots. Conn. Sci. 29(3), 230–241 (2017)
Doran, D., Schulz, S., Besold, T.R.: What does explainable AI really mean? A new conceptualization of perspectives. arXiv preprint arXiv:1710.00794 (2017)
Kurd, Z., Kelly, T., Austin, J.: Developing artificial neural networks for safety critical systems. Neural Comput. Appl. 16(1), 11–19 (2007). https://doi.org/10.1007/s00521-006-0039-9
Fournaris, A.P., Lalos, A.S., Serpanos, D.: Generative adversarial networks in AI-enabled safety-critical systems: friend or foe? Computer 52(9), 78–81 (2019)
Cummings, M.L., Britton, D.: Regulating safety-critical autonomous systems: past, present, and future perspectives. In: Living with Robots, pp. 119–140. Academic Press (2020)
Yampolskiy, R.V.: Artificial Intelligence Safety and Security. Chapman and Hall/CRC, Boca Raton (2018)
Yampolskiy, R.V.: Artificial intelligence safety engineering: Why machine ethics is a wrong approach. In: Müller, V. (ed.) Philosophy and Theory of Artificial Intelligence. Studies in Applied Philosophy, Epistemology and Rational Ethics, vol. 5, pp. 389–396. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-31674-6_29
Yampolskiy, R.V.: Taxonomy of pathways to dangerous artificial intelligence. In: Workshops at the Thirtieth AAAI Conference on Artificial Intelligence (2016)
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., Mané, D.: Concrete problems in AI safety. arXiv preprint arXiv:1606.06565 (2016)
Mohseni, S., Pitale, M., Singh, V., Wang, Z.: Practical solutions for machine learning safety in autonomous vehicles. arXiv preprint arXiv:1912.09630, (2019)
Alexander, R.D., Ashmore, R., Banks, A.: The state of solutions for autonomous systems safety (2018). http://eprints.whiterose.ac.uk/127573/1/02_ALEXANDER_ASHMORE_AND_BANKS_THE_STATE_OF_SOLUTIONS_FOR_AUTONOMOUS_SYSTEM_SAFETY.PDF. Accessed 23 Feb 2020
Norros, L.: Acting Under Uncertainty – The Core-Task Analysis in Naturalistic Study of Work and Expertise. VTT, Espoo (2004)
Vicente, K.J.: Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work. CRC Press, Mahwah (1999)
Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1(5), 206–215 (2019)
Lee, J.D., See, K.A.: Trust in automation: designing for appropriate reliance. Hum. Factors J. Hum. Factors Ergon. Soc. 46(1), 50–80 (2004)
Forsberg, K., Mooz, H.: The relationship of systems engineering to the project cycle. Eng. Manage. J. 4(3), 36–43 (1992)
Fairley, R., Thayer, R.: The concept of operations: the bridge from operational requirements to technical specifications. Ann. Soft. Eng. 3(1), 417–432 (1997). https://doi.org/10.1023/A:1018985904689
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Karvonen, H., Heikkilä, E., Wahlström, M. (2020). Safety Challenges of AI in Autonomous Systems Design – Solutions from Human Factors Perspective Emphasizing AI Awareness. In: Harris, D., Li, WC. (eds) Engineering Psychology and Cognitive Ergonomics. Cognition and Design. HCII 2020. Lecture Notes in Computer Science(), vol 12187. Springer, Cham. https://doi.org/10.1007/978-3-030-49183-3_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-49183-3_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-49182-6
Online ISBN: 978-3-030-49183-3
eBook Packages: Computer ScienceComputer Science (R0)