Phoebe Li
ABSTRACT
The recent achievements of Artificial intelligence (AI) open up opportunities for new tools to assist medical diagnosis and care delivery. However, the optimal process for the development of AI is through repeated cycles of learning and implementation that may pose challenges to our existing system of regulating medical devices. Product developers face the tensions between the benefits of continuous improvement/deployment of algorithms and of keeping products unchanged to collect evidence for safety assurance processes. The challenge is how to balance potential benefits with the need to assure their safety. Governance and assurance requirements that can accommodate the live or near-live machine learning (ML) approach will be needed soon, as it is an approach likely to soon be of high importance in healthcare and in other fields of application.
We have entered a phase of regulatory experimentation with various novel approaches emerging around the world. The process of social learning is not only about the application of AI but also about the institutional arrangements for its safe and dependable deployment, including regulatory experimentation, likely within sandboxes.
This paper will reflect on the discussions from two recent Chatham House workshops on regulating AI in software as a medical device (SaMD), hosted by the UKRI/EPSRC project on ‘Trustworthy Autonomous Systems: Regulation and Governance’ node, with a special focus on the recent regulatory attempts in the UK and internationally.
- Elisabetta Biasin, Erik Kamenjasevic, and Kaspar Rosager Ludvigsen. 2023. Cybersecurity of AI medical devices: risks, legislation, and challenges. https://doi.org/10.48550/arXiv.2303.03140Google Scholar
Cross Ref
- Nicola Davis. 2021. AI skin cancer diagnoses risk being less accurate for dark skin – study. https://www.theguardian.com/society/2021/nov/09/ai-skin-cancer-diagnoses-risk-being-less-accurate-for-dark-skin-studyGoogle Scholar
- EU. 2016. EUR-Lex - 310401_2 – General data protection regulation (GDPR). https://eur-lex.europa.eu/EN/legal-content/summary/general-data-protection-regulation-gdpr.htmlGoogle Scholar
- Luciano Floridi. 2020. The Fight for Digital Sovereignty: What It Is, and Why It Matters, Especially for the EU. Philosophy & Technology 33, 3 (Sept. 2020), 369–378. https://doi.org/10.1007/s13347-020-00423-6Google ScholarCross Ref
- Karoline Freeman, Julia Geppert, Chris Stinton, Daniel Todkill, Samantha Johnson, Aileen Clarke, and Sian Taylor-Phillips. 2021. Use of artificial intelligence for image analysis in breast cancer screening programmes: systematic review of test accuracy. BMJ 374 (2021), n1872. https://doi.org/10.1136/bmj.n1872 arXiv:https://www.bmj.com/content/374/bmj.n1872.full.pdfGoogle ScholarCross Ref
- Marzyeh Ghassemi, Luke Oakden-Rayner, and Andrew L. Beam. 2021. The false hope of current approaches to explainable artificial intelligence in health care. The Lancet Digital Health 3, 11 (Nov. 2021), e745–e750. https://doi.org/10.1016/S2589-7500(21)00208-9Google ScholarCross Ref
- Stephen Gilbert, Stuart Anderson, Martin Daumer, Phoebe Li, Tom Melvin, and Robin Williams. 2023. Learning From Experience and Finding the Right Balance in the Governance of Artificial Intelligence and Digital Health Technologies. Journal of Medical Internet Research 25 (April 2023), e43682. https://doi.org/10.2196/43682Google ScholarCross Ref
- Stephen Gilbert, Matthew Fenech, Martin Hirsch, Shubhanan Upadhyay, Andrea Biasiucci, and Johannes Starlinger. 2021. Algorithm Change Protocols in the Regulation of Adaptive Machine Learning–Based Medical Devices. Journal of Medical Internet Research 23, 10 (Oct. 2021), e30545. https://doi.org/10.2196/30545Google ScholarCross Ref
- UK Government. 2022. Regulatory Horizons Council: The regulation of Artificial Intelligence as a Medical Device. https://www.gov.uk/government/publications/regulatory-horizons-council-the-regulation-of-artificial-intelligence-as-a-medical-deviceGoogle Scholar
- UK Government. 2022. Software and AI as a Medical Device Change Programme. https://www.gov.uk/government/publications/software-and-ai-as-a-medical-device-change-programmeGoogle Scholar
- Center for Devices and Radiological Health. 2022. Artificial Intelligence and Machine Learning in Software as a Medical Device. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-deviceGoogle Scholar
- Hussein Ibrahim, Xiaoxuan Liu, Nevine Zariffa, Andrew D. Morris, and Alastair K. Denniston. 2021. Health data poverty: an assailable barrier to equitable digital health care. The Lancet. Digital Health 3, 4 (April 2021), e260–e265. https://doi.org/10.1016/S2589-7500(20)30317-4Google ScholarCross Ref
- IEEE Standard. 2021. IEEE Standard for Transparency of Autonomous Systems. https://standards.ieee.org/ieee/7001/6929/Google Scholar
- IMDRF. 2021. Machine Learning-enabled Medical Devices - A subset of Artificial Intelligence-enabled Medical Devices: Key Terms and Definitions. https://www.imdrf.org/consultations/machine-learning-enabled-medical-devices-subset-artificial-intelligence-enabled-medical-devices-key-terms-and-definitionsGoogle Scholar
- ITU. 2018. Focus Group on "Artificial Intelligence for Health". https://www.itu.int:443/en/ITU-T/focusgroups/ai4h/Pages/default.aspxGoogle Scholar
- PMDA (Japan). 2017. Subcommittee on AI (Artificial Intelligence) | Pharmaceuticals and Medical Devices Agency. https://www.pmda.go.jp/english/rs-sb-std/sb/subcommittees-3rd/0024.htmlGoogle Scholar
- Saad M. Khan, Xiaoxuan Liu, Siddharth Nath, Edward Korot, Livia Faes, Siegfried K. Wagner, Pearse A. Keane, Neil J. Sebire, Matthew J. Burton, and Alastair K. Denniston. 2021. A global review of publicly available datasets for ophthalmological imaging: barriers to access, usability, and generalisability. The Lancet Digital Health 3, 1 (Jan. 2021), e51–e66. https://doi.org/10.1016/S2589-7500(20)30240-5Google ScholarCross Ref
- Phoebe Li, Alex Faulkner, and Nicholas Medcalf. 2020. 3D bioprinting in a 2D regulatory landscape: gaps, uncertainties, and problems. Law, Innovation and Technology 12, 1 (Jan. 2020), 1–29. https://doi.org/10.1080/17579961.2020.1727054Google ScholarCross Ref
- Kaspar Ludvigsen, Shishir Nagaraja, and Angela Daly. 2022. When Is Software a Medical Device? Understanding and Determining the “Intention” and Requirements for Software as a Medical Device in European Union Law. European Journal of Risk Regulation 13, 1 (March 2022), 78–93. https://doi.org/10.1017/err.2021.45Google ScholarCross Ref
- Scott Mayer McKinney, Marcin Sieniek, Varun Godbole, Jonathan Godwin, Natasha Antropova, Hutan Ashrafian, Trevor Back, Mary Chesus, Greg S. Corrado, Ara Darzi, Mozziyar Etemadi, Florencia Garcia-Vicente, Fiona J. Gilbert, Mark Halling-Brown, Demis Hassabis, Sunny Jansen, Alan Karthikesalingam, Christopher J. Kelly, Dominic King, Joseph R. Ledsam, David Melnick, Hormuz Mostofi, Lily Peng, Joshua Jay Reicher, Bernardino Romera-Paredes, Richard Sidebottom, Mustafa Suleyman, Daniel Tse, Kenneth C. Young, Jeffrey De Fauw, and Shravya Shetty. 2020. International evaluation of an AI system for breast cancer screening. Nature 577, 7788 (Jan. 2020), 89–94. https://doi.org/10.1038/s41586-019-1799-6Google ScholarCross Ref
- Timo Minssen, Sara Gerke, Mateo Aboy, Nicholson Price, and Glenn Cohen. 2020. Regulatory responses to medical machine learning. Journal of Law and the Biosciences 7, 1 (July 2020), lsaa002. https://doi.org/10.1093/jlb/lsaa002Google ScholarCross Ref
- Jessica Morley, Caio C. V. Machado, Christopher Burr, Josh Cowls, Indra Joshi, Mariarosaria Taddeo, and Luciano Floridi. 2020. The ethics of AI in health care: A mapping review. Social Science & Medicine (1982) 260 (Sept. 2020), 113172. https://doi.org/10.1016/j.socscimed.2020.113172Google ScholarCross Ref
- Jakob Mökander, Prathm Juneja, David S. Watson, and Luciano Floridi. 2022. The US Algorithmic Accountability Act of 2022 vs. The EU Artificial Intelligence Act: what can they learn from each other?Minds and Machines 32, 4 (Dec. 2022), 751–758. https://doi.org/10.1007/s11023-022-09612-yGoogle ScholarDigital Library
- Council of the European Union. 1990. Council Directive 90/385/EEC of 20 June 1990 on the approximation of the laws of the Member States relating to active implantable medical devices. http://data.europa.eu/eli/dir/1990/385/oj/engGoogle Scholar
- Council of the European Union. 2017. Medical devices. https://single-market-economy.ec.europa.eu/single-market/european-standards/harmonised-standards/medical-devices_enGoogle Scholar
- Council of the European Union. 2021. Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts - Presidency compromise text. https://data.consilium.europa.eu/doc/document/ST-14278-2021-INIT/en/pdfGoogle Scholar
- European Parliament. 2017. Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC (Text with EEA relevance.). http://data.europa.eu/eli/reg/2017/745/oj/engGoogle Scholar
- Huw Roberts, Josh Cowls, Emmie Hine, Francesca Mazzi, Andreas Tsamados, Mariarosaria Taddeo, and Luciano Floridi. 2021. Achieving a ‘Good AI Society’: Comparing the Aims and Progress of the EU and the US. Science and Engineering Ethics 27, 6 (Nov. 2021), 68. https://doi.org/10.1007/s11948-021-00340-7Google ScholarCross Ref
- Graham Spinardi and Robin Williams. 2001. Environmental Innovation in Refining and Chemicals. In Ahead of the Curve: Cases of Innovation in Environmental Management, Ken Green, Peter Groenewegen, and Peter S. Hofman (Eds.). Springer Netherlands, Dordrecht, 165–191. https://doi.org/10.1007/978-94-010-0908-9_10Google ScholarCross Ref
- UK Office for Product Safety and Standards. 2021. Study on the impact of artificial intelligence on product safety. https://www.gov.uk/government/publications/study-on-the-impact-of-artificial-intelligence-on-product-safetyGoogle Scholar
- Effy Vayena, Alessandro Blasimme, and I. Glenn Cohen. 2018. Machine learning in medicine: Addressing ethical challenges. PLOS Medicine 15, 11 (Nov. 2018), e1002689. https://doi.org/10.1371/journal.pmed.1002689Google ScholarCross Ref
- Effy Vayena, Alessandro Blasimme, and I. Glenn Cohen. 2018. Machine learning in medicine: Addressing ethical challenges. PLOS Medicine 15, 11 (Nov. 2018), e1002689. https://doi.org/10.1371/journal.pmed.1002689Google ScholarCross Ref
- Ernst Wellnhofer. 2022. Real-World and Regulatory Perspectives of Artificial Intelligence in Cardiovascular Imaging. Frontiers in Cardiovascular Medicine 9 (2022), online. https://doi.org/10.3389/fcvm.2022.890809Google ScholarCross Ref
Index Terms
- Regulating AI/ML-enabled Medical Devices in the UK
Recommendations
Regulating artificial intelligence: a technology regulator's perspective
ICAIL '21: Proceedings of the Eighteenth International Conference on Artificial Intelligence and LawArtificial Intelligence (AI) and the regulation thereof is a topic that is increasingly being discussed and various proposals have been made in literature for defining regulatory bodies and/or related regulation. In this paper, we present a pragmatic ...
A Systematic Review: Role of Artificial Intelligence During the COVID-19 Pandemic in the Healthcare System
Certainty, artificial intelligence (AI) has a remarkable role in the control of the COVID-19 pandemic globally. This review article aimed to determine the outcomes/benefits associated with the use of artificial intelligence during the COVID-19 pandemic ...
Application of Artificial Intelligence Technology in the Field of Traditional Chinese Medicine
ISAIMS '22: Proceedings of the 3rd International Symposium on Artificial Intelligence for Medicine SciencesArtificial Intelligence (AI), as a broad cross and frontier science, has been integrated into many fields of human society. It has also been innovatively developed in the field of Traditional Chinese Medicine (TCM), and the achievements of artificial ...
Comments