skip to main content
10.1145/3447548.3470823acmconferencesArticle/Chapter ViewAbstractPublication PageskddConference Proceedingsconference-collections
abstract

Software as a Medical Device: Regulating AI in Healthcare via Responsible AI

Published: 14 August 2021 Publication History

Abstract

With the increased adoption of AI in healthcare, there is a growing recognition and demand to regulate AI in healthcare to avoid potential harm and unfair bias against vulnerable populations. Around a hundred governmental bodies and commissions as well as leaders in the tech sector have proposed principles to create responsible AI systems. However, most of these proposals are short on specifics which has led to charges of ethics washing. In this tutorial we offer a guide to help navigate through complex governmental regulations and explain the various constituent practical elements of a responsible AI system in healthcare in the light of proposed regulations. Additionally, we breakdown and emphasize that the recommendations from regulatory bodies like FDA or the EU are necessary but not sufficient elements of creating a responsible AI system. We elucidate how regulations and guidelines often focus on epistemic concerns to the detriment of practical concerns e.g., requirement for fairness without explicating what fairness constitutes for a use case. FDA's Software as a medical device document and EU's GDPR among other AI governance documents talk about the need for implementing sufficiently good machine learning practices. In this tutorial we elucidate what that would mean from a practical perspective for real world use cases in healthcare throughout the machine learning cycle i.e., Data Management, Data Specification, Feature Engineering, Model Evaluation, Model Specification, Model Explainability, Model Fairness, Reproducibility, checks for data leakage and model leakage. We note that conceptualizing responsible AI as a process rather than an end goal accords well with how AI systems are used in practice. We also discuss how a domain centric stakeholder perspective translates into balancing requirements for multiple competing optimization criteria.

Supplementary Material

MP4 File (SaMDKDDTutorialShort20210712.mp4)
Short version of the presentation video for the KDD 2021 Tutorial Software as a Medical Device: Regulating AI in Healthcare via Responsible AI

References

[1]
Larry Adelman. 2007. Unnatural causes: Is inequality making us sick? Preventing Chronic Disease 4, 4 (2007).
[2]
Muhammad Aurangzeb Ahmad, Arpit Patel, Carly Eckert, Vikas Kumar, and Ankur Teredesai. 2020. Fairness in Machine Learning for Healthcare. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 3529--3530.
[3]
Stan Benjamens, Pranavsingh Dhunnoo, and Bertalan Meskó. 2020. The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database. NPJ digital medicine 3, 1 (2020), 1--8.
[4]
Esther H Chen, Frances S Shofer, Anthony J Dean, Judd E Hollander, William G Baxt, Jennifer L Robey, Keara L Sease, and Angela M Mills. 2008. Gender disparity in analgesic treatment of emergency department patients with acute abdominal pain. Academic Emergency Medicine 15, 5 (2008), 414--418.
[5]
Sam Corbett-Davies and Sharad Goel. 2018. The measure and mismeasure of fairness: A critical review of fair machine learning. arXiv preprint arXiv:1808.00023 (2018).
[6]
Allan Dafoe. 2018. AI governance: a research agenda. Governance of AI Program, Future of Humanity Institute, University of Oxford: Oxford, UK (2018).
[7]
Robyn M Dawes, David Faust, and Paul E Meehl. 1989. Clinical versus actuarial judgment. Science 243, 4899 (1989), 1668--1674.
[8]
Barbara J Evans and Frank A Pasquale. 2020. Product Liability Suits for FDA-Regulated AI/ML Software. The Future of Medical Device Regulation: Innovation and Protection (I. Glenn Cohen, Timo Minssen, W. Nicholson Price II, Christopher Robertson & Carmel Shachar eds., Cambridge University Press, 2021 forthcoming) (2020).
[9]
Urs Gasser and Virgilio AF Almeida. 2017. A layered model for AI governance. IEEE Internet Computing 21, 6 (2017), 58--62.
[10]
Marzyeh Ghassemi, Tristan Naumann, Peter Schulam, Andrew L Beam, and Rajesh Ranganath. 2018. Opportunities in machine learning for healthcare. arXiv preprint arXiv:1806.00388 (2018).
[11]
Thomas Grote and Philipp Berens. 2020. On the ethics of algorithmic decision making in healthcare. Journal of medical ethics 46, 3 (2020), 205--211.
[12]
H Benjamin Harvey and Vrushab Gowda. 2020. How the FDA regulates AI. Academic radiology 27, 1 (2020), 58--61.
[13]
Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudik, and Hanna Wallach. 2019. Improving fairness in machine learning systems: What do industry practitioners need?. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1--16.
[14]
Filippo Pesapane, Caterina Volonté, Marina Codari, and Francesco Sardanelli. 2018. Artificial intelligence as a medical device in radiology: ethical and regulatory issues in Europe and the United States. Insights into imaging 9, 5 (2018), 745--753.
[15]
Wenjun Wu, Tiejun Huang, and Ke Gong. 2020. Ethical principles and governance technology development of AI in China. Engineering 6, 3 (2020), 302--309

Cited By

View all
  • (2025)Regulating learning module for patient monitoring interactive event detecting robotsExpert Systems with Applications10.1016/j.eswa.2024.125383260(125383)Online publication date: Jan-2025
  • (2024)Navigating the Future Through Adoption of AI in HealthcareAI Healthcare Applications and Security, Ethical, and Legal Considerations10.4018/979-8-3693-7452-8.ch015(240-260)Online publication date: 30-Jun-2024
  • (2024)Patent Applications as Glimpses into the Sociotechnical Imaginary: Ethical Speculation on the Imagined Futures of Emotion AI for Mental Health Monitoring and DetectionProceedings of the ACM on Human-Computer Interaction10.1145/36373838:CSCW1(1-43)Online publication date: 26-Apr-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
KDD '21: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining
August 2021
4259 pages
ISBN:9781450383325
DOI:10.1145/3447548
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 August 2021

Check for updates

Author Tags

  1. ai in healthcare
  2. explainable ai
  3. fairness in machine learning
  4. interpretable machine learning
  5. responsible ai
  6. xai

Qualifiers

  • Abstract

Conference

KDD '21
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)92
  • Downloads (Last 6 weeks)14
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Regulating learning module for patient monitoring interactive event detecting robotsExpert Systems with Applications10.1016/j.eswa.2024.125383260(125383)Online publication date: Jan-2025
  • (2024)Navigating the Future Through Adoption of AI in HealthcareAI Healthcare Applications and Security, Ethical, and Legal Considerations10.4018/979-8-3693-7452-8.ch015(240-260)Online publication date: 30-Jun-2024
  • (2024)Patent Applications as Glimpses into the Sociotechnical Imaginary: Ethical Speculation on the Imagined Futures of Emotion AI for Mental Health Monitoring and DetectionProceedings of the ACM on Human-Computer Interaction10.1145/36373838:CSCW1(1-43)Online publication date: 26-Apr-2024
  • (2024)Using artificial intelligence and predictive modelling to enable learning healthcare systems (LHS) for pandemic preparednessComputational and Structural Biotechnology Journal10.1016/j.csbj.2024.05.01424(412-419)Online publication date: Dec-2024
  • (2024)Exploring key stakeholders’ perspectives on integrating the EU AI Act with the MDR for certifying AI medical devicesAI and Ethics10.1007/s43681-024-00612-5Online publication date: 3-Dec-2024
  • (2024)Opportunities and challenges for the application of artificial intelligence paradigms into the management of endemic viral infections: The example of Chronic Hepatitis C VirusReviews in Medical Virology10.1002/rmv.251434:2Online publication date: 2-Feb-2024
  • (2023)Applications of Artificial Intelligence for Health Informatics: A Systematic ReviewJournal of Artificial Intelligence for Medical Sciences10.55578/joaims.230920.0014:2(19-46)Online publication date: 2023
  • (2023)Is medical device regulatory compliance growing as fast as extended reality to avoid misunderstandings in the future?Health and Technology10.1007/s12553-023-00775-x13:5(831-842)Online publication date: 2-Sep-2023
  • (2023)Initial clinical experience with a predictive clinical decision support tool for anatomic and reverse total shoulder arthroplastyEuropean Journal of Orthopaedic Surgery & Traumatology10.1007/s00590-023-03796-434:3(1307-1318)Online publication date: 14-Dec-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media