skip to main content
10.1145/3287560.3287593acmconferencesArticle/Chapter ViewAbstractPublication PagesfacctConference Proceedingsconference-collections
extended-abstract

Dissecting Racial Bias in an Algorithm that Guides Health Decisions for 70 Million People

Published: 29 January 2019 Publication History

Abstract

A single algorithm drives an important health care decision for over 70 million people in the US. When health systems anticipate that a patient will have especially complex and intensive future health care needs, she is enrolled in a 'care management' program, which provides considerable additional resources: greater attention from trained providers and help with coordination of her care.
To determine which patients will have complex future health care needs, and thus benefit from program enrollment, many systems rely on an algorithmically generated commercial risk score. In this paper, we exploit a rich dataset to study racial bias in a commercial algorithm that is deployed nationwide today in many of the US's most prominent Accountable Care Organizations (ACOs).
We document significant racial bias in this widely used algorithm, using data on primary care patients at a large hospital. Blacks and whites with the same algorithmic risk scores have very different realized health. For example, the highest-risk black patients (those at the threshold where patients are auto-enrolled in the program), have significantly more chronic illnesses than white enrollees with the same risk score. We use detailed physiological data to show the pervasiveness of the bias: across a range of biomarkers, from HbA1c levels for diabetics to blood pressure control for hypertensives, we find significant racial health gaps conditional on risk score. This bias has significant material consequences for patients: it effectively means that white patients with the same health as black patients are far more likely be enrolled in the care management program, and benefit from its resources. If we simulated a world without this gap in predictions, blacks would be auto-enrolled into the program at more than double the current rate.
An unusual aspect of our dataset is that we observe not just the risk scores but also the input data and objective function used to construct it. This provides a unique window into the mechanisms by which bias arises. The algorithm is given a data frame with (1) Yit (label), total medical expenditures ('costs') in year t; and (2) Xi,t--1 (features), fine-grained care utilization data in year t -- 1 (e.g., visits to cardiologists, number of x-rays, etc.). The algorithm's predicted risk of developing complex health needs is thus in fact predicted costs. And by this metric, one could easily call the algorithm unbiased: costs are very similar for black and white patients with the same risk scores. So far, this is inconsistent with algorithmic bias: conditional on risk score, predictions do not favor whites or blacks.
The fundamental problem we uncover is that when thinking about 'health care needs,' hospitals and insurers focus on costs. They use an algorithm whose specific objective is cost prediction, and from this perspective, predictions are accurate and unbiased. Yet from the social perspective, actual health -- not just costs -- also matters. This is where the problem arises: costs are not the same as health. While costs are a reasonable proxy for health (the sick do cost more, on average), they are an imperfect one: factors other than health can drive cost -- for example, race.
We find that blacks cost more than whites on average; but this gap can be decomposed into two countervailing effects. First, blacks bear a different and larger burden of disease, making them costlier. But this difference in illness is offset by a second factor: blacks cost less, holding constant their exact chronic conditions, a force that dramatically reduces the overall cost gap. Perversely, the fact that blacks cost less than whites conditional on health means an algorithm that predicts costs accurately across racial groups will necessarily also generate biased predictions on health.
The root cause of this bias is not in the procedure for prediction, or the underlying data, but the algorithm's objective function itself. This bias is akin to, but distinct from, 'mis-measured labels': it arises here from the choice of labels, not their measurement, which is in turn a consequence of the differing objective functions of private actors in the health sector and society. From the private perspective, the variable they focus on -- cost -- is being appropriately optimized. But our results hint at how algorithms may amplify a fundamental problem in health care as a whole: externalities produced when health care providers focus too narrowly on financial motives, optimizing on costs to the detriment of health. In this sense, our results suggest that a pervasive problem in health care -- incentives that induce health systems to focus on dollars rather than health -- also has consequences for the way algorithms are built and monitored.

Cited By

View all
  • (2025)Ethical Considerations in the Use of AI and Big Data in Corporate Decision-MakingDigital Citizenship and the Future of AI Engagement, Ethics, and Privacy10.4018/979-8-3693-9015-3.ch017(467-532)Online publication date: 21-Feb-2025
  • (2024)A Comprehensive Review of AI Techniques for Addressing Algorithmic Bias in Job HiringAI10.3390/ai50100195:1(383-404)Online publication date: 7-Feb-2024
  • (2024)NLP for Maternal Healthcare: Perspectives and Guiding Principles in the Age of LLMsProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658982(1446-1463)Online publication date: 3-Jun-2024
  • Show More Cited By

Index Terms

  1. Dissecting Racial Bias in an Algorithm that Guides Health Decisions for 70 Million People

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    FAT* '19: Proceedings of the Conference on Fairness, Accountability, and Transparency
    January 2019
    388 pages
    ISBN:9781450361255
    DOI:10.1145/3287560
    Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 29 January 2019

    Check for updates

    Author Tags

    1. algorithms
    2. bias
    3. health policy
    4. medicine
    5. racial disparities

    Qualifiers

    • Extended-abstract
    • Research
    • Refereed limited

    Funding Sources

    • National Institute for Health Care Management

    Conference

    FAT* '19
    Sponsor:

    Upcoming Conference

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)169
    • Downloads (Last 6 weeks)13
    Reflects downloads up to 22 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Ethical Considerations in the Use of AI and Big Data in Corporate Decision-MakingDigital Citizenship and the Future of AI Engagement, Ethics, and Privacy10.4018/979-8-3693-9015-3.ch017(467-532)Online publication date: 21-Feb-2025
    • (2024)A Comprehensive Review of AI Techniques for Addressing Algorithmic Bias in Job HiringAI10.3390/ai50100195:1(383-404)Online publication date: 7-Feb-2024
    • (2024)NLP for Maternal Healthcare: Perspectives and Guiding Principles in the Age of LLMsProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658982(1446-1463)Online publication date: 3-Jun-2024
    • (2024)The Dark Side of Dataset Scaling: Evaluating Racial Classification in Multimodal ModelsProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658968(1229-1244)Online publication date: 3-Jun-2024
    • (2024)More than the Sum of its Parts: Susceptibility to Algorithmic Disadvantage as a Conceptual FrameworkProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658944(909-919)Online publication date: 3-Jun-2024
    • (2024)In the Walled Garden: Challenges and Opportunities for Research on the Practices of the AI Tech IndustryProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658918(456-466)Online publication date: 3-Jun-2024
    • (2024)Ethnic Classifications in Algorithmic Fairness: Concepts, Measures and Implications in PracticeProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658902(237-253)Online publication date: 3-Jun-2024
    • (2024)AI auditing: The Broken Bus on the Road to AI Accountability2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML)10.1109/SaTML59370.2024.00037(612-643)Online publication date: 9-Apr-2024
    • (2024)Mitigating Racial Algorithmic Bias in Healthcare Artificial Intelligent Systems: A Fairness-Aware Machine Learning Approach2024 5th International Conference on Smart Sensors and Application (ICSSA)10.1109/ICSSA62312.2024.10788666(1-6)Online publication date: 10-Sep-2024
    • (2024)Modeling Gender Bias in Eastern and Western Artificial Intelligence from a Cross-Cultural Perspective2024 4th International Conference on Educational Technology (ICET)10.1109/ICET62460.2024.10868787(253-257)Online publication date: 13-Sep-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media