Skip to main content
Journal of the American Medical Informatics Association : JAMIA logoLink to Journal of the American Medical Informatics Association : JAMIA
letter
. 2021 Sep 8;28(11):2543–2544. doi: 10.1093/jamia/ocab177

Response to: Looking for clinician involvement under the wrong lamp post: the need for collaboration measures

Jessica M Schwartz 1,, Amanda J Moy 2, Sarah C Rossetti 1,2, Noémie Elhadad 2, Kenrick D Cato 1,3
PMCID: PMC8510279  PMID: 34498082

Dear JAMIA Editors and Readers:

We appreciate the critiques that Dr. Sendak and colleagues have brought forward regarding our scoping review of clinician involvement in predictive CDSS design.1 In their letter, Sendak and colleagues argue that our review too narrowly defined clinician involvement and that relationships established between clinician leaders, often coauthors on manuscripts, and other research team members is a valuable form of clinician involvement not adequately captured in our review.2 We recognize and agree that we should have more prominently highlighted the possibility that clinically affiliated coauthors’ contributions may have represented clinician involvement in one of the ways we charted or in a different relationship-oriented way that is also important for predictive CDSS success. We also should have consistently referred to our results finding that involvement is not widely reported instead of not widely practiced.

We also acknowledge that reaching out to authors to gather information about clinician involvement, as Sendak et al propose, would have further strengthened our review. Yet, we argue that explicitly stating clinician involvement in each manuscript is key to scientific rigor. This goes the same for inferring that clinically affiliated coauthors were involved as clinician experts. Activities and tasks completed as part of, and to inform, the study should be stated for readers to appraise and replicate, rather than relying on assumptions.

We would also like to respond to a mischaracterization of our review. The authors state that 3 of their research publications3–5 were excluded from our review, implying this is a result of our criteria. However, the 3 publications were published in 2020, months after our search concluded in October 2019, which we stated in our manuscript. They could not possibly have been included given the timing of our search, therefore the authors’ implication that our eligibility criteria were too narrow to include their publications is a mischaracterization. We suggested that another review be conducted in the future to illuminate progression on the topic. If done, that review would certainly include Sendak et al’s 3 articles published in 2020.3–5

Furthermore, while we provided an example of wording used that, to our team, indicated clinicians were involved, this is simply an example and not intended to imply restrictive language. As a scoping review is intended for studying a topic with potentially wide-ranging characteristics,6,7 we intentionally took a broad approach to defining clinician involvement. After reading Sendak et al’s 3 2020 publications,3–5 these certainly would have been categorized as involving clinicians.

We wholeheartedly agree with Sendak et al, that standardized measures and reporting of clinician involvement should be developed and practiced and that those should include relationships established between clinical experts, machine learning experts, and integration leaders. Yet, we argue that such relationships cannot replace the expertise provided by currently practicing end-user clinicians. For example, clinical coauthors may be experts in their field and establish valuable relationships between clinical and data science teams, yet if 100% of their time is dedicated to research and/or leadership, they cannot provide the same expertise that a practicing end-user clinician can provide. Our understanding is that this point of view is in fact consistent with a recent perspective piece on which Dr. Sendak is a coauthor8: interdisciplinary teams best consist of stakeholders from 3 categories—knowledge experts, decision-makers, and users; and it is clear that Sepsis Watch was developed with stakeholders from each category.3 In the absence of explicit language about clinician involvement, it may be ambiguous to most readers whether clinically affiliated coauthors functioned as knowledge experts, decision-makers, or users.

We appreciate the enthusiasm for this topic and are hopeful that discussions such as these will further awareness of the importance of clinician involvement and its practice, as well as the development of consistent reporting criteria, as predictive CDSS research and design proliferates.

FUNDING

Amanda Moy is supported by the National Library of Medicine grant 5T15LM007079. Jessica Schwartz was supported by the National Institute for Nursing Research (NINR) training grant 5T32NR007969 during the writing and publication of the scoping review of interest.

AUTHOR CONTRIBUTIONS

JMS and KDC conceptualized the correspondence. JMS wrote the initial draft. SCR, NE, AM, and KDC provided feedback and revisions.

DATA AVAILABILITY STATEMENT

No new data were generated or analyzed for this correspondence.

CONFLICT OF INTEREST STATEMENT

None declared.

References

  • 1.Schwartz JM, Moy AJ, Rossetti SC, Elhadad N, Cato KD.. Clinician involvement in research on machine learning-based predictive clinical decision support for the hospital setting: a scoping review. J Am Med Inform Assoc 2021; 28 (3): 653–63. doi:10.1093/jamia/ocaa296 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Sendak MP, Gao M, Ratliff W, et al. Looking for clinician involvement under the wrong lamp post: the need for collaboration measures. J Am Med Inform Assoc 2021; doi: 10.1093/jamia/ocab129. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Sendak MP, Ratliff W, Sarro D, et al. Real-world integration of a sepsis deep learning technology into routine clinical care: implementation study. JMIR Med Inform 2020; 8 (7): e15182. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Sendak M, Futoma J, Bedoya A, et al. “The Human Body is a Black Box”: supporting clinical decision-making with deep learning. In: Proceedings of ACM Conference on Fairness, Accountability, and Transparency (FAT 2020); Barcelona, Spain. 2020. doi:10.1145/3351095.3372827.
  • 5.Elish MC, Ae W.. Repairing Innovation: A Study of Integrating AI in Clinical Care. Data & Society Research Institute; 2020. https://datasociety.net/pubs/repairing-innovation.pdf. Accessed July 21, 2021 [Google Scholar]
  • 6.Arksey H, O'Malley L.. Scoping studies: towards a methodological framework. Int J Soc Res Methodol Theory Pract 2005; 8 (1): 19–32. doi:10.1080/1364557032000119616 [Google Scholar]
  • 7.Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med 2018; 169 (7): 467–73. [DOI] [PubMed] [Google Scholar]
  • 8.Wiens J, Saria S, Sendak M, et al. Do no harm: a roadmap for responsible machine learning for health care. Nat Med 2019; 25 (9): 1337–40. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Availability Statement

No new data were generated or analyzed for this correspondence.


Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of Oxford University Press

RESOURCES