Quantity and type of peer-reviewed evidence for popular free medical apps: Cross-sectional review

https://doi.org/10.1016/j.ijmedinf.2021.104416Get rights and content

Highlights

  • Medical Apps are not commonly supported by academic peer-reviewed evidence.

  • Apps that entail greater clinical risk are more likely to have associated evidence.

  • Expert-curated apps are not associated with greater levels of associated evidence.

Abstract

Introduction

– Mobile apps are being increasingly used as a tool to deliver clinical care. Evidence of efficacy for such apps varies, and appropriate levels of evidence may depend on the app’s intended use. The UK’s National Institute for Health and Care Excellence (NICE) recently developed an evidence standards framework, aiming to explicitly set out the required standards of evidence for different categories of digital health technologies. To determine current compliance with the evidence standards framework, the current study quantified the amount and type of peer-reviewed evidence associated with a cross-section of popular medical apps.

Methods

– Apps were identified by selecting the top 100 free medical apps in the Apple App Store and all free apps in the NHS Apps Library. Each app was assigned to one of the four tiers (1, 2, 3a, 3b) in the NICE evidence standards framework. For each app, we conducted searches in Ovid-MEDLINE, Web of Science, Google Scholar, and via manufacturer websites to identify any published articles that assessed the app. This allowed us to determine our primary outcome, whether apps in tiers 3a/3b were more likely than apps in tier 1/2 to be associated with academic peer-reviewed evidence.

Results

– We reviewed 125 apps in total (Apple App Store (n = 72), NHS Apps Library (n = 45), both (n = 8), of which 54 were categorized into the higher evidence standards framework tiers, 3a/3b. After screening, we extracted 105 relevant articles which were associated with 25 of the apps. Only 6 articles, pertaining to 3 apps, were reports of randomised controlled trials. Apps in tiers 3a/3b were more likely to be associated with articles than apps in lower tiers (χ2 = 5.54, p = .01). The percentage of tier 3a/3b apps with associated articles was similar for both the NHS Apps Library (10/28) and Apple App store (7/24), (χ2 = 0.042, p = .84).

Discussion

– Apps that were in higher tiers 3a and 3b, indicating higher clinical risk, were more likely to have an associated article than those in lower categories. However, even in these tiers, supporting peer-reviewed evidence was missing in the majority of instances. In our sample, Apps from the NHS Apps Library were more no more likely to have supporting evidence than popular Apple App Store apps. This is of concern, given that NHS approval may influence uptake of app usage.

Introduction

The versatility of mobile healthcare (mHealth) apps has meant that, historically, it has been challenging to establish confidence in their clinical quality [1]. More widely, the popularity of healthcare technology is not always associated with strong evidence of the technology’s effectiveness. For instance, Cristea et al. recently showed how the highest-valued healthtech start-ups had little supporting academic evidence [2].

Whilst there is robust evidence of effectiveness for some specific mHealth interventions, evidence for the efficacy of mHealth more generally remains limited [3]. One complementary problem is that there is no standard acceptable level of evidence for an mhealth app and indeed, different apps may require different evidence [4]. Some have previously suggested that evidence required to demonstrate safety and effectiveness ought to be proportional to an app’s clinical risk and technical complexity [5].

The UK’s National Institute for Health and Care Excellence (NICE) recently developed an evidence standards framework, aiming to explicitly set out the required standards of evidence for different categories of digital health technologies (DHTs) [6]. Whilst the framework is designed primarily for DHTs commissioned in the UK health and care system (where professional duty of care is clear) [7], the recommended standards may also be appropriate for direct-to-user applications. The framework has previously been used in this context (e.g by Clarke et al. [8]).

Given the new framework, we sought to assess whether a wide selection of popular mHealth apps met the corresponding clinical evidence criteria. Specifically, technologies assigned to the highest tier within the framework (tier 3a or 3b, apps to prevent, detect, manage or treat one or more specific conditions) should, at a minimum, demonstrate effectiveness via ‘high quality observational or quasi-experimental studies demonstrating relevant outcomes’ [6]. Technologies in this tier are defined as those that are used to ‘prevent and manage disease’ (3a), or those with ‘measurable user benefits, including tools used for treatment and diagnosis, as well as influencing clinical management through active monitoring or calculation’ (3b). We reviewed this by assessing the quantity and type of peer-reviewed evidence associated with the identified apps.

Section snippets

Methods

This study was a cross-sectional review of a sample of mHealth apps across all health domains.

App search and classification

We identified 170 apps in the Apple App Store (n = 100) and NHS Apps Library (n = 70). Following screening, 125 apps (n = 72 from Apple App Store n = 45 from NHS Apps Library, n = 8 in both) were retained (Fig. 1).

Of the 20 apps excluded as being irrelevant, 15 were digital medical textbooks or educational resources. Of the other 5 apps, 2 were Bluetooth controllers for peripheral medical devices, 1 reported reasons for a student’s absence from school, 1 located cannabis retailers, and 1 was a

Discussion

The NICE evidence standards framework sets the minimum standard of evidence for tier 3a and 3b digital health technologies to demonstrate effectiveness via ‘high quality observational or quasi-experimental studies’ (interventional for tier 3b) that show improvements in relevant outcomes.

In our cross-sectional review of free popular and NHS-recommended apps, 31% (17/54) of tier 3a/b apps were associated with peer-reviewed evidence. Although we did not assess study quality, this figure is an

Conclusion

This cross-sectional review of supporting evidence for free apps showed that most apps did not have sufficient evidence to meet the minimum criteria within the NICE Evidence Standards Framework. Furthermore, we found no indication that apps curated within the NHS Apps Library had any more evidence than popular apps within the Apple App Store.

The introduction of the evidence standards framework provides a benchmark that can help to improve the quality of digital health technologies by

Author’s contributions

All authors contributed to the design, analysis, and final manuscript writing. KN undertook the initial app and literature searches. DW, KN, ML, NN undertook the app data extraction. DW, NN, RE undertook the literature data extraction.

Summary Table

What was already known on the topic:

  • -

    evidence for the efficacy of medical mobile apps has historically been very limited.

  • -

    the UK’s National Institute for Health and Care Excellence (NICE) have developed a new framework to assess the evidence required

Transparency document

.

Declaration of Competing Interest

DW receives royalties from Sensyne Health. Sensyne Health license the GDm-Health app that was included in this review.

Acknowledgements

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. KN was supported by a Chevening Scholarship.

References (24)

  • F. Greaves et al.

    What is an appropriate level of evidence for a digital health intervention?

    Lancet

    (2018)
  • L.C. Abroms et al.

    A content analysis of popular smartphone apps for smoking cessation

    Am. J. Prev. Med.

    (2013)
  • F. Magrabi et al.

    Why is it so difficult to govern mobile apps in healthcare?

    BMJ Health Care Inform.

    (2019)
  • I.A. Cristea et al.

    Stealth research: lack of peer-reviewed evidence from healthcare unicorns

    Eur. J. Clin. Invest.

    (2019)
  • M.S. Marcolino et al.

    The impact of mHealth interventions: systematic review of systematic reviews

    JMIR Mhealth Uhealth

    (2018)
  • T.L. Lewis et al.

    mHealth and mobile medical Apps: a framework to assess risk and promote safer use

    J. Med. Internet Res.

    (2014)
  • Evidence Standards Framework for Digital Health Technologies

    (2019)
  • C. Burr et al.

    Empowerment or engagement? Digital healthtechnologies for mentalhealthcare

  • S. Clarke et al.

    A systematic review and meta-analysis of digital health technologies effects on psychotic symptoms in adults with psychosis

    Psychosis.

    (2019)
  • NHS Digital

    NHS Apps Library

    (2019)
  • NHS Digital

    Digital Assessment Questionnaire V2.2

    (2021)
  • Apple Inc

    App Store

    (2020)
  • Cited by (4)

    View full text