Skip to main content

Advertisement

Springer Nature Link
Log in

An App a Day will (Probably Not) Keep the Doctor Away: An Evidence Audit of Health and Medical Apps Available on the Apple App Store

  • Published:
Minds and Machines Aims and scope Submit manuscript

Abstract

There are more than 350,000 health apps available in public app stores. The extolled benefits of health apps are numerous and well documented. However, there are also concerns that poor-quality apps, marketed directly to consumers, threaten the tenets of evidence-based medicine and expose individuals to the risk of harm. This study addresses this issue by assessing the overall quality of evidence publicly available to support the effectiveness claims of health apps marketed directly to consumers. To assess the quality of evidence available to the public to support the effectiveness claims of health apps marketed directly to consumers, an audit was conducted of a purposive sample of apps available on the Apple App Store. We find the quality of evidence available to support the effectiveness claims of health apps marketed directly to consumers to be poor. Less than half of the 220 apps (44%) we audited state that they have evidence to support their claims of effectiveness and, of these allegedly evidence-based apps, more than 70% rely publicly on either very low or low-quality evidence. For the minority of app developers that do publish studies, significant methodological limitations are commonplace. Finally, there is a pronounced tendency for apps—particularly mental health and diagnostic apps—to either borrow evidence generated in other (typically offline) contexts or to rely exclusively on unsubstantiated, unpublished user metrics as evidence to support their effectiveness claims. Health apps represent a significant opportunity for individual consumers and healthcare systems. Nevertheless, this opportunity will be missed if the health apps market continues to be flooded by poor quality, poorly evidenced, and potentially unsafe apps. It must be accepted that a continuing lag in generating high-quality publicly available evidence of app effectiveness and safety is not inevitable: it is a choice. Just because it will be challenging to raise the quality of the evidence base publicly available to support the claims of health apps, this does not mean that the bar for evidence quality should be lowered. Innovation for innovation’s sake must not be prioritized over public health and safety.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. The Apple App Store was used because it provides more results per query (200) than the Google Play Store (30), allowing a larger, sufficient-sized sample.

  2. As Apple’s APIs do not allow the extraction of the data points required for our analysis, we used a third-party SerpAPI to scrape the store and collect the mentioned data points (see https://serpapi.com/).

  3. These search queries were selected for three reasons: (i) they align with consumer preferences for apps that provide information, improve treatment compliance, monitor physical parameters, and check symptoms (Paganini et al., 2023); (ii) they align with international policy priorities for digital health (Karpathakis et al., 2024); and (iii) they align with the generic definition of SaMD i.e., a device (including an app) that may be used for the purposes of diagnosis, prevention, monitoring, treatment, or alleviation of disease (Carroll and Richardson 2016). In each store native spelling was used.

  4. This is to assess the level of risk to users of the app. Apps that make explicit or specific claims about health outcomes, beyond wellness or lifestyle benefits, pose greater risk than those that vague allude to potential health benefits (Cohen et al., 2020).

  5. The intended use statement is important as it determines the usage and classification of an app and the regulation it should be compliant with (Carroll and Richardson 2016).

  6. An app with a CE mark or ‘Conformité Européenne’ is compliant with the current EU Medical Device Regulation, which came into effect in 2021 (Sadare et al., 2023). Now that the UK has left the European Union it is developing its own medical device regulation, and conformity will be demonstrated by the presence of a UKCA mark. However, as MDR CE mark certificates are valid for five years, apps that demonstrated compliance with the MDR in 2021 are unlikely to transition to being compliant with a UKCA mark until 2026.

  7. We used Google Scholar and Google Search rather than PubMed as the intention is to focus on evidence available to the general public who may be considering downloading the app and members of the public are more likely to consult Google than PubMed. This is also why we searched only for 15 min each time.

  8. An alphabetised list of all apps identified through the API searches is available online: https://docs.google.com/spreadsheets/d/1Y-skoarIyhalVc0v4J-S8ki735k19jfUVq4X5BmieOY/edit#gid=1873584707.

  9. A redacted database of all the included apps, and the results from the audit can be accessed here https://docs.google.com/spreadsheets/d/1JrHyU-iNTJyhVzGyH6QcC1Qyf6zLV24rdnDxJi3ZKwU/edit#gid=1873584707.

  10. This sums to 524 apps because, as stated, apps can have more than one classification.

  11. Pearson’s correlation was used in all instances.

  12. Studies by Lagan et al. (2021) and Lau et al. (2021) also found no correlation between app store rating and app effectiveness. Instead consumer ratings were found to be related to functionality and aesthetics as well as usability.

  13. A systematic review by Akbar et al. 2020 identified a total of 80 safety concerns associated with consumer-facing health apps.

  14. See for example: Buijink et al. (2013), Charani et al. (2014), Cowan et al. 2013, Kumar et al. (2013), McCartney (2013).

  15. 66 were available on the Google Play Store (Sadare et al., 2023) suggesting that Apple’s developer guidelines are more effective than Google’s but testing this was outside the scope of this specific audit.

  16. Identifying the precise requirements is challenging, requiring developers to refer to multiple standards including, but not limited to: IEC 62304, IEC 61508, IEC 62366:2007, IEC 6061-1-1-11:2010, ISO 14971;2007, ISO/IEC 15504-5:2012, IEC/TR80002-1:2009, ISO 14971;2012, IEC TR 80002-1:2009, IEEE 829.1998, ISO13485:2003, ISO 13485; 2003, ISO9001: 2008, BS EN ISO 14971:2012, BS EN 62304:2006, IEC 62366:2005 (Carroll and Richardson 2016; Gordon and Stern 2019; McCarthy and Lawford 2015).

  17. Exceptions include apps that are intended to directly influence biological functions or apps that claim to be capable of direct diagnosis. There are also nuances depending on the end user. For example, an app that provides a list of potential diagnoses, ordered by likelihood, for range of symptoms may be interpreted by a professional as a list of indicative diagnoses (making it a class I medical device), but interpreted by a lay consumer as a direct diagnosis (making it a class II medical device) (Guidance 2022).

  18. A class I device is not exempt from 510(k) notification requirements if it is intended for a use of substantial importance in preventing impairment of health, or presents a potential unreasonable risk of illness or injury. Similarly, exemptions may not apply if there is no comparable device on the market (Health. Class I and Class II Device Exemptions, 2023; Lee and Kesselheim 2018).

  19. Class I devices may be required to conduct primary clinical evaluation if there is insufficient existing data to adequately address the risk/benefit, efficacy, and safety profile of the device. This includes if the device involves new technology (European Commission, 2016).

  20. The National Institute for Clinical Excellence (NICE) Evidence Standards Framework for Digital Health Technologies is an exception. However, it is primarily intended to guide the development of apps destined to be deployed in clinical settings, not sold directly to consumers on the iOS app store, and when independently assessed it was also found to be highly ambiguous and difficult to apply in practice (Nwe et al., 2020).

  21. The NHS Apps Library was intended to signpost patients to “NHS approved apps” for specific conditions. Apps were listed on the library if they had been assessed by NHS staff using the Digital Assessment Questionnaire. The project failed, however, when it was found that several apps on the library were insecure from a data protection perspective and unevidenced (Armstrong 2016).

  22. ORCHA is a digital health company founded in 2015. It enables individual organisations to create bespoke apps libraries—stocked with apps that have been reviewed by ORCHA for safety, efficacy, and security—for their customers. The exact methodology used by ORCHA to review apps is proprietary and not available for inspection. (https://us.orchahealth.com/digital-health-products/health-app-library/).

References

  • Aboy, M., Crespo, C., & Stern, A. (2024). Beyond the 510(k): The regulation of novel moderate-risk medical devices, intellectual property considerations, and innovation incentives in the FDA’s De Novo pathway. NPJ Digital Medicine, 7, 29.

    Article  Google Scholar 

  • Akbar, S., Coiera, E., & Magrabi, F. (2020). Safety concerns with consumer-facing mobile health applications and their consequences: A scoping review. Journal of the American Medical Informatics Association, 27, 330–340.

    Article  Google Scholar 

  • Alqahtani, F., & Orji, R. (2020). Insights from user reviews to improve mental health apps. Health Informatics Journal, 26, 2042–2066.

    Article  Google Scholar 

  • Anthes, E. (2016). Pocket psychiatry: Mobile mental-health apps have exploded onto the market, but few have been thoroughly tested. Nature, 532.

  • Armstrong, S. (2015). Which app should I use? BMJ, h4597. https://doi.org/10.1136/bmj.h4597

  • Armstrong, S. (2016). What happens to data gathered by health and wellness apps? BMJ (Online), 353.

  • Babic, B., Gerke, S., Evgeniou, T., & Cohen, I. G. (2021). Direct-to-consumer medical machine learning and artificial intelligence applications. Nature Machine Intelligence, 3, 283–287.

    Article  Google Scholar 

  • Bates, D. W., Landman, A., & Levine, D. M. (2018). Health apps and health policy: What is needed? JAMA, 320, 1975–1976.

    Article  MATH  Google Scholar 

  • Batterham, P. J., Calear, A. L., O’Dea, B., Larsen, M. E., Kavanagh, D. J., Titov, N., March, S., Hickie, I., Teesson, M., Dear, B. F., Reynolds, J., Lowinger, J., Thornton, L., & Gorman, P. (2019). Stakeholder perspectives on evidence for digital mental health interventions: Implications for accreditation systems. DIGITAL HEALTH, 5, 205520761987806

  • Baxter, C., Carroll, J.-A., Keogh, B., & Vandelanotte, C. (2020). Assessment of mobile health apps using built-in smartphone sensors for diagnosis and treatment: Systematic survey of apps listed in international curated health app libraries. JMIR mHealth and uHealth, 8, e16741.

    Article  Google Scholar 

  • Buechi, R., Faes, L., Bachmann, L. M., Thiel, M. A., Bodmer, N. S., Schmid, M. K., Job, O., & Lienhard, K. R. (2017). Evidence assessing the diagnostic performance of medical smartphone apps: a systematic review and exploratory meta-analysis. British Medical Journal Open, 7, e018280.

    Google Scholar 

  • Buijink, A. W. G., Visser, B. J., & Marshall, L. (2013). Medical apps for smartphones: Lack of evidence undermines quality and safety. BMJ Evidence-Based Medicine, 18, 90–92.

    Article  Google Scholar 

  • Burns, P. B., Rohrich, R. J., & Chung, K. C. (2011). The levels of evidence and their role in evidence-based medicine. Plastic and Reconstructive Surgery, 128, 305–310.

    Article  MATH  Google Scholar 

  • Byambasuren, O., Sanders, S., Beller, E., & Glasziou, P. (2018). Prescribable mHealth apps identified from an overview of systematic reviews. NPJ Digital Medicine, 1, 12.

    Article  Google Scholar 

  • Cameron, B., Crawley, E., & Selva, D. (2016). Systems architecture, global edition. Pearson.

    MATH  Google Scholar 

  • Carroll, N., & Richardson, I. (2016). Software-as-a-medical device: Demystifying connected health regulations. JSIT, 18, 186–215.

    Article  Google Scholar 

  • Center for Devices and Radiological Health. (2023) How to determine if your product is a medical device. FDA.

  • Chan, A. H. Y., & Honey, M. L. L. (2022). User perceptions of mobile digital apps for mental health: Acceptability and usability—An integrative review. Journal of Psychiatric and Mental Health Nursing, 29, 147–168.

    Article  MATH  Google Scholar 

  • Chan, S., Torous, J., Hinton, L., & Yellowlees, P. (2015). Towards a framework for evaluating mobile mental health apps. Telemedicine and e-Health, 21, 1038–1041.

    Article  Google Scholar 

  • Chandrashekar, P. (2018). Do mental health mobile apps work: evidence and recommendations for designing high-efficacy mental health mobile apps. Mhealth, 4.

  • Charani, E., Castro-Sánchez, E., Moore, L. S., & Holmes, A. (2014). Do smartphone applications in healthcare require a governance and legal framework? It depends on the application! BMC Medicine, 12, 1–3.

    Article  Google Scholar 

  • Cohen, A. B., Mathews, S. C., Dorsey, E. R., Bates, D. W., & Safavi, K. (2020). Direct-to-consumer digital health. The Lancet Digital Health, 2, e163–e165.

    Article  Google Scholar 

  • Cowan, L. T., Van Wagenen, S. A., Brown, B. A., Hedin, R. J., Seino-Stephan, Y., Hall, P. C., & West, J. H. (2013). Apps of steel: are exercise apps providing consumers with realistic expectations? A content analysis of exercise apps for presence of behavior change theory. Health Education & Behavior, 40, 133–139.

    Article  Google Scholar 

  • Cristea, I. A., Cahan, E. M., & Ioannidis, J. P. A. (2019). Stealth research: Lack of peer-reviewed evidence from healthcare unicorns. Eur J Clin Investigation, 49, e13072.

    Article  Google Scholar 

  • Dawson, R. M., Felder, T. M., Donevant, S. B., McDonnell, K. K., Card, E. B., King, C. C., & Heiney, S. P. (2020). What makes a good health ‘app’? Identifying the strengths and limitations of existing mobile application evaluation tools. Nursing Inquiry, 27, e12333.

    Article  Google Scholar 

  • Day, S., Shah, V., Kaganoff, S., Powelson, S., & Mathews, S. C. (2022). Assessing the clinical robustness of digital health startups: Cross-sectional observational analysis. Journal of Medical Internet Research, 24, e37677.

    Article  Google Scholar 

  • Digital Therapeutics Allianc. (2022). DTx Regulatory & reimbursement pathways. Infographic. https://dtxalliance.org/wp-content/uploads/2022/01/US-Regulatory-and-Reimbursement-Pathways.pdf.

  • Essén, A., Stern, A.D., Haase, C.B., Car, J., Greaves, F., Paparova, D., Vandeput, S., Wehrens, R., & Bates, D. W. (2022). Health app policy: international comparison of nine countries’ approaches. npj Digital Medicine, 5.

  • European Commission. (2016). Clinical evaluation: a guide for manufacturers and notified bodies under directives 93/42 and 90/385.

  • Eysenbach, G. (2004). Tackling publication bias and selective reporting in health informatics research: Register your eHealth Trials in the International eHealth Studies Registry. Journal of Medical Internet Research, 6, e35.

    Article  Google Scholar 

  • FDA. (2019a). Step 3: Pathway to Approval. FDA.

  • FDA. (2019b) Step 4: FDA Device Review. FDA.

  • Floridi, L. (2008). The method of levels of abstraction. Minds and Machines, 18, 303–329.

    Article  MATH  Google Scholar 

  • Floridi, L. (2017). The logic of design as a conceptual logic of information. Minds & Machines, 27, 495–519.

    Article  MATH  Google Scholar 

  • Fraser, H., Coiera, E., & Wong, D. (2018). Safety of patient-facing digital symptom checkers. The Lancet, 392, 2263–2264.

    Article  MATH  Google Scholar 

  • FTC (2022). Mobile health app interactive tool. Federal Trade Commission https://www.ftc.gov/business-guidance/resources/mobile-health-apps-interactive-tool.

  • Gordon, W. J., Landman, A., Zhang, H., & Bates, D. W. (2020). Beyond validation: Getting health apps into clinical practice. Npj Digital Medicine, 3, 14.

    Article  Google Scholar 

  • Gordon, W. J., & Mandl, K. D. (2020). The 21st century cures act: A competitive apps market and the risk of innovation blocking. Journal of Medical Internet Research, 22, e24824.

    Article  Google Scholar 

  • Gordon, W. J., & Stern, A. D. (2019). Challenges and opportunities in software-driven medical devices. Nature Biomedical Engineering, 3, 493–497.

    Article  MATH  Google Scholar 

  • Greaves, F., Joshi, I., Campbell, M., Roberts, S., Patel, N., & Powell, J. (2018). What is an appropriate level of evidence for a digital health intervention? The Lancet, 392, 2665–2667.

    Article  Google Scholar 

  • Grundy, Q. H., Wang, Z., & Bero, L. A. (2016). Challenges in assessing mobile health app quality. American Journal of Preventive Medicine, 51, 1051–1059.

    Article  MATH  Google Scholar 

  • Guo, C., Ashrafian, H., Ghafur, S., Fontana, G., Gardner, C., & Prime, M. (2020). Challenges for the evaluation of digital health solutions—A call for innovative evidence generation approaches. NPJ Digital Medicine, 3, 110.

    Article  Google Scholar 

  • Health, C. for D. and R. Classify Your Medical Device. FDA https://www.fda.gov/medical-devices/overview-device-regulation/classify-your-medical-device (2023).

  • Health. Class I and Class II Device Exemptions. FDA (2023).

  • Ho, A., & Quick, O. (2018). Leaving patients to their own devices? Smart technology, safety and therapeutic relationships. BMC Medical Ethics, 19, 18.

    Article  MATH  Google Scholar 

  • Hoffman, L., Benedetto, E., Huang, H., Grossman, E., Kaluma, D., Mann, Z., & Torous, J. (2019). Augmenting mental health in primary care: A 1-year study of deploying smartphone apps in a multi-site primary care/behavioral health integration program. Frontiers in Psychiatry, 10, 94.

    Article  Google Scholar 

  • Holl, F., & Hrynyschyn, R. (2023). How to evaluate health and medical apps in public health—What are alternatives to randomized controlled trials? Population Medicine, 5.

  • Huckvale, K., Prieto, J. T., Tilney, M., Benghozi, P.-J., & Car, J. (2015). Unaddressed privacy risks in accredited health and wellness apps: A cross-sectional systematic assessment. BMC Medicine, 13, 214.

    Article  Google Scholar 

  • Huguet, A., Rao, S., McGrath, P. J., Wozney, L., Wheaton, M., Conrod, J., & Rozario, S. (2016). A systematic review of cognitive behavioral therapy and behavioral activation apps for depression. PLoS ONE, 11, e0154248.

    Article  Google Scholar 

  • IMDRF Software as a Medical Device (SaMD) Working Group. (2014). ‘Software as a medical device’: Possible framework for risk categorization and corresponding considerations. https://www.imdrf.org/sites/default/files/docs/imdrf/final/technical/imdrf-tech-140918-samd-framework-risk-categorization-141013.pdf.

  • Jacob, C., Lindeque, J., Klein, A., Ivory, C., Heuss, S., & Peter, M. K. (2023). Assessing the quality and impact of eHealth tools: Systematic literature review and narrative synthesis. JMIR Human Factors, 10, 4514.

    Article  Google Scholar 

  • Jake-Schoffman, D. E., Silfee, V. J., Waring, M. E., Boudreaux, E. D., Sadasivam, R. S., Mullen, S. P., Carey, J. L., Hayes, R. B., Ding, E. Y., Bennett, G. G., & Pagoto, S. L. (2017). Methods for evaluating the content, usability, and efficacy of commercial mobile health apps. JMIR mHealth and uHealth, 5, e190.

    Article  Google Scholar 

  • Kao, C.-K., & Liebovitz, D. M. (2017). Consumer mobile health apps: Current state, barriers, and future directions. PM&R, 9, S106–S115.

    Article  MATH  Google Scholar 

  • Karpathakis, K., Morley, J. & Floridi, L. (2024). A justifiable investment in AI for healthcare: aligning ambition with reality. Minds and Machines, 34(4), 38. https://doi.org/10.1007/s11023-024-09692-y.

    Article  MATH  Google Scholar 

  • Kasperbauer, T. J., & Wright, D. E. (2020). Expanded FDA regulation of health and wellness apps. Bioethics, 34, 235–241.

    Article  MATH  Google Scholar 

  • Kenny, R., Fitzgerald, A., Segurado, R., & Dooley, B. (2020). Is there an app for that? A cluster randomised controlled trial of a mobile app–based mental health intervention. Health Informatics Journal, 26, 1538–1559.

    Article  Google Scholar 

  • Kumar, A., Ross, J. S., Patel, N. A., Rathi, V., Redberg, R. F., & Dhruva, S. S. (2023). Studies of prescription digital therapeutics often lack rigor and inclusivity: Study examines prescription digital therapeutics standards. Health Affairs, 42, 1559–1567.

    Article  Google Scholar 

  • Kumar, S., Nilsen, W. J., Abernethy, A., Atienza, A., Patrick, K., Pavel, M., Riley, W. T., Shar, A., Spring, B., Spruijt-Metz, D., Hedeker, D., Honavar, V., Kravitz, R., Lefebvre, R. C., Mohr, D. C., Murphy, S. A., Quinn, C., Shusterman, V., & Swendeman, D. (2013). Mobile health technology evaluation. American Journal of Preventive Medicine, 45, 228–236.

    Article  Google Scholar 

  • Lagan, S., D’Mello, R., Vaidyam, A., Bilden, R., & Torous, J. (2021). Assessing mental health apps marketplaces with objective metrics from 29,190 data points from 278 apps. Acta Psychiatrica Scand, 144, 201–210.

    Article  Google Scholar 

  • Larsen, M. E., Huckvale, K., Nicholas, J., Torous, J., Birrell, L., Li, E., & Reda, B. (2019). Using science to sell apps: Evaluation of mental health app store quality claims. Npj Digital Medicine, 2, 18.

    Article  Google Scholar 

  • Larsen, M. E., Nicholas, J., & Christensen, H. (2016). Quantifying app store dynamics: Longitudinal tracking of mental health apps. JMIR mHealth and uHealth, 4, e6020.

    Article  Google Scholar 

  • Lau, N., et al. (2020). Android and iPhone mobile apps for psychosocial wellness and stress management: Systematic search in app stores and literature review. JMIR mHealth and uHealth, 8, e17798.

    Article  Google Scholar 

  • Lau, N., O’Daffer, A., Yi-Frazier, J. P., & Rosenberg, A. R. (2021). Popular evidence-based commercial mental health apps: Analysis of engagement, functionality, aesthetics, and information quality. JMIR mHealth and uHealth, 9, e29689.

    Article  Google Scholar 

  • Lee, T. T., & Kesselheim, A. S. U. S. (2018). Food and drug administration precertification pilot program for digital health software: Weighing the benefits and risks. Annals of Internal Medicine, 168, 730–732.

    Article  MATH  Google Scholar 

  • Leigh, S., & Ashall-Payne, L. (2019). The role of health-care providers in mHealth adoption. The Lancet Digital Health, 1, e58–e59.

    Article  MATH  Google Scholar 

  • Levine, D. M., Co, Z., Newmark, L. P., Groisser, A. R., Holmgren, A. J., Haas, J. S., & Bates, D. W. (2020). Design and testing of a mobile health application rating tool. NPJ Digital Medicine, 3, 74.

    Article  Google Scholar 

  • Levy, J., & Romo-Avilés, N. (2019). ‘A good little tool to get to know yourself a bit better’: A qualitative study on users’ experiences of app-supported menstrual tracking in Europe. BMC Public Health, 19.

  • Llorens-Vernet, P., & Miró, J. (2020). Standards for mobile health–related apps: Systematic review and development of a guide. JMIR mHealth and uHealth, 8, e13057.

    Article  Google Scholar 

  • Lupton, D., & Jutel, A. (2015). ‘It’s like having a physician in your pocket!’A critical analysis of self-diagnosis smartphone apps. Social Science & Medicine, 133, 128–135.

    Article  Google Scholar 

  • Magrabi, F., Habli, I., Sujan, M., Wong, D., Thimbleby, H., Baker, M., & Coiera, E. (2019). Why is it so difficult to govern mobile apps in healthcare? BMJ Health & Care Informatics, 26, e100006.

    Article  Google Scholar 

  • Mathews, S. C., McShea, M. J., Hanley, C. L., Ravitz, A., Labrique, A. B., & Cohen, A. B. (2019). Digital health: a path to validation. NPJ Digital Medicine, 2, 38.

    Article  Google Scholar 

  • May, L. C. (2023). Mind the gap(s) in mental health care. On the claimed and actual evidence-base of publicly available top-rated mHealth applications for depression: a scoping review.

  • McCarthy, A. D., & Lawford, P. V. (2015). Standalone medical device software: The evolving regulatory framework. Journal of Medical Engineering & Technology, 39, 441–447.

    Article  MATH  Google Scholar 

  • McCartney, M. (2013). How do we know whether medical apps work? BMJ, 346.

  • McKay, F. H., Cheng, C., Wright, A., Shill, J., Stephens, H., & Uccellini, M. (2018). Evaluating mobile phone applications for health behaviour change: A systematic review. Journal of Telemedicine and Telecare, 24, 22–30.

    Article  Google Scholar 

  • MHRA. (2022). Guidance: Medical device stand-alone software including apps (including IVDMDs). https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1105233/Medical_device_stand-alone_software_including_apps.pdf.

  • Milne-Ives, M., Lam, C., De Cock, C., Van Velthoven, M. H., & Meinert, E. (2020). Mobile apps for health behavior change in physical activity, diet, drug and alcohol use, and mental health: Systematic review. JMIR mHealth and uHealth, 8, e17046.

    Article  Google Scholar 

  • Mohr, D. C., Schueller, S. M., Riley, W. T., Brown, C. H., Cuijpers, P., Duan, N., Kwasny, M. J., Stiles-Shields, C., & Cheung, K. (2015). Trials of intervention principles: Evaluation methods for evolving behavioral intervention technologies. Journal of Medical Internet Research, 17, e166.

    Article  Google Scholar 

  • Moshi, M. R., Tooher, R., & Merlin, T. (2020). Development of a health technology assessment module for evaluating mobile medical applications. International Journal of Technology Assessment in Health Care, 36, 252–261.

    Article  MATH  Google Scholar 

  • Müller, R., Primc, N., & Kuhn, E. (2023). ‘You have to put a lot of trust in me’: Autonomy, trust, and trustworthiness in the context of mobile apps for mental health. Medicine, Health Care and Philosophy, 26, 313–324.

    Article  Google Scholar 

  • Nouri, R., Kalhori, S. R. N., Ghazisaeedi, M., Marchand, G., & Yasini, M. (2018). Criteria for assessing the quality of mHealth apps: a systematic review. Journal of the American Medical Informatics Association, 25, 1089–1098.

    Article  MATH  Google Scholar 

  • Nwe, K., Larsen, M. E., Nelissen, N., & Wong, D.C.-W. (2020). Medical mobile app classification using the national institute for health and care excellence evidence standards framework for digital health technologies: Interrater reliability study. Journal of Medical Internet Research, 22, e17457.

    Article  Google Scholar 

  • O’Rourke, T., Pryss, R., Schlee, W., & Probst, T. (2020). Development of a multidimensional app-quality assessment tool for health-related apps (AQUA). Digit Psychology, 1, 13–23.

    Article  Google Scholar 

  • Paganini, S., Meier, E., Terhorst, Y., Wurst, R., Hohberg, V., Schultchen, D., Strahler, J., Wursthorn, M., Baumeister, H., & Messner, E. M. (2023). Stress management apps: Systematic search and multidimensional assessment of quality and characteristics. JMIR mHealth and uHealth, 11, e42415.

    Article  Google Scholar 

  • Paripoorani, D., Gasteiger, N., Hawley-Hague, H., & Dowding, D. (2023). A systematic review of menopause apps with an emphasis on osteoporosis. BMC Women’s Health, 23, 518.

    Article  Google Scholar 

  • Parker, L., Halter, V., Karliychuk, T., & Grundy, Q. (2019). How private is your mental health app data? An empirical study of mental health app privacy policies and practices. International Journal of Law and Psychiatry, 64, 198–204.

    Article  Google Scholar 

  • Peiris, D., Miranda, J. J., & Mohr, D. C. (2018). Going beyond killer apps: Building a better mHealth evidence base. BMJ Global Health, 3, e000676.

    Article  Google Scholar 

  • Pham, Q., Wiljer, D., & Cafazzo, J. A. (2016). Beyond the randomized controlled trial: A review of alternatives in mHealth clinical trial methods. JMIR mHealth and uHealth, 4, e107.

    Article  Google Scholar 

  • Sadare, O., Melvin, T., Harvey, H., Vollebregt, E., & Gilbert, S. (2023). Can Apple and Google continue as health app gatekeepers as well as distributors and developers? NPJ Digital Medicine, 6, 8.

    Article  Google Scholar 

  • Safavi, K., Mathews, S. C., Bates, D. W., Dorsey, E. R., & Cohen, A. B. (2019). Top-funded digital health companies and their impact on high-burden. High-cost conditions. Health Affairs, 38, 115–123.

    Article  Google Scholar 

  • Schueller, S. M., Neary, M., O’Loughlin, K., & Adkins, E. C. (2018). Discovery of and interest in health apps among those with mental health needs: Survey and focus group study. Journal of Medical Internet Research, 20.

  • Scott, I. A., Scuffham, P., Gupta, D., Harch, T. M., Borchi, J., & Richards, B. (2020). Going digital: A narrative overview of the effects, quality and utility of mobile apps in chronic disease self-management. Australian Health Review, 44, 62.

    Article  Google Scholar 

  • Sedhom, R., McShea, M. J., Cohen, A. B., Webster, J. A., & Mathews, S. C. (2021). Mobile app validation: a digital health scorecard approach. npj Digital Medicine, 4.

  • Sherman, R. E., Anderson, S. A., Dal Pan, G. J., Gray, G. W., Gross, T., Hunter, N. L., LaVange, L., Marinac-Dabic, D., Marks, P. W., Robb, M. A., Shuren, J., Temple, R., Woodcock, J., Yue, L. Q., & Califf, R. M. (2016). Real-world evidence—What Is it and what can it tell us? New England Journal of Medicine, 375, 2293–2297.

    Article  Google Scholar 

  • Stoyanov, S. R., Hides, L., Kavanagh, D. J., Zelenko, O., Tjondronegoro, D., & Mani, M. (2015). Mobile app rating scale: A new tool for assessing the quality of health mobile apps. JMIR mHealth and uHealth, 3, e27.

    Article  Google Scholar 

  • The Medical Devices Regulations (2002). https://www.legislation.gov.uk/uksi/2002/618/regulation/2/made.

  • The Design Council & The Point People. (2020). System-shifting design: An emerging practice explored. https://www.designcouncil.org.uk/fileadmin/uploads/dc/Documents/Systemic%2520Design%2520Report.pdf.

  • Torous, J., Andersson, G., Bertagnoli, A., Christensen, H., Cuijpers, P., Firth, J., Haim, A., Hsin, H., Hollis, C., Lewis, S., Mohr, D. C., Pratap, A., Roux, S., Sherrill, J., & Arean, P. A. (2019). Towards a consensus around standards for smartphone apps and digital mental health. World Psychiatry, 18, 97–98.

    Article  Google Scholar 

  • Torous, J., & Roberts, L. W. (2017). Needed innovation in digital health and smartphone applications for mental health: Transparency and trust. JAMA Psychiatry, 74, 437.

    Article  Google Scholar 

  • van Velthoven, M., & Powell, J. (2017). Do health apps need endorsement? Challenges for giving advice about which health apps are safe and effective to use. Digital Health, 3, 2055207617701342.

    Article  Google Scholar 

  • Van Velthoven, M. H., Smith, J., Wells, G., & Brindley, D. (2018b). Digital health app development standards: A systematic review protocol. British Medical Journal Open, 8, e022969.

    Google Scholar 

  • Van Velthoven, M. H., Wyatt, J. C., Meinert, E., Brindley, D., & Wells, G. (2018a). How standards and user involvement can improve app quality: A lifecycle approach. International Journal of Medical Informatics, 118, 54–57.

    Article  Google Scholar 

  • Winograd, T., & Flores, F. (1986). Understanding computers and cognition: A new foundation for design. Ablex Pub. Corp.

    MATH  Google Scholar 

  • Wisniewski, H., Liu, G., Henson, P., Vaidyam, A., Hajratalli, N. K., Onnela, J.-P., & Torous, J. (2019). Understanding the quality, effectiveness and attributes of top-rated smartphone health apps. Evidence Based Mental Health, 22, 4–9.

    Article  Google Scholar 

  • Wykes, T., & Schueller, S. (2019). Why reviewing apps is not enough: Transparency for trust (T4T) principles of responsible health app marketplaces. Journal of Medical Internet Research, 21, e12390.

    Article  Google Scholar 

  • Zhou, T., Salman, D., & McGregor, A. (2024). mHealth Apps for the self-management of low back pain: Systematic search in app stores and content analysis. JMIR mHealth and uHealth, 12, e53262.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Digital Ethics Center, Yale University, 85 Trumbull Street, New Haven, CT, 06511, USA

    Jessica Morley, Joel Laitila & Luciano Floridi

  2. Section of General Medicine, Department of Medicine, Yale School of Medicine, 333 Cedar Street, New Haven, CT, 06519, USA

    Joseph S. Ross

  3. Faculty of Population Health Sciences, University College London, Gower Street, London, WC1E 6BT, UK

    Joel Schamroth

  4. London AI Centre, Guy’s and St. Thomas’ Hospital, London, SE1 7EH, UK

    Joe Zhang

  5. Institute of Global Health Innovation, Imperial College London, London, SW7 2AZ, UK

    Joe Zhang

  6. Alma Mater Studiorum - Università di Bologna, Via Zamboni, 33, 40126, Bologna, Italy

    Luciano Floridi

Authors
  1. Jessica Morley
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Joel Laitila
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Joseph S. Ross
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Joel Schamroth
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Joe Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Luciano Floridi
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Jessica Morley.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

See Tables

Table 6 Summary statistics—mental health apps which mention the use of CBT/DBT
Full size table

6,

Table 7 Summary statistics by query
Full size table

7,

Table 8 Summary statistics—overall, specific, and vague claims
Full size table

8,

Table 9 Summary statistics—by Apple App store categorisation
Full size table

9 and

Table 10 Apps and SaMD
Full size table

10.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Morley, J., Laitila, J., Ross, J.S. et al. An App a Day will (Probably Not) Keep the Doctor Away: An Evidence Audit of Health and Medical Apps Available on the Apple App Store. Minds & Machines 35, 11 (2025). https://doi.org/10.1007/s11023-025-09710-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11023-025-09710-7

Keywords