Skip to main content
SpringerLink
Log in

A Systematic Review and Taxonomy of Published Quality Criteria Related to the Evaluation of User-Facing eHealth Programs

  • Patient Facing Systems
  • Published:
Journal of Medical Systems Aims and scope Submit manuscript

Abstract

The purpose of this review was to identify and classify key criteria concepts related to the evaluation of user-facing eHealth programs. In line with the PRISMA statement methodology, computer searches of relevant databases were conducted for studies published between January 1, 2000 and March 1, 2016 that contained explicit quality criteria related to mHealth and eHealth products. Reference lists of included articles, review articles, and grey literature (e.g., books, websites) were searched for additional sources. A team of nine experts led by the first author was gathered to support the classification of these criteria. Identified criteria were extracted, grouped and organized using an inductive thematic analysis. Eighty-four sources – emanating from 26 different courtiers – were included in this review. The team extracted 454 criteria that were grouped into 11 quality domains, 58 criteria concepts and 134 concepts’ sub-groups. Quality domains were: Usability, Visual Design, User Engagement, Content, Behavior Change/Persuasive Design, Influence of Social Presence, Therapeutic Alliance, Classification, Credibility/Accountability, and Privacy/Security. Findings suggest that authors around the globe agree on key criteria concepts when evaluating user-facing eHealth products. The high proportion of new published criteria in the second half of this review time-frame (2008–2016), and more specifically, the high proportion of criteria relating to persuasive design, therapeutic alliance and privacy/security within this time-frame, points to the advancements made in recent years within this field.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Bennett, G.G., and Glasgow, R.E., The delivery of public health interventions via the internet: Actualizing their potential. Annu Rev Public Health. 30:273–292, 2009.

    Article  PubMed  Google Scholar 

  2. Gibbons, M.C., A historical overview of health disparities and the potential of eHealth solutions. J Med Internet Res. 7(5):e50, 2005. doi:10.2196/jmir.7.5.e50.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Aitken, M., and Gauntlett, C., Patient apps for improved healthcare from novelty to mainstream. IMS Institute for Healthcare Informatics, Parsippany, NJ, 2013.

    Google Scholar 

  4. Baumel, A., Making the case for a feasible evaluation method of available e-mental health products. Administration and Policy in Mental Health and Mental Health Services Research:1–4, 2016. doi:10.1007/s10488-016-0764-z.

  5. Powell, A.C., Landman, A.B., and Bates, D.W., In search of a few good apps. JAMA. 311(18):1851–1852, 2014.

    Article  CAS  PubMed  Google Scholar 

  6. Kumar, S., Nilsen, W.J., Abernethy, A., Atienza, A., Patrick, K., Pavel, M., Riley, W.T., Shar, A., Spring, B., Spruijt-Metz, D., Hedeker, D., Honavar, V., Kravitz, R., Lefebvre, R.C., Mohr, D.C., Murphy, S.A., Quinn, C., Shusterman, V., and Swendeman, D., Mobile health technology evaluation: The mHealth evidence workshop. Am J Prev Med. 45(2):228–236, 2013. doi:10.1016/j.amepre.2013.03.017.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Kim, P., Eng, T.R., Deering, M.J., and Maxfield, A., Published criteria for evaluating health related web sites: Review. BMJ. 318:647–649, 1999.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Olsina, L., and Rossi, G., Measuring web application quality with WebQEM. IEEE Multimedia. 9(4):20–29, 2002.

    Article  Google Scholar 

  9. Provost, M., Koompalum, D., Dong, D., and Martin, B.C., The initial development of the WebMedQual scale: Domain assessment of the construct of quality of health web sites. Int J Med Inform. 75(1):42–57, 2006.

    Article  PubMed  Google Scholar 

  10. Charnock, D., Shepperd, S., Needham, G., and Gann, R., DISCERN: An instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health. 53(2):105–111, 1999.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  11. Nahum-Shani I, Smith SN, Tewari A, Witkiewitz K, Collins LM, Spring B, Murphy S (2014) Just in time adaptive interventions (jitais): An organizing framework for ongoing health behavior support. (technical report no. 14-126). The methodology center, Penn State.

  12. Muench, F., and Baumel, A., More than a text message: Dismantling digital triggers to curate behavior change in patient centered health interventions. J Med Internet Res., in press. doi:10.2196/jmir.7463.

  13. Kelders, S.M., Kok, R.N., Ossebaard, H.C., and Van Gemert-Pijnen, J.E., Persuasive system design does matter: A systematic review of adherence to web-based interventions. J Med Internet Res. 14(6):e152, 2012.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Webb, T., Joseph, J., Yardley, L., and Michie, S., Using the internet to promote health behavior change: A systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res. 12(1):e4, 2010.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Baumel, A., Faber, K., Mathur, N., Kane, J.M., and Muench, F., Enlight: A comprehensive quality and therapeutic potential evaluation tool for mobile and web-based eHealth interventions. J Med Internet Res. 19(3):e82, 2017. doi:10.2196/jmir.7270.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Liberati, A., Altman, D.G., Tetzlaff, J., Mulrow, C., Gøtzsche, P.C., Ioannidis, J.P., Clarke, M., Devereaux, P., Kleijnen, J., and Moher, D., The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. Ann Intern Med. 151(4):W-65–W-94, 2009.

    Article  Google Scholar 

  17. Baumel, A., and Muench, F., Heuristic evaluation of Ehealth interventions: Establishing standards that relate to the therapeutic process perspective. JMIR Mental Health. 3(1):e5, 2016. doi:10.2196/mental.4563.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Braun, V., and Clarke, V., Using thematic analysis in psychology. Qual Res Psychol. 3(2):77–101, 2006.

    Article  Google Scholar 

  19. FDA clearance - BioGaming's physiotherapy software. http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfpmn/pmn.cfm?ID=K151955. Accessed February 10, 2017. Archived by WebCite® at http://www.webcitation.org/6oBIa65Hf.

  20. "At Last -- The Full Story Of How Facebook Was Founded". Carlson, N. (March 5 2010). Business Insider. Axel Springer SE. Retrieved March 23, 2017.

  21. Wilson TV, Fenlon W (2007) How the iPhone works. Retrieved from: on Apr 24 (2009):9

  22. Horvath, A.O., and Luborsky, L., The role of the therapeutic alliance in psychotherapy. J Consult Clin Psychol. 61(4):561, 1993.

    Article  CAS  PubMed  Google Scholar 

  23. Armontrout, J., Torous, J., Fisher, M., Drogin, E., and Gutheil, T., Mobile mental health: Navigating new rules and regulations for digital tools. Current psychiatry reports. 18(10):91, 2016. doi:10.1007/s11920-016-0726-x.

    Article  PubMed  Google Scholar 

  24. Higgins J, Green S, Scholten RJPM (2011) Chapter 3: Maintaining reviews: updates, amendments and feedback. In: Higgins J, Green S (eds) Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.

Download references

Author information

Authors and Affiliations

  1. The Zucker Hillside Hospital, Psychiatry Research, Northwell Health, New York, NY, USA

    Amit Baumel & Michael L. Birnbaum

  2. Hofstra Northwell School of Medicine, Hempstead, NY, USA

    Amit Baumel & Michael L. Birnbaum

  3. Department of Community Mental Health, University of Haifa, Haifa, Israel

    Amit Baumel

  4. Department of Oncological Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, USA

    Madalina Sucala

  5. Department of Clinical Psychology and Psychotherapy, Babes-Bolyai University, 400084, Cluj-Napoca, Romania

    Madalina Sucala

Authors
  1. Amit Baumel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael L. Birnbaum
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Madalina Sucala
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amit Baumel.

Ethics declarations

Conflicts of Interest

All authors report no potential conflicts of interest to disclose.

Human and Animal Rights

This research did not involve human participants and/or animals.

Additional information

This article is part of the Topical Collection on Patient Facing Systems.

Electronic supplementary material

ESM 1

(DOCX 70 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Baumel, A., Birnbaum, M.L. & Sucala, M. A Systematic Review and Taxonomy of Published Quality Criteria Related to the Evaluation of User-Facing eHealth Programs. J Med Syst 41, 128 (2017). https://doi.org/10.1007/s10916-017-0776-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10916-017-0776-6

Keywords

Search

Navigation