Skip to main content

Advertisement

SpringerLink
Log in

Methods to Evaluate Health information Systems in Healthcare Settings: A Literature Review

  • Published:
Journal of Medical Systems Aims and scope Submit manuscript

Abstract

Although information technology (IT)-based applications in healthcare have existed for more than three decades, methods to evaluate outputs and outcomes of the use of IT-based systems in medical informatics is still a challenge for decision makers, as well as to those who want to measure the effects of ICT in healthcare settings. The aim of this paper is to review published articles in the area evaluations of IT-based systems in order to gain knowledge about methodologies used and findings obtained from the evaluation of IT-based systems applied in healthcare settings. The literature review includes studies of IT-based systems between 2003 and 2005. The findings show that economic and organizational aspects dominate evaluation studies in this area. However, the results focus mostly on positive outputs such as user satisfaction, financial benefits and improved organizational work. This review shows that there is no standard framework for evaluation effects and outputs of implementation and use of IT in the healthcare setting and that until today no studies explore the impact of IT on the healthcare system’ productivity and effectiveness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Kaplan, B., Evaluating informatics applications—some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int. J. Med. Inform. 64:39–56, 2001.

    Article  Google Scholar 

  2. Ammenwerth, E., Brender, J., Nykanen, P., Prokosch, H. U., Rigby, M., and Talmon, J., Visions and strategies to improve evaluation of health information systems. Reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int. J. Med. Inform. 73:479–491, 2004.

    Article  Google Scholar 

  3. Ammenwerth, E., Iller, C., and Mansmann, U., Can evaluation studies benefit from triangulation? A case study. Int. J. Med. Inform. 70:237–248, 2003.

    Article  Google Scholar 

  4. Andersen, J. G., Aydin, C. E., and Jay, S. J., Evaluating Healthcare Information Systems: Methods and Applications. Thousand Oaks, CA: Sage publication, 1994.

    Google Scholar 

  5. Friedman, C. P., and Wyatt, J. C., Evaluation Methods in Medical Informatics. New York: Springer, 1996.

    Google Scholar 

  6. Kaplan, B., and Shaw, N. T., Future directions in evaluation research: People, organizational, and social issues. Methods Inf. Med. 43:215–231, 2004.

    Google Scholar 

  7. Phil, W., An assessment of strategies for information system evaluation: Lessons for education. Int. J. Educ. Res. 25:361–371, 1996.

    Article  Google Scholar 

  8. Delpierre, C., Cuzin, L., Fillaux, J., Alvarez, M., Massip, P., and Lang, T., A systematic review of computer-based patient record systems and quality of care: More randomized clinical trials or a broader approach. Int. J. Qual. Health Care. 16:407–416, 2004.

    Article  Google Scholar 

  9. Wang, S. J., Middleton, B., Bardon, C. G., Spurr, C. D., Carchidi, P. J., Kittler, A. f., Goldszar, R. C., Fairchild, D. G., Sussman, A. J., Kuperman, G. J., and Bates, D. W., A cost-benefit analysis of electronic medical records in primary care. Am. J. Med. 114:397–403, 2003.

    Article  Google Scholar 

  10. Laerum, H., Karlsen, T. H., and Faxvaag, A., Effects of scanning and eliminating paper-based medical records on hospital physicians’ clinical work practice. J. Am. Med. Inform. Assoc. 10:160–588, 2003.

    Article  Google Scholar 

  11. Ruland, C. M., and Ravn, I. H., Usefulness and effects on costs and staff management of a nursing resource management information system. J. Nurs. Manag. 11:208–215, 2003.

    Google Scholar 

  12. Houston, T. K., Ray, M. N., Crawford, M. A., Giddens, T., and Berner, E. S., Patient perceptions of physician use of handheld computers. AMIA Annu. Symp. Proc. 299–303, 2003.

  13. Ammenwerth, E., Mansmann, U., Iller, C., and Eichstädter, R., Factors affecting and affected by user acceptance of computer-based nursing documentation: Results of a two-year study. J. Am. Med. Inform. Assoc. 10:69–84, 2003.

    Article  Google Scholar 

  14. Zhang, W. P., Yamauchi, K., Mizuno, S., Zhang, R., and Huang, D. M., Analysis of cost and assessment of computerized patient record systems in Japan based on questionnaire survey. Med. Inform. Internet Med. 29:229–238, 2004.

    Article  Google Scholar 

  15. Laerum, H., Karlsen, T. H., and Faxvaag, A., Use of and attitudes to a hospital information system by medical secretaries, nurses and physicians deprived of the paper-based medical record: A case report BMC Med. Inform. Decis. Mak. 4:18, 2004.

    Article  Google Scholar 

  16. Hier, D. B., Rothschild, A., LeMaistre, A., and Keeler, J., Differing faculty and housestaff acceptance of an electronic health record one year after implementation. Medinfo. 11:1300–1303, 2004.

    Google Scholar 

  17. Rotich, J. K., Hannan, T. J., Smith, F. E., Bii, J., Odero, W. W., Vu, N., Mamlin, B. W., Mamlin, J. J., Einterz, R. M., and Tierney, W. M., Installing and implementing a computer-based patient record system in sub-Saharan Africa: The Mosoriot Medical Record System. Med. Inform. Internet Med. 10:295–303, 2003.

    Google Scholar 

  18. Stengel, D., Bauwens, K., Walter, M., Kopfer, T., and Ekkernkamp, A., Comparison of handheld computer-assisted and conventional paper chart documentation of medical records. A randomized, controlled trial. J. Bone Joint Surg. Am. 86:553–560, 2004.

    Google Scholar 

  19. Likourezos, A., Chalfin, D. B., Murphy, D. G., Sommer, B., Darcy, K., and Davidson, S. J., Physician and nurse satisfaction with an Electronic Medical Record System. J. Emerg. Med. 27:419–424, 2004.

    Article  Google Scholar 

  20. Pagliari, C., Gilmour, M., and Sullivan, F., Evaluating national electronic clinical communications implementation in Scotland. Medinfo. 11:1794, 2004.

    Google Scholar 

  21. Baum, A., Figar, S., Serverino, J., Assale, D., Schachner, B., Otero, P., Luna, D., and de Quiros, F. G., Assessing the impact of change in the organization of a technical support system for a health information systems (HIS). Medinfo. 11:1367–1370, 2004.

    Google Scholar 

  22. Hobson, J. C., Khemani, S., and Singh, A., Prospective audit of the quality of ENT emergency clinic notes before and after introduction of a computerized template. J. Laryngol. Otol. 119:264–266, 2005.

    Article  Google Scholar 

  23. Kaushal, R., Blumenthal, D., Poon, E. G., Jha, A. K., Franz, C., Middleton, B., Glaser, J., Kuperman, G., Christino, M., Fernandopulle, R., Newhouse, J. P., and Bates, D. W., The costs of a national health information network. Ann. Intern. Med. 143:165–173, 2005.

    Google Scholar 

  24. Raja, E. E., Mahal, R., and Masih, V. B., An exploratory study to assess the computer knowledge, attitude and skill among nurses in healthcare setting of a selected hospital in Ludhiana, Punjab, India. Medinfo. 11:1304–1307, 2004.

    Google Scholar 

  25. Clayton, P. D., Naus, S. P., Bowes, W. A., Madsen, T. S., Wilcox, A. B., Orsmond, G., Rocha, B., Thornton, S. N., Jones, S., Jacobsen, C. A., Udall, M. R., Rhodes, M. L., Wallace, B. E., Cannon, W., Gardner, J., Huff, S. M., and Leckman, L., Physician use of electronic medical records: issues and successes with direct data entry and physician productivity. AMIA Annu. Symp. Proc. 141–145, 2005.

  26. White, M., Computerised versus handwritten records. Paediatr. Nurs. 17:15–18, 2005.

    Google Scholar 

  27. Wilcox, A. B., Jones, S. S., Dorr, D. A., Cannon, W., Burns, L., Radican, K., Christensen, K., Brunker, C., Larsen, A., Narus, S. P., Thornton, S. N., and Clayton, P. D., Use and impact of a computer-generated patient summary worksheet for primary care. AMIA Annu. Symp. Proc. 824–828, 2005.

  28. Makela, K., Virjo, I., Aho, J., Kalliola, P., Koivukoski, A. M., Kurunmaki, H., Kahara, M., Uusitalo, L., Valli, M., Vuotari, V., and Ylinen, S., Electronic patient record systems and the general practitioner: an evaluation study. J. Telemed. Telecare. 11:S2:66–68, 2005.

    Article  Google Scholar 

  29. Zurita, L., and Nohr, C., Patient opinion–EHR assessment from the users perspective. Medinfo. 11:1333–1336, 2004.

    Google Scholar 

  30. Breslow, M. J., Rosenfeld, B. A., Doerfler, M., Burke, G., Yates, G., Stone, D. J., Tomaszewicz, P., Hochman, R., and Plocher, D. W., Effect of a multiple-site intensive care unit telemedicine program on clinical and economic outcomes: an alternative paradigm for intensivist staffing. Crit. Care Med. 32:31–38, 2004.

    Article  Google Scholar 

  31. Scalvini, S., Capomolla, S., Zanelli, E., Benigno, M., Domenighini, D., Paletta, L., Glisenti, F., and Giordano, A., Effect of home-based telecardiology on chronic heart failure: Costs and outcomes. J. Telemed. Telecare. 11:S1:16–18, 2005.

    Article  Google Scholar 

  32. Noble, S. M., Coast, J., and Benger, J. R., A cost-consequences analysis of minor injuries telemedicine. J. Telemed. Telecare. 11:15–19, 2005.

    Article  Google Scholar 

  33. Persaud, D. D., Jreige, S., Skedgel, C., Finley, J., Sargeant, J., and Hanlon, N., An incremental cost analysis of telehealth in Nova Scotia from a societal perspective. J. Telemed. Telecare. 11:77–84, 2005.

    Article  Google Scholar 

  34. Dillon, E., Loermans, J., Davis, D., and Xu, C., Evaluation of the westerrn Australian Department of Health telehealth project. J. Telemed. Telecare. 11:S2:19–S2:121, 2005.

    Article  Google Scholar 

  35. Larcher, B., Arisi, E., Berloffa, F., Demichelis, F., Eccher, C., Galligioni, E., Galvagni, M., Martini, G., Sboner, A., Tomio, L., Zumiani, G., Graiff, A., and Forti, S., Analysis of user-satisfaction with the use of a teleconsultation system in oncology. Med. Inform. Internet. Med. 28:73–84, 2003.

    Article  Google Scholar 

  36. Bratan, T., Clarke, M., Jones, R., Larkworthy, A., and Paul, R., Evaluation of the practical feasibility and acceptability of home monitoring in residential homes. J. Telemed. Telecare. 11(Suppl 1):29–31, 2005.

    Article  Google Scholar 

  37. Mair, F. S., Goldstein, P., May, C., Angus, R., Shiels, C., Hibbert, D., O’Connor, J., Boland, A., Roberts, C., Haycox, A., and Capewell, S., Patient and provider perspectives on home telecare: preliminary results from a randomized controlled trial. J. Telemed. Telecare. 11(Suppl 1):95–97, 2005.

    Article  Google Scholar 

  38. Linassi, A. G., and Li Pi Shan, R., User satisfaction with a telemedicine amputee clinic in Saskatchewan. J. Telemed. Telecare. 11:414–418, 2005.

    Article  Google Scholar 

  39. Young, T. L., and Ireson, C., Effectiveness of school-based telehealth care in urban and rural elementary schools. Pediatrics. 112:1088–1094, 2003.

    Article  Google Scholar 

  40. Baldwin, L., Clarke, M., Hands, L., Knott, M., and Jones, R., The effect of telemedicine on consultation time. J. Telemed. Telecare. 9:S1:71–73, 2003.

    Article  Google Scholar 

  41. Finkelstein, J., Khare, R., and Ansell, J., Feasibility and patients’ acceptance of home automated telemanagement of oral anticoagulation therapy. Proc. AMIA Symp. 230–234, 2003.

  42. Vontetsianos, T., Giovas, P., Katsaras, T., Rigopoulou, A., Mpirmpa, G., Giaboudakis, P., Koyrelea, S., Kontopyrgias, G., and Tsoulkas, B., Telemedicine-assisted home support for patients with advanced chronic obstructive pulmonary disease: preliminary results after nine-month follow-up. J. Telemed. Telecare. 11:S1:86–S1:88, 2005.

    Article  Google Scholar 

  43. Grady, B. J., and Melcer, T., A retrospective evaluation of TeleMental Healthcare services for remote military populations. Telemed J E Health. 11:551–558, 2005.

    Article  Google Scholar 

  44. Wong, Y. K., Hui, E., and Woo, J., A community-based exercise programme for older persons with knee pain using telemedicine. J. Telemed. Telecare. 11:310–315, 2005.

    Article  Google Scholar 

  45. Beuscart-Zephir, M. C., Pelayo, S., Degoulet, P., Anceaux, F., Guerlinger, S., and Meaux, J. J., A usability study of CPOE’s medication administration functions: impact on physician-nurse cooperation. Medinfo. 11:1018–1022, 2004.

    Google Scholar 

  46. Galanter, W. L., Polikaitis, A., and DiDomenico, R. J., A trial of automated safety alerts for inpatient digoxin use with computerized physician order entry. J. Am. Med. Inform. Assoc. 11:270–277, 2004.

    Article  Google Scholar 

  47. Shulman, R., Singer, M., Goldstone, J., and Bellingan, G., Medication errors: a prospective cohort study of hand-written and computerised physician order entry in the intensive care unit. Crit. Care. 9:R516–R521, 2005.

    Article  Google Scholar 

  48. Galanter, W. L., Didomenico, R. J., and Polikaitis, A., A trial of automated decision support alerts for contraindicated medications using computerized physician order entry. J. Am. Med. Inform. Assoc. 12:269–274, 2005.

    Article  Google Scholar 

  49. Sintchenko, V., Iredell, J. R., Gilbert, G. L., and Coiera, E., Handheld computer-based decision support reduces patient length of stay and antibiotic prescribing in critical care. J. Am. Med. Inform. Assoc. 12:398–402, 2005.

    Article  Google Scholar 

  50. Short, D., Frischer, M., and Bashford, J., The development and evaluation of a computerised decision support system for primary care based upon ‘patient profile decision analysis’. Inform. Prim. Care. 11:195–202, 2003.

    Google Scholar 

  51. Ruland, C. M., White, T., Stevens, M., Fanciullo, G., and Khilani, S. M., Effects of a computerized system to support shared decision making in symptom management of cancer patients: Preliminary results. J. Am. Med. Inform. Assoc. 10:573–579, 2003.

    Article  Google Scholar 

  52. Park, R. W., Shin, S. S., Choi, Y. I., Ahn, J. O., and Hwang, S. C., Computerized physician order entry and electronic medical record systems in Korean teaching and general hospitals: Results of a 2004 survey. J. Am. Med. Inform. Assoc. 12:642–647, 2005.

    Article  Google Scholar 

  53. Gandhi, T. K., Weingart, S. N., Seger, A. C., Borus, J., Burdick, E., Poon, E. G., Leape, L. L., and Bates, D. W., Outpatient prescribing errors and the impact of computerized prescribing. J. Gen. Intern. Med. 20:837–841, 2005.

    Article  Google Scholar 

  54. Ohsfeldt, R. L., Ward, M. M., Schneider, J. E., Jaana, M., Miller, T. R., Lei, Y., and Wakefield, S., Implementation of hospital computerized physician order entry systems in a rural state: Feasibility and financial impact. J. Am. Med. Inform. Assoc. 12:20–27, 2005, Jan.

    Article  Google Scholar 

  55. Bogucki, B., Jacobs, B. R., and Hingle, J., Computerized reminders reduce the use of medications during shortages. J. Am. Med. Inform. Assoc. 11:278–280, 2004.

    Article  Google Scholar 

  56. Van den Brink, J. L., Moorman, P. W., de Boer, M. F., Pruyn, J. F., Verwoerd, C. D., and van emmel, J. H., Involving the patient: a prospective study on use, appreciation and effectiveness of an information system in head and neck cancer care. Int. J. Med. Inform. 74:839–849, 2005.

    Article  Google Scholar 

  57. Choi, S. S., Jazayeri, D. G., Mitnick, C. D., Chalco, K., Bayona, J., and Fraser, H. S., Implementation and initial evaluation of a Web-based nurse order entry system for multidrug-resistant tuberculosis patients in Peru. Medinfo. 11:202–206, 2004.

    Google Scholar 

  58. Odhiambo-Otieno, G. W., Evaluation of existing district health management information systems a case study of the district health systems in Kenya. Int. J. Med Inform. 74:733–744, 2005.

    Article  Google Scholar 

  59. Scott, J. T., Rundall, T. G., Vogt, T. M., and Hsu, J., Kaiser Permanente’s experience of implementing an electronic medical record: A qualitative study. BMJ. 331:1313–1316, 2005.

    Article  Google Scholar 

  60. Aarts, J., and Berg, M., A tale of two hospitals: a sociotechnical appraisal of the introduction of computerized physician order entry in two Dutch hospitals. Medinfo. 11:999–1002, 2004.

    Google Scholar 

  61. Koppel, R., Metlay, J. P., Cohen, A., Abaluck, B., Localio, A. R., Kimmel, S. E., and Strom, B. L., Role of computerized physician order entry systems in facilitating medication errors. JAMA. 293:1197–1203, 2005.

    Article  Google Scholar 

  62. Vimarlund, V., and Olve, N. G., Economic analyses for ICT in elderly healthcare: Questions and challenges. Health Informatics J. 11:309–321, 2005.

    Article  Google Scholar 

  63. Olve, N. G., Vimarlund, V., and Agerbo, M. Evaluation as multi-actor trade-off—A challenge in introducing ICT innovations in the health sector. Invited paper, 4th WSEAS International Conference on E-Activities. Miami, Florida, USA, November 17–19, 2005.

  64. Adams, W. G., Mann, A. M., and Bauchner, H., Use of an electronic medical record improves the quality of urban pediatric primary care. Pediatrics. 111:626–632, 2003.

    Article  Google Scholar 

  65. Pizziferri, L., Kittler, A. F., Volk, L. A., Honour, M. M., Gupta, S., Wang, S., Wang, T., Lippincott, M., Li, Q., and Bates, D. W., Primary care physician time utilization before and after implementation of an electronic health record: A time-motion study. J. Biomed. Inform. 38:176–188, 2005.

    Article  Google Scholar 

  66. Bryson, M., Tidy, N., Smith, M., and Levy, S., An online survey of nurses’ perceptions, knowledge and expectations of the National Health Service modernization programme. J. Telemed. Telecare. 11:S1:64–66, 2005.

    Article  Google Scholar 

  67. Hebert, M. A., and Korabek, B., Stakeholder readiness for telehomecare: implications for implementation. Telemed J E Health. 10:85–92, 2004.

    Article  Google Scholar 

  68. Kaufman, D. R., Starren, J., Patel, V. L., Morin, P. C., Hilliman, C., Pevzner, J., Weinstock, R. S., Goland, R., and Shea, S. A cognitive framework for understanding barriers to the productive use of a diabetes home telemedicine system. AMIA Annu. Symp. Proc. 356–360, 2003.

  69. Ash, J. S., Sittig, D. F., Seshadri, V., Dykstra, R. H., Carpenter, J. D., and Stavri, P. Z., Adding insight: a qualitative cross-site study of physician order entry. Medinfo. 11:1013–1017, 2004.

    Google Scholar 

  70. Ash, J. S., Gorman, P. N., Seshadri, V., and Hersh, W. R., Computerized physician order entry in U.S. hospitals: results of a 2002 survey. J. Am. Med. Inform. Assoc. 11:95–99, 2004.

    Article  Google Scholar 

  71. Ammenwerth, E., and de Keizer, N., An inventory of evaluation studies of information technology in healthcare trends in evaluation research 1982–2002. Methods Inf. Med. 44:44–56, 2005.

    Google Scholar 

  72. Kaplan, B., Evaluating informatics applications—Clinical decision support systems literature review. Int. J. Med. Inform. 64:39–56, 2001.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer and Information Science, Linköping University, Campus valla, Linköping, 58183, Sweden

    Bahlol Rahimi & Vivian Vimarlund

  2. Department of Social Medicine, Urmia University of Medical Sciences, Urmia, Iran

    Bahlol Rahimi

Authors
  1. Bahlol Rahimi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vivian Vimarlund
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bahlol Rahimi.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Rahimi, B., Vimarlund, V. Methods to Evaluate Health information Systems in Healthcare Settings: A Literature Review. J Med Syst 31, 397–432 (2007). https://doi.org/10.1007/s10916-007-9082-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10916-007-9082-z

Keywords

Search

Navigation