The initial development of the WebMedQual scale: Domain assessment of the construct of quality of health web sites

Summary

Objective

To develop a comprehensive instrument assessing quality of health-related web sites.

Methods

Phase I consisted of a literature review to identify constructs thought to indicate web site quality and to identify items. During content analysis, duplicate items were eliminated and items that were not clear, meaningful, or measurable were reworded or removed. Some items were generated by the authors. Phase II: a panel consisting of six healthcare and MIS reviewers was convened to assess each item for its relevance and importance to the construct and to assess item clarity and measurement feasibility.

Results

Three hundred and eighty-four items were generated from 26 sources. The initial content analysis reduced the scale to 104 items. Four of the six expert reviewers responded; high concordance on the relevance, importance and measurement feasibility of each item was observed: 3 out of 4, or all raters agreed on 76–85% of items. Based on the panel ratings, 9 items were removed, 3 added, and 10 revised. The WebMedQual consists of 8 categories, 8 sub-categories, 95 items and 3 supplemental items to assess web site quality. The constructs are: content (19 items), authority of source (18 items), design (19 items), accessibility and availability (6 items), links (4 items), user support (9 items), confidentiality and privacy (17 items), e-commerce (6 items).

Conclusion

The “WebMedQual” represents a first step toward a comprehensive and standard quality assessment of health web sites. This scale will allow relatively easy assessment of quality with possible numeric scoring.

Introduction

Worldwide, the use of the Internet by citizens seeking health information is rising; a point prevalence estimate of more than six million people are using the Internet for health issues daily in the United States (USA) [1]. Among adults with Internet access, the lifetime prevalence (ever looked) of seeking health care information on the Internet is high: 90% (Japan), 86% (Germany), 80% (USA), and 69% (France) [2]. In the USA, this represents about 80–90 million Americans [3], [4]. Other reports estimated lower health seeking rates of 40% of adults with Internet access (USA) [5]. In line with public health mindfulness, the quality of health-related web sites has been of concern for researchers and health professionals due to uncontrollable amount and variety of content now available online to consumers. In the name of free speech, it is unlikely that the Internet needs to be, or will be, regulated. However, several organizations in several countries and governments’ agencies are evaluating or are developing mechanisms to accredit health web sites. In the U.S., the URAC (American Accreditation HealthCare Commission, Inc. formerly Utilization Review Accreditation Commission) [6] and the American Medical Association (AMA) guidelines [7] have been developed, while in Europe, a European Union (EU) commission recently published their criteria as well [8]. Also, more than 3000 web sites currently abide by the Health On the Net (HON) Foundation's Code of Conduct [9], [10]. The Internet is a global entity and will probably always be so. Currently there is no single international standard assessing the quality of health information on the Internet, however, harmonization of Internet-based health information evaluative efforts would benefit all users and international researchers.

Some have challenged the notion of regulating or imposing quality standards for the Internet because the Internet should not be more regulated than other mediums for exchanging information [11], [12], [13]. Nonetheless consumers can be empowered with more health information [12], [13], [14]. Critics, like cyber-community police—e.g. QuackWatch [15], feel consumers can take care of themselves when advised by community watchdogs [12], [16]. At the moment, we do not know if misinformation or bad health web sites can have a negative impact on web users’ health. There is only isolated evidence reported information found on the web had any harmful effects [17], [18], [19], [20] but at least one online surveillance initiative exits [21], [22].

In a recently published review, there have been 79 studies that evaluated health web sites against certain criteria [23]. A wide array of approaches to assess the web-based health information quality have been developed with differing conceptual definitions and scoring, however many of those were based on the frequently cited Silberg et al.'s criteria [24]. Most guidelines proposed thus far are limited in their interpretability and ease of use for quality measurement. However, they represent a credible source of aspects to consider when evaluating quality. Another method of assessing the quality relates to the accuracy of the web site's content per se [25]. Some of the methods used include a comparative assessment of treatment guidelines and medical expert's opinions inspecting the veracity of the information. The limitations of these methods relate to time, expertise and availability of funding. While some reported good inter-rater agreement [26], others reported that reliability of medical experts to be low when rating Internet health content [27]. A number of evaluation criterion for health web sites have been referred to as “accuracy”, “completeness” of information, “balance”, “readability” or the effect on health outcomes [28]. A reflection on the construct and definition of “quality” of health-related web site will be presented.

Due to the lack of standardized measures to assess the quality of health information on the Internet, evaluative studies assessing quality of health-related web sites could not be meaningfully compared. As a start, there needs to be standardized operational definitions developed to assess the quality of health web sites in research [23]. Not only in research, but also for health professionals and webmasters, an objective instrument to evaluate web sites would benefit them so that they could recommend a site or approve the link to a partner site. The present study aimed at developing an instrument to assess the quality of health web sites that considers existing guidelines and rating tools. This study represents the first phase of instrument development, which includes identification of constructs, item generation and reduction, and content validation.