Measuring user perceptions of Web site reputation

Abstract

In this study, we compare a search tool, TOPIC, with three other widely used tools that retrieve information from the Web: AltaVista, Google, and Lycos. These tools use different techniques for outputting and ranking Web sites: external link structure (TOPIC and Google) and semantic content analysis (AltaVista and Lycos). TOPIC purports to output, and highly rank within its hit list, reputable Web sites for searched topics. In this study, 80 participants reviewed the output (i.e., highly ranked sites) from each tool and assessed the quality of retrieved sites. The 4800 individual assessments of 240 sites that represent 12 topics indicated that Google tends to identify and highly rank significantly more reputable Web sites than TOPIC, which, in turn, outputs more than AltaVista and Lycos, but this was not consistent from topic to topic. Metrics derived from reputation research were used in the assessment and a factor analysis was employed to identify a key factor, which we call ‘repute’. The results of this research include insight into the factors that Web users consider in formulating perceptions of Web site reputation, and insight into which search tools are outputting reputable sites for Web users. Our findings, we believe, have implications for Web users and suggest the need for future research to assess the relationship between Web page characteristics and their perceived reputation.

Introduction

Multiple search engines retrieve and rank Web sites based on their semantic content and/or on their positioning within the link structure of the Web. These engines output from a dozen to a million or more sites on almost any topic, but often leave the user to sort and sift through the results looking for quality information. The search engines do a reasonable job of ranking Web sites according to their semantic content, but they do not incorporate into search algorithms indicators of quality. Assessing the quality of a Web site’s content is left to the individual, who often uses criteria other than topicality in his or her assessment (Rieh & Belkin, 1998).

One criterion used in evaluating Web sites is reputation. In general, reputation is an important personal asset in which to invest and protect (Weigelt & Camerer, 1988) and a strategic asset of public and private organizations (Fombrun, 2001). It can be considered an expectation of quality (Shapiro, 1982). In essence, one perceives through a variety of factors that a person or object has a good reputation and it raises expectations about the interactions one might have with that person or object. This situation is similar to interactions on the Web. For example, one has expectations about interactions with Amazon.com and those same expectations may not exist for a host of other online booksellers. On exposure to a Web site, however, a user may not know about the reputation or have perceptions about the reputation of that site and may mentally ask a number of questions: Is this “for real”? Can I trust it? Is it accurate? And so on. Users sometimes make split-second decisions about whether to stay and read, or move on quickly to other sites. This decision is based on how the individual perceives a whole series of visual and content cues emanating from the site. From this surface inspection, individuals derive an impression––they assess the quality of that site and thus, infer its reputation.

The volume of Web sites and the predicted growth of Web content preclude the ability to provide human indexing and ranking of Web sites for all given topics. Search engines must be able to automatically sift through this content and highly rank sites that are perceived as reputable to users, and not merely identify pages that are on a topic. At present, identifying good pages is left to users, who have to sift through results lists and the related Web sites, and make quick judgements about a page. In this research, we tested the ability of current search engines to output ‘reputable’ Web sites.

Unlike typical search tools, TOPIC (http://www.cs.toronto.edu/db/topic/search.html), developed by Rafiei and Mendelzon (2000), identifies the topics (or subjects) for which a Web site has a good reputation. It does so by mining both the link structure and the textual content of that site to indicate how well connected the site is to other well-connected sites on the Web. Thus, upon searching the Web for sites on a topic, TOPIC purports to output and highly rank sites that are highly perceived by the Web community for that topic.

To do this research, which is a follow-up to a pilot study (Keast, Toms, & Cherry, 2001), we develop a metric for testing the reputation of Web sites and compare the output from TOPIC with that of three other search engines (Google, AltaVista and Lycos) to ascertain if certain tools more often output and highly rank ‘reputable’ Web sites for given topics. Google (http://www.google.com) and TOPIC primarily rely on link structure analysis whereas AltaVista (http://www.altavista.com) and Lycos (http://www.lycos.com) depend on semantic content analysis when outputting and ranking Web sites for a search on a specific topic. In our pilot study, we included search engines and directories with human-selected and indexed content (Yahoo and Britannica.com). This third type was dropped in this larger study, as it performed no better than the automated tools in the pilot, and thus was deemed less essential to the intent of our study––the abilities of automatic indexing and ranking tools to output quality Web sites. Automated tools must be able to address the task of indexing and sifting through a large number of Web sites to allow users to access information they consider to be salient and of good quality. Thus, the work reported here assesses the ability of two types of indexing and ranking tools, represented by four search engines, to output Web sites that are perceived by Web users to be highly reputable on a specific topic.