A Theory and Approach to Improving Relevance Ranking in Web Retrieval

Abstract

The development of the World Wide Web (WWW) makes a huge amount of information available on-line, and the amount of information continues to increase. As of March 2001 the Google search engine searches 1,346,966,000 Web pages. Many search systems have been developed to manage this massive collection of information. Investigation shows that the primary method used by these systems is classification. Unfortunately, classification has an intrinsic restriction. Consider this example. Recently, we sent a query that consists of the word x“computer ” to Google, and Google found 33,220,000 relevant Web pages. This number far exceeds anything that people can possibly begin to read. This problem is intrinsic to classification, which means it cannot be avoided. The problem is explained by the Pigeonhole Principle (i.e. Dirichlet’s Box Principle) [10]. Suppose we can classify Web pages using all the English words in a dictionary. Given a particular keyword, let us calculate on average how many Web pages will be classified as relevant. Let totalKeywords be the number of all keywords in a vocabulary list. Let averageKeywords be the average number of keywords that a Web document may have. Let the number of all Web pages be n. Let the number of relevant Web pages be numberRelevant. Then we have:

$$ number Relevant \approx \frac{{n \times average Keywords}} {{total Keywords}}. $$

If n = 1346966000, averageKeywords = 100, and totalKeywords = 10000, then numberRelevant is 13469660.