Relevance in the eye of the search software

The purpose of this article is to look into relevance ranking and its importance in trying to bring some order to the deluge of results in response to a query.

A large‐scale analysis of detailed web logs of various search engines was performed. Sample tests were made on five to eight versions of MEDLINE, ERIC, and PsycINFO on hosts which have comparable versions of the databases and offer relevance ranking.

It was found that, for fairness, it must be ensured that the implementations are identical, they have the same retrospective coverage, the same MEDLINE/PubMed subsets, and (quasi) identical update.

The tests were made early September 2005. As databases are updated at different times, perfect synchronicity is not easy to achieve. When new records are added to the database, they may change the ranking of the test result set. Similarly, a small change in the fine‐tuning of the algorithm may yield different rank order positions of the same record the next time.

Brings together important research findings and suggests a topic for the next column.