A split-list approach for relevance feedback in information retrieval

Abstract

In this paper we present a new algorithm for relevance feedback (RF) in information retrieval. Unlike conventional RF algorithms which use the top ranked documents for feedback, our proposed algorithm is a kind of active feedback algorithm which actively chooses documents for the user to judge. The objectives are (a) to increase the number of judged relevant documents and (b) to increase the diversity of judged documents during the RF process. The algorithm uses document-contexts by splitting the retrieval list into sub-lists according to the query term patterns that exist in the top ranked documents. Query term patterns include a single query term, a pair of query terms that occur in a phrase and query terms that occur in proximity. The algorithm is an iterative algorithm which takes one document for feedback in each of the iterations. We experiment with the algorithm using the TREC-6, -7, -8, -2005 and GOV2 data collections and we simulate user feedback using the TREC relevance judgements. From the experimental results, we show that our proposed split-list algorithm is better than the conventional RF algorithm and that our algorithm is more reliable than a similar algorithm using maximal marginal relevance.

Introduction

Relevance feedback (RF) is known to be effective for improving retrieval effectiveness (Harman, 1992, Rocchio, 1971, Salton and Buckley, 1990). RF requires effort and time from users to judge whether a document is relevant to the user’s information need. When a user judges a particular document to be non-relevant, in the point of view from users, some effort and time is wasted because the document provides no relevant information to the user. As a result, it is better to have more relevant documents to be judged by the user in the RF process. However, judging two very similar relevant documents also wastes effort and time from users because the information contained in the two documents is nearly the same. Thus, no additional relevant information is provided to the user. Therefore, two main factors would affect the user’s satisfaction in the RF process:

• the number of relevant documents (the more the better), and

• the diversity of the documents (the more diverse the better).

In conventional RF, the user would judge documents from the top ranked ones in the result list by assuming that the top ranked documents contain more relevant information. In some cases, the top ranked documents are very similar to each other or even identical. Judging the relevance of the nearly identical documents provides no additional useful information to both the user and the retrieval system. Therefore, the set of documents used for RF may not be the top ranked ones. This is called active feedback (Chen and Lu, 2010, Shen and Zhai, 2005, SIA et al., 2007, Xu and Akella, 2008a, Xu and Akella, 2008b, Xu et al., 2007, Zhang and Zhang, 2010) in which the retrieval system actively chooses suitable documents for the user to judge for relevance.

Our proposed algorithm is a kind of active feedback algorithm which uses document-contexts by splitting the retrieval list into sub-lists according to the query term patterns that exist in the top ranked documents. Fig. 1 shows an example of the lists of document-contexts for the query “Hubble Telescope Achievements”. Note that only lists for single query terms are shown. Document-contexts are used because we believe that relevant information is located around query terms (Wu et al., 2007, Wu et al., 2008). By splitting the retrieval list into sub-lists, we hope that the proportion of relevant documents in a particular sub-list will be higher than that of the others. Therefore, the scores of document-contexts in the particular sub-list will be higher. By that we can increase the number of relevant documents judged by the user in the RF process. Also, the set of documents being judged by the user in the split-list approach is different from the set of top ranked documents. Hence, it can increase the diversity of the documents being judged.

The rest of the paper is organized as follows. In Section 2, we describe some active feedback algorithms including the gapped method and cluster method in Shen and Zhai (2005) and the maximal marginal relevance method (Carbonell & Goldstein, 1998). Section 3 describes our split-list approach using document-contexts. We show the experiment results in Sections 4 Experiments, 5 Experimental results concludes the paper.