Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval

Abstract

We report on the progressive investigation of techniques for supporting interactive query reformulation in the TREC Interactive Track. Two major issues were explored over four successive years: methods of term suggestion; and, interface design to support different system functionalities. Each year's results led to the following year's investigation, with respect to both of these issues. This paper presents first the general motivation for the entire series of studies; then an overview of each year's investigation, its results, and how they influenced the next year's investigation. We discuss what has been learned through this series of investigations about effective term suggestion, usable and useful interface design, and the relationships between these two in support of the TREC Interactive Track task. We conclude with comments about the general methodology employed over this series of studies, and its relevance to the development and evaluation of interactive information retrieval systems.

Introduction

Query formulation, and especially query reformulation, are understood to be among the most difficult tasks that users in interactive information retrieval (IR) systems face. A variety of techniques have been proposed for addressing this general problem, throughout the entire history of IR research (cf. Efthimiadis, 1996). However, very few of them have actually been tested in interactive IR environments. The research described here represents a principled attempt at taking one well-known technique for supporting query reformulation, relevance feedback (RF), and investigating its effectiveness and usability by implementing and evaluating it in the context of a specific interactive IR task.

Interface design for IR systems, although long understood to be an important problem for IR in general (cf. Walker, 1971), has only fairly recently become an active area of research in IR (e.g., Fox et al., 1993, Hearst and Karadi, 1997, Swan and Allan, 1998, Williamson and Shneiderman, 1992). Most IR research has focused upon system functionalities, ignoring interface questions, leading to a situation in which interfaces are often seen as things that are put on top of IR systems, rather than being integral parts of them (cf. the excellent review by Marchionini & Komlodi, 1999). Here, we present the results of a series of studies of interactive IR systems which attempted to address this problem by integrating interface design with development of the RF and other functionalities related to query reformulation. We construe this series of studies as an example of the iterative evaluation/design cycle, as in Egan et al. (1989), but perhaps extending that model in our emphasis on integration of interface with function.

We describe the progressive development of an IR system and its interface designed to support a particular IR task, and to support specific user problems with that task, and with IR systems in general. This was embodied in a series of studies carried out within the Text REtrieval Conferences (TREC) Interactive Track. The starting point for this series of studies was the idea of testing the usability, use and effectiveness of automatic relevance feedback (RF) in interactive IR, embodied in work reported in Belkin, Cool, Koenemann, Ng, and Park (1996). Some results from this investigation, and related work (Koenemann's, 1996), suggested that although system-controlled RF using only positive relevance judgments was usable and useful, additional functionality might be desirable. Specifically, user-controlled RF (RF as a term suggestion device) and RF using both negative and positive judgments were features that users seemed to desire. These results led to a sequence of four studies, implementing and/or studying these new functionalities in various interface structures, using a basic common underlying IR system (Belkin et al., 1997, Belkin et al., 1998, Belkin et al., 1999, Belkin et al., 2000). The reports of these individual studies discussed issues specific to each one, and generally did not focus on explicit interface issues. In this paper, we present an integrated discussion of the entire series, with equal emphasis on functionality and interface design. For detailed discussion of each study, see the appropriate TREC publication.

We began this series of studies in the TREC-5 Interactive Track (Belkin et al., 1997), by investigating the use, usability and effectiveness of a system which implemented RF in the standard way in which it has been suggested for interactive IR; that is, by automatically adding terms to a query, based on the documents which had been judged relevant by the user. This study, like all subsequent ones in the series, was based on the instance recall task set by the TREC Interactive Track (Over, 2001). This task requires subjects to identify the different aspects or instances of a topic, and to save documents which represent those instances. The results of this study (and of a previous study in TREC-4, Belkin et al., 1996), led to some quite explicit changes in both the RF functionality, and the system and interface in which it was implemented, for our next study, in the TREC-6 interactive track.

Three major changes were made with respect to RF for our TREC-6 investigation (Belkin et al., 1998). One was the implementation of RF as a term suggestion device, rather than an automatic query expansion device. This followed from both the results of our TREC-5 study, and Koenemann's (1996) results indicating that RF as term suggestion is preferred to, and works at least as well as automatic RF. Another was allowing both positive and negative relevance judgments to be made, leading to the suggestion of both “good” terms to add to the query with positive weights, and “bad” terms to add to the query with negative weights. The third change, related to the second, was to implement what we have elsewhere termed a “revisionist” version of RF (Cool et al., 1996, Belkin et al., 1997). Since that model is rather different from the “standard” version of RF, and since it was used in our studies in TRECs 6, 7 and 8, we discuss it in some detail in Section 1.2. The results of our TREC-6 study led us to make changes in the way in which RF was presented to the system user, and in a variety of interface-related issues, whose effects were investigated in our TREC-7 study.

In TREC-7 (Belkin et al., 1999), the underlying RF implementation was as in our TREC-6 system, and the main goal was still to investigate the utility of negative RF as implemented in our revisionist model. However, the conceptual model that was presented to the user became one of “term suggestion” rather than automatic, system-controlled RF, and the interface was redesigned to take account of these changes, to respond to some specific problems indicated by the subjects in TREC-6, and to better respond to some general HCI design principles. The results of the TREC-7 study were by-and-large positive with respect to usability issues, but effectiveness and perceived usefulness of term suggestion for the task were not what might have been wished. These results led us to investigate a different mode of term suggestion in our TREC-8 study, as well as to make the system design more directly related to the task itself.

In TREC-8 (Belkin et al., 2000), instead of comparing different versions of RF for term suggestion, we compared RF-based term suggestion to another mode which we hypothesized would be better suited to the instance recall task. In addition, we continued to change the conceptual model of the system that was presented to the user, and to change various interface characteristics to respond both to the task, and to difficulties experienced by the users in TREC-7. This study concluded the series of investigations reported here. Although there are still a few open questions, the results of our TREC-8 study lead us to believe that we have arrived at what seems to be a reasonable and effective way to support query reformulation for the instance recall task in terms of both functionality and interface design.

Automatic RF is well known to be an effective tool for query reformulation in situations in which large numbers of relevance judgments are available, for instance, in information routing or information filtering. Its efficacy has been demonstrated in such situations in a wide variety of studies, almost all of them being done in what can be roughly characterized as batch-mode, non-interactive, experimental IR test-collection situations (cf. Salton and Buckley, 1990, Spink and Losee, 1996). However, there have been rather few investigations of RF in interactive IR environments in which there are small numbers of relevance judgments (e.g., the TREC studies reported on in this paper; Koenemann's, 1996, Robertson et al., 1999, Yang et al., 1999). The results of most of these studies have been either inconclusive, or negative, with the notable exception of Koenemann's (1996), at least when evaluated by traditional IR measures.

The basic requirement for RF to be implemented in batch-mode environments is the existence of a large set of documents on which exhaustive judgments of their relevance, or not, to a specific query have been made. RF of this sort, which we here call the “classic” model of RF, is based on ideas first proposed by Rocchio (1971), and can be succinctly characterized as follows:

• Based on the concept of reaching an “ideal query”.

• The ideal query is the best discriminator between relevant and non-relevant documents.

• Query terms should be “optimal” discriminators.

• Terms which appear only in the original query, or only in positively judged texts are good.

• Terms which appear in both positively and negatively judged texts are bad because they are poor discriminators.

• Terms which appear only in negatively judged texts are ignored, since they offer no information about discrimination value.

Under this model, the query-term weights of terms which appear in the query and positively judged documents are progressively increased through iterations of RF, while the weights of terms which appear in both positively and negatively judged documents are progressively reduced until they reach zero weight, when they are typically removed from the query (some unpublished experiments in non-interactive environments have shown that using negative weights decreases performance). Query expansion, the most effective aspect of RF (Harman, 1992) is through adding terms to the query, with positive weights, which are important terms in the set of positively judged texts.

In contrast to the classic model, we suggest a new, revisionist model of RF, which attempts to take into account the results of our studies of people's information seeking behaviors in interactive IR environments, and which can be characterized as follows:

• The distinction between relevant and non-relevant texts which have the same terms is:

  • the terms are used in different contexts, or

  • the topics are treated peripherally, or

  • the topics are treated from an inappropriate point of view, or

  • polysemy

• RF should distinguish between appropriate and inappropriate treatments of topics.

• Terms which appear in the query, or in positively judged texts, whether or not they also appear in negative texts, are good.

• Terms which appear only in negatively judged texts are bad because they are indicators of inappropriate context, peripheral or inappropriate treatment of the topic, etc.

• Bad terms should be used for query expansion with negative weights, and good terms with positive weights.

In this model, important terms in the negatively judged documents, which do not appear in positively judged documents, are understood as indicators of the inappropriate context, or the main topic, or the inappropriate point of view. This model thus leads us to a quite different way to implement RF. Query terms which appear in positively judged documents (irrespective of their appearance in negatively judged documents) have their query-term weights increased; and, the query is expanded by both the important terms in the positively judged documents (with positive weights) and by the important terms in the negatively judged documents which do not appear in the query or the positively judged documents (with negative weights). Details of how this model of RF was implemented in our studies are presented in Section 4.2.

The next section describes the general methodology that we used in all four of the studies in this series, and defines the basic measures that were used to evaluate each system. Then, we give an overview of each of the four studies, in 3 TREC-5, 4 TREC-6, 5 TREC-7, 6 TREC-8, describing the study goals, the systems that were used, the experimental subjects, and the results, concluding with how the results affected what we investigated in the next study. We then discuss the results of the entire series of studies, and conclude with some remarks on the implications of this work for the design and evaluation of interactive IR systems.