Upper bounds for retrieval performance and their use measuring performance and generating optimal boolean queries: Can it get any better than this?

doi:10.1016/0306-4573(94)90064-7

Information Processing & Management

Volume 30, Issue 2, March–April 1994, Pages 193-203

https://doi.org/10.1016/0306-4573(94)90064-7 Get rights and content

Abstract

The best-case, random, and worst-case document rankings and retrieval performance may be determined using a method discussed here. Knowledge of the best case performance allows users and system designers to (a) determine how close to optimality their search is and (b) select queries and matching functions that will produce the best results. A method for deriving the optimal Boolean query for a given level of recall is suggested, as is a method for determining the quality of a Boolean query. Measures are proposed that modify conventional text retrieval measures such as precision, E, and average search length, so that the values for these measures are 1 when retrieval is optimal, 0 when retrieval is random, and −1 when worst-case. Tests using one of these measures show that many retrievals are optimal. Consequences for retrieval research are examined.

References (26)

M.B. Eisenberg
Measuring relevance judgments
Information Processing & Management
(1988)
E.M. Keen
Presenting results of experimental retrieval comparisons
Information Processing & Management
(1992)
D.H. Kraft et al.
Stopping rules and their effect on expected search length
Information Processing & Management
(1979)
R.M. Losee
An analytic measure predicting information retrieval system performance
Information Processing & Management
(1991)
R.M. Losee
Term dependence: Truncating the Bahadur Lazarsfeld expansion
Information Processing & Management
(1994)
R.M. Losee et al.
Integrating Boolean queries in conjunctive normal form with probabilistic retrieval models
Information Processing & Management
(1988)
A. Bookstein
Information retrieval: A sequential learning process
Journal of the American Society for Information Science
(1983)
R. Burgin et al.
Improving disambiguation in FASIT
Journal of the American Society for Information Science
(1992)
W.S. Cooper
Expected search length: A single measure of retrieval effectiveness based on weak ordering action of retrieval systems
Journal of the American Society for Information Science
(1968)
D.H. Kraft et al.
Evaluation of information retrieval systems: A decision theory approach
Journal of the American Society for Information Science
(1978)

R.M. Losee

Parameter estimation for probabilistic document retrieval models

Journal of the American Society for Information Science

(1988)

S.E. Robertson et al.

Weighting, ranking and relevance feedback in a front-end system

Journal of Information Science

(1986)

S.E. Robertson

The probability ranking principle in IR

Journal of Documentation

(1977)

Cited by (12)

Is 1 noun worth 2 adjectives? Measuring relative feature utility
2006, Information Processing and Management
Are two adjectives worth the same as a single noun when documents are ordered based on decreasing topicality? We propose an easy to interpret single number Relative Feature Utility (RFU) measure of the relative worth of using specific linguistic or non-linguistic features or sets of features in computational systems that order or filter media, such as information retrieval and classification systems. This measure allows one to make easily interpreted claims about the relative utility of features such as parts-of-speech, term suffixes, phrases vs. single terms, annotations, hyperlinks, citations, index terms, and metadata when ordering natural language text or other media. Data is provided for the RFU for stemming characteristics, part-of-speech tags, and phrase lengths, as well as retrieval characteristics and procedures. Using this linear measure of the relative utility of features makes available a wide range of cost-benefit analyses and decision theoretic techniques, allowing the study of whether or not to use many different kinds of representational information or tagging systems, and for the design of indexing and metadata systems. Some characteristics of natural languages used in the spectrum from softer to harder sciences, as well as medical terminology, are studied.
Adaptive relevance feedback method of extended Boolean model using hierarchical clustering techniques
2006, Information Processing and Management
The relevance feedback process uses information obtained from a user about a set of initially retrieved documents to improve subsequent search formulations and retrieval performance. In extended Boolean models, the relevance feedback implies not only that new query terms must be identified and re-weighted, but also that the terms must be connected with Boolean And/Or operators properly. Salton et al. proposed a relevance feedback method, called DNF (disjunctive normal form) method, for a well established extended Boolean model. However, this method mainly focuses on generating Boolean queries but does not concern about re-weighting query terms. Also, this method has some problems in generating reformulated Boolean queries. In this study, we investigate the problems of the DNF method and propose a relevance feedback method using hierarchical clustering techniques to solve those problems. We also propose a neural network model in which the term weights used in extended Boolean queries can be adjusted by the users’ relevance feedbacks.
Determining information retrieval and filtering performance without experimentation
1995, Information Processing and Management
The performance of an information retrieval or text and media filtering system may be determined through analytic methods as well as by traditional simulation or experimental methods. These analytic methods can provide precise statements about expected performance. They can thus determine which of two similarly performing systems is superior. For both a single query term and for a multiple query term retrieval model, a method for comparing the performance of different probabilistic retrieval methods is developed. This method may be used in computing the average search length for a query, given only knowledge of database parameter values. Predictive models for inverse document frequency, binary independence, and relevance feedback based retrieval and filtering are described. Simulations illustrate how the single term model performs and sample performance predictions are given for single term and multiple term problems.
Building a framework for the probability ranking principle by a family of expected weighted rank
2009, ACM Transactions on Information Systems
Measuring retrieval effectiveness with Average Distance Measure (ADM)
2006, Information-Wissenschaft und Praxis
A similarity-based method for retrieving documents from the SCI/SSCI database
2006, Journal of Information Science

View all citing articles on Scopus

View full text

Upper bounds for retrieval performance and their use measuring performance and generating optimal boolean queries: Can it get any better than this?

Abstract

Information Processing & Management

Information Processing & Management

Information Processing & Management

Information Processing & Management

Information Processing & Management

Information Processing & Management

Information retrieval: A sequential learning process

Journal of the American Society for Information Science

Improving disambiguation in FASIT

Journal of the American Society for Information Science

Expected search length: A single measure of retrieval effectiveness based on weak ordering action of retrieval systems

Journal of the American Society for Information Science

Evaluation of information retrieval systems: A decision theory approach

Journal of the American Society for Information Science

Parameter estimation for probabilistic document retrieval models

Journal of the American Society for Information Science

Weighting, ranking and relevance feedback in a front-end system

Journal of Information Science

The probability ranking principle in IR

Journal of Documentation