Retrieval performance and information theory

doi:10.1016/0306-4573(77)90034-6

Information Processing & Management

Volume 13, Issue 3, 1977, Pages 155-165

https://doi.org/10.1016/0306-4573(77)90034-6 Get rights and content

Abstract

This paper challenges the meaningfulness of precision and recall values as a measure of performance of a retrieval system. Instead, it advocates the use of a normalised form of Shannon's functions (entropy and mutual information). Shannon's four axioms are replaced by an equivalent set of five axioms which are more readily shown to be pertinent to document retrieval.

The applicability of these axioms and the conceptual and operational advantages of Shannon's functions are the central points of the work.

The applicability of the results to any automatic classification is also outlined.

References (12)

C.E. Shannon
The mathematical theory of communication
Bell Syst. Tech. J.
(1948)
C.E. Shannon et al.
The Mathematical Theory of Communication
(1949)
A.I. Khinchin
Mathematical Foundations of Information Theory
(1957)
G. Salton
Automatic Information Organisation and Retrieval
(1968)
J.A. Swets
Effectiveness of information retrieval methods
Am. Docum.
(1976)

There are more references available in the full text version of this article.

Cited by (11)

Human subjectivity and performance limits in document retrieval
1996, Information Processing and Management
Test sets for the document-retrieval task composed of human relevance judgments have been constructed that allow one to compare human performance directly with that of automatic methods and that place absolute limits on performance by any method. Current retrieval systems are found to generate only about half of the information allowed by these absolute limits. The data presented here strongly suggest that most of the improvement that could be achieved consistent with these limits can only be achieved by incorporating specific subject information into retrieval systems.
An information measure of retrieval performance
1992, Information Systems
The classical measures of information retrieval quality, namely precision and recall, each have disadvantages depending on the circumstances. In judging the output of an automatic retrieval procedure it would be useful to have a single number which is practical to compute and summarizes the quality of the output. Such a measure could facilitate the study of retrieval methodology and clarify efforts to optimize retrieval. We show how such a measure may be defined for retrieval methods with ranked output. It is based on the change in the Shannon entropy of the distribution of relevant documents in the database that is produced by the act of retrieval. Properties of this new measure are derived and reveal that it conforms to intuitive expectations.
Dynamic thesaural systems: A bibliometric study of terminological and conceptual change in sociology and economics with application to the design of dynamic thesaural systems
1989, Information Processing and Management
Thesauri have been used in the library and information science field to provide a standard descriptor language for indexers or searchers to use in an information storage and retrieval system. One difficulty has been the maintenance and updating of thesauri since terms used to describe concepts change over time and vary between users. This study investigates a mechanism by which thesauri can be updated and maintained using citation, co-citation analysis and citation context analysis.
Data in twenty-six specialty areas in economics and sociology from the Social Sciences Citation Indexes, 1966–1967, 1973–1974, and 1980–1981 was used. Candidate thesaurus terms were developed semi-automatically from the citation contexts of papers citing in these specialty areas. Experts in these disciplines concluded that these terms were appropriate terms to describe these specialty areas. These candidate thesaurus terms compared favourably to terminology found in a standard subject heading list (L.C. subject headings). The terms generated provided a rich source of alternate terminology.
Use of descriptor search language in automated patent information system (experimental study)
1987, World Patent Information
The main objective of this study undertaken at VNII Informelektro (Information Institute of Electrical Engineering Institute) was to investigate the feasibility of improving the precision of retrieval in searching for patent information. The first part of the study involved a comparison between the abstract patent information databases of VINITI (All-Union Institute of Scientific and Technical Information) and VNIIPI (All-Union Institute of Patent Information) and the second part of the study comprised an experimental comparison of techniques of reducing ‘noise’ (i.e. irrelevant information) in patent searches. The conclusions are:
- 1.
  1. Ranking classification symbols within the field of search in accordance with the expected probability of their being allotted to relevant documents and searching only the symbols of the higher ranks does not, on average, reduce the number of irrelevant documents to be perused per retrieval relevant document.
- 2.
  2. Among the compared techniques of enhancing precision of retrieval, the most successful is a combination of classification and descriptors (i.e. thesaurus terms).
- 3.
  3. Further improvements in precision are dependant upon improving the quality of abstract information.
- 4.
  4. By employing the ‘aphabetic word operator’, i.e. the use of terms outside a thesaurus, it is possible to raise the quality of searches on VNIIPI abstracts to that conducted in machine-readable files of VINITI abstracts.
Ten years progress in quantitative research on libraries
1978, Socio-Economic Planning Sciences
Approximately ten years ago there developed a widespread interest in the application of Operations Research techniques to library problems. An extensive body of literature now exists. However, implementation has been limited and some important topics have been neglected. This survey reviews the areas in which most work has been done, identifies neglected areas and concludes that the progress and lack of it is typical of the application of quantitative research in social service situations. Numerous references are made to the literature.
An entropy-based interpretation of retrieval status value-based retrieval, and its application to the computation of term and query discrimination value
2004, Journal of the American Society for Information Science and Technology

View all citing articles on Scopus

View full text

Retrieval performance and information theory

Abstract

The mathematical theory of communication

Bell Syst. Tech. J.

The Mathematical Theory of Communication

Mathematical Foundations of Information Theory

Automatic Information Organisation and Retrieval

Effectiveness of information retrieval methods

Am. Docum.