Abstract
The Ostensive Model proposes a manner of structuring the uncertainty associated with individual relevance judgements as sources of evidence in relevance feedback. It proposes temporal profiles of uncertainty, motivating the application of a particular class of discount function with respect to the age of the evidence. This paper presents an initial evaluation of the relative effectiveness of different uncertainty discount functions.
A novel direct manipulation interface to a multimedia retrieval system embodying the Ostensive Model is outlined briefly. The paper describes the construction and characteristics of a new image test collection utilising multiple binary relevance assessments. The use of such multiple assessments and multiple interpretations of them are discussed. The evaluation environment is detailed in terms of the interface, test collection, and tasks set to users. Multiple interpretations of the results, and the statistical significance of comparisons are presented.
The results obtained in the evaluation are consistent with the proposals of the Ostensive Model—reinforcing a particular evidence profile. The results give clear pointers to further, more specific, evaluations.
Article PDF
Similar content being viewed by others
References
Agosti M, Crestani F and Melucci M (1994) TACHIR: a tool for the automatic construction of hypertexts for information retrieval. In: Proceedings of RIAO 94, Rockerfeller University, pp. 338–357.
Bates MJ (1989) The design of browsing and berrypicking techniques for the online search interface. Online Review, 13(5):407–423.
Belkin NJ, OddyRand Brooks H(1982)ASKfor information retrieval: Part I. Journal of Documentation, 38:61–71.
Borlund P and Ingwersen P (1997) The development of a method for the evaluation of interactive information retrieval systems. Journal of Documentation, 53(3):225–250.
Borlund P and Ingwersen P (1998) Measures of relative relevance and ranked half-life: Performance indicators for interactive IR. In: Proceedings of ACM SIGIR '98, Melbourne, pp. 324–331.
Campbell I and van Rijsbergen CJ (1996) The ostensive model of developing information. In: Ingwersen P and Pors NO, Eds. Information Science: Integration in Perspective, Proceedings of the CoLIS-2 conference, Copenhagen, pp. 251–268.
ChalmersMand Chitson P (1992) Bead: explorations in information visualisation. In: Proceedings ofACM-SIGIR conference, pp. 330–337.
Chiaramella Yand Mechkour M(1997) Indexing an ImageTest Collection,Technical report, Chapter of Deliverable D10, FERMI ESPRIT Basic Research Action 81034.
Ellis D, Furner-Hines J and Willett P (1994) On the measurement of inter-linker consistency and retrieval effectiveness in hypertext databases. In: Proceedings of ACM SIGIR '94, Dublin, pp. 51–60.
FERMI project (1994) ESPRIT Basic Research Action 81034. Documentation and Technical reports web server. "http://www.dcs.gla.ac.uk/fermi".
Ingwersen P (1994) Polyrepresentation of information needs and semantic entities. In: Proceedings of 17th ACMSIGIR, Dublin, pp. 51–60.
Ingwersen P (1996) Cognitive perspectives of information retrieval interaction. Journal of Documentation, 52(1): 3–59.
Lamping J and Rao R (1994) Laying out and visualising large trees using hyperbolic space. In: Proceedings of ACM-UIST 1994, pp. 13–14.
Lesk ME and Salton G. (1969) Relevance assessments and retrieval system evaluation. Information storage and retrieval, 4:343–359.
Quine WvO (1953) From a logical point of view. Harvard University Press.
Quine WvO (1969) Ontological relativity and other essays. Columbia University Press.
Ounis I and Pasca M (1998) Modelling, indexing and retrieving images using conceptual graphs. In: Proceedings of the 9th International Conference on Database and Expert Systems Applications (DEXA'98), Vienna, pp. 24–28.
Oxford English Dictionary CD-ROM Edition (1993) Oxford University Press.
Porter MF (1980) An algorithm for suffix stripping. Program, 14(3):130–137.
Rennison E (1994) Galaxy of news. An approach to visualising and understanding expansive news landscapes. In:Proceedings of ACM-UIST 1994.
Robertson SE and Sparck Jones K (1976) Relevance weighting of search terms. Journal of the American Society of Information Science, 27:129–146.
Sanderson M (1994) Word Sense Disambiguation and Information Retrieval. In: Croft WB and van Rijsbergen CJ, Eds. Proceedings of ACM SIGIR conference, pp. 142–151.
Saracevic T (1996) Relevance reconsidered '96. In: Ingwersen P and Pors NO, Eds. Information Science: Integration in Perspective, Proceedings of the CoLIS-2 conference, Copenhagen, pp. 201–218.
Thompson RH (1989) The design and implementation of an intelligent interface for information retrieval. Ph.D. thesis, University of Massachusetts.
Voorhees EM (1998) Variations in relevance judgements and the measurement of retrieval effectiveness. In: Proceedings of ACM SIGIR '98, Melbourne, pp. 315–323.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Campbell, I. Interactive Evaluation of the Ostensive Model Using a New Test Collection of Images with Multiple Relevance Assessments. Information Retrieval 2, 89–114 (2000). https://doi.org/10.1023/A:1009902203782
Issue Date:
DOI: https://doi.org/10.1023/A:1009902203782