Abstract
Consider information retrieval systems that respond to a query (a natural language statement of a topic, an information need) with an ordered list of 1000 documents from the document collection. From the responses to queries that all express the same topic, one can discern how the words associated with a topic result in particular system behavior. From what is discerned from different topics, one can hypothesize abstract topic factors that influence system performance. An example of such a factor is the specificity of the topic's primary key word. This paper shows that statements about the effect of abstract topic factors on system performance can be supported empirically. A combination of statistical methods is applied to system responses from NIST's Text REtrieval Conference. We analyze each topic using a measure of irrelevant-document exclusion computed for each response and a measure of dissimilarity between relevant-document return orders computed for each pair of responses. We formulate topic factors through graphical comparison of measurements for different topics. Finally, we propose for each topic a four-dimensional summarization that we use to select topic comparisons likely to depict topic factors clearly.
Article PDF
Similar content being viewed by others
References
Allan J, Connell WB, Croft WB, Feng F-F, Fisher D and Li X (2001) INQUERY and TREC-9. In: Voorhees EM and Harman DK, Eds., The Ninth Text REtrieval Conference (TREC-9), NIST Special Publication 500–249. U.S. Government Printing Office, Washington, DC, pp. 551–562. (Available at http://trec.nist.gov)
Banks D, Over P and Zhang N (1999) Blind men and elephants: Six approaches to TREC data. Information Retrieval, 1:7–34.
Bartholomew DJ (1996) The Statistical Approach to Social Measurement. Academic Press, San Diego.
Bartholomew DJ and Knott M (1999) Latent Variable Models and Factor Analysis. Oxford University Press, NewYork.
Berry MW and Browne M (1999) Understanding Search Engines: Mathematical Modeling and Text Retrieval. Society for Industrial and Applied Mathematics, Philadelphia.
Buckley C (2001) The TREC-9 query track. In: Voorhees EM and Harman DK, Eds., The Ninth Text Retrieval Conference (TREC-9), NIST Special Publication 500–249. U.S. Government Printing Office, Washington, DC, pp. 81–86. (Available at http://trec.nist.gov)
Buckley C and Walz J (2001) Sabir research at TREC-9. In: Voorhees EM and Harman DK, Eds., The Ninth Text REtrieval Conference (TREC-9), NIST Special Publication 500–249. U.S. Government Printing Office, Washington, DC, pp. 475–478. (Available at http://trec.nist.gov)
Cox TF and Cox MAA (2001) Multidimensional Scaling, 2 edition. Chapman & Hall, London.
Gibbons JD (1985) Nonparametric Statistical Inference, Marcel Dekker, New York, pp. 226–235.
Kruskal JB and Wish M (1978) Multidimensional Scaling. SAGE Publications, Newbury Park, CA.
Liggett W (1999) Topic by topic performance of information retrieval systems. In: Voorhees EM and Harman DK, Eds., The Seventh Text REtrieval Conference (TREC-7), NIST Special Publication 500–242. U.S. Government Printing Office, Washington, DC, pp. 105–114. (Available at http://trec.nist.gov)
Liggett W and Buckley C (2001) Query expansion seen through return order of relevant documents. In: Voorhees EM and Harman DK, Eds., The Ninth Text REtrieval Conference (TREC-9), NIST Special Publication 500–249. U.S. Government Printing Office, Washington, DC, pp. 51–70. (Available at http://trec.nist.gov)
Robertson SE and Walker S (2001) Microsoft Cambridge at TREC-9: Filtering track. In: Voorhees EM and Harman DK, Eds., The Ninth Text REtrieval Conference (TREC-9), NIST Special Publication 500–249. U.S. Government Printing Office, Washington, DC, pp. 361–368. (Available at http://trec.nist.gov)
Rorvig M (1999) Images of similarity: A visual exploration of optimal similarity metrics and scaling properties of TREC topic-document sets. Journal of the American Society for Information Science, 50:639–651.
Tomlinson S and Blackwell T (2001) Hummingbird's Fulcrum SearchServer at TREC-9. In: Voorhees EM and Harman DK, Eds., The Ninth Text REtrieval Conference (TREC-9), NIST Special Publication 500–249. U.S. Government Printing Office, Washington, DC, pp. 209–222. (Available at http://trec.nist.gov).
Venables WN and Ripley BD (1999) Modern Applied Statistics with S-PLUS, 3 edition. Springer-Verlag, New York.
Voorhees EM and Harman D (2000) Overview of the Eighth Text REtrieval Conference (TREC-8). In: Voorhees EM and Harman DK, Eds., The Eighth Text REtrieval Conference (TREC-8). NIST Special Publication 500–246. U.S. Government Printing Office, Washington, DC, pp. 1–24. (Available at http://trec.nist.gov).
Voorhees EM and Harman D (2001) Overview of the Ninth Text Retrieval Conference (TREC-9). In: Voorhees EM and Harman DK, Eds., The Ninth Text REtrieval Conference (TREC-9), NIST Special Publication 500–249. U.S. Government Printing Office, Washington, DC, pp. 1–14. (Available at http://trec.nist.gov)
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Liggett, W., Buckley, C. System Performance and Natural Language Expression of Information Needs. Information Retrieval 8, 101–128 (2005). https://doi.org/10.1023/B:INRT.0000048493.67375.93
Issue Date:
DOI: https://doi.org/10.1023/B:INRT.0000048493.67375.93