Abstract
Some insight into the behavior of adaptive filtering systems may be gained by comparing them with similar ranked-output retrieval systems. This is not easy; however, a new optimization measure, introduced for the TREC-9 filtering track, makes some such comparison possible. A series of experiments using the TREC-9 filtering data shows that filtering effectiveness is comparable to routing effectiveness, and demonstrates the gains to be made from adaptation.
Article PDF
Similar content being viewed by others
References
Ault T and Yang Y (2001) KNN at TREC-9. In: Voorhees EM and Harman DK, Eds., The Ninth Text REtrieval Conference (TREC-9). MD, Gaithersburg, NIST (NIST Special Publication 500-249), pp. 127–134.
Buckley C, trec eval [program for generating a range of evaluation measures from ranked-output data and relevance judgements]. Available from ftp://ftp.cs.cornell.edu/pub/smart/.
Buckley C and Voorhees E (2000) Evaluating evaluation measure stability. In: SIGIR 2000: Proceedings of the 23rd Annual International Conference on Research and Development in Information Retrieval. ACM, pp. 33–40.
Cleverdon CW, Mills J and Keen EM (1966) Factors Determining the Performance of Indexing Systems. (2 vols.) Aslib Cranfield Research Project. College of Aeronautics, Cranfield, UK.
Hersh WR, Buckley C, Leone TJ and Hickam DH (1994) OHSUMED: An interactive retrieval evaluation and new large test collection for research. In: SIGIR'94: Proceedings of the 17th Annual ACM SIGIR Conference on Research and Development in Information Retrieval. Springer-Verlag, Berlin, pp. 192–201.
Hull DA(1999) The TREC-7 filtering track: Description and analysis. In:Voorhees EM and Harman DK, Eds., The Seventh Text REtrieval Conference (TREC-7). NIST, Gaithersburg, MD (NIST Special Publication 500–242), pp. 33–56.
Hull DA and Robertson SE (2000) The TREC-8 filtering track report. In: Voorhees EM and Harman DK, Eds., The 8th Text Retrieval Conference (TREC-8). NIST, Gaithersburg, MD (NIST Special Publication 500–246), pp. 35–56.
Järvelin K and Kekäläinen J (2000) IR evaluation methods for retrieving highly relevant documents. In: SIGIR 2000: Proceedings of the 23rd Annual International Conference on Research and Development in Information Retrieval. ACM, pp. 41–48.
Robertson SE (1997), Ed. Journal of Documentation, 53(1) (special issue).
Robertson SE (2002a) Threshold setting and performance optimization in adaptive filtering. Information Retrieval, 5:239–256.
Robertson SE (2002b) Introduction to the special issue: Overview of the TREC routing and filtering tasks. Information Retrieval, 5:127–137.
Robertson SE and Hull DA (2001) The TREC-9 filtering track final report. In: Voorhees EM and Harman DK, Eds., The Ninth Text REtrieval Conference (TREC-9). NIST, Gaithersburg, MD (NIST Special Publication 500–249), pp. 25–40.
Robertson SE and Walker S (2000a) Threshold setting in adaptive filtering. Journal of Documentation, 56:312–331.
Robertson SE and Walker S (2000b) Okapi/Keenbow at TREC-8. In: Voorhees EM and Harman DK, Eds., The Eighth Text REtrieval Conference (TREC-8). NIST, Gaithersburg, MD (NIST Special Publication no. 500–246), pp. 151–162.
Robertson SE and Walker S (2001) Microsoft Cambridge at TREC-9: Filtering track. In: The Ninth Text REtrieval Conference (TREC-9). NIST, Gaithersburg, MD (NIST Special Publication 500-249), pp. 361–368.
Robertson SE, Walker S and Beaulieu M(1999) Okapi at TREC-7: Automatic ad hoc, filtering,VLC and interactive track. In: Voorhees EM and Harman DK, Eds., The Seventh Text REtrieval Conference (TREC-7). NIST, Gaithersburg, MD (NIST Special Publication 500–242), pp. 253–264.
Sparck Jones K, Walker S and Robertson SE (2000) A probabilistic model of information retrieval: Development and comparative experiments. Information Processing and Management, 36:779–808 (Part 1) and 809–840 (Part 2).
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Robertson, S. Comparing the Performance of Adaptive Filtering and Ranked Output Systems. Information Retrieval 5, 257–268 (2002). https://doi.org/10.1023/A:1015754113585
Issue Date:
DOI: https://doi.org/10.1023/A:1015754113585