Abstract
The BM25 similarity computation has been shown to provide effective document retrieval. In operational terms, the formulae which form the basis for BM25 employ both term frequency and document length normalization. This paper considers an alternative form of normalization using document-centric impacts, and shows that the new normalization simplifies BM25 and reduces the number of tuning parameters. Motivation is provided by a preliminary analysis of a document collection that shows that impacts are more likely to identify documents whose lengths resemble those of the relevant judgments.Experiments on TREC data demonstrate that impact-based BM25 is as good as or better than the original term frequency-based BM25 in terms of retrieval effectiveness.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Anh, V.N., Moffat, A.: Simplified similarity scoring using term ranks. In: Marchionini, G., Moffat, A., Tait, J., Baeza-Yates, R., Ziviani, N. (eds.) Proc. 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 2005, pp. 226–233. ACM Press, New York (2005)
Buckley, C., Salton, G., Allan, J.: Automatic retrieval with locality information using SMART. In: Harman, D.K. (ed.) Proceedings of the First Text REtrieval Conference (TREC-1), November 1993, pp. 59–72. National Institute of Standards and Technology (Special Publication 500-251), Gaithersburg (1993)
Clarke, C.L.A., Fuhr, N., Kando, N., Kraaij, W., de Vries, A.P. (eds.): Proc. 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands, July 2007. ACM Press, New York (2007)
Cleveland, W.S., Grosse, E., Shyu, W.M.: Local regression models. In: Chambers, J.M., Hastie, T.J. (eds.) Statistical Models in S, ch. 8. Chapman & Hall/CRC, Boca Raton (1992)
Fuhr, N.: Models in information retrieval. Lectures on information retrieval, pp. 21–50. Springer, Heidelberg (2001)
Metzler, D., Croft, W.B.: Latent concept expansion using Markov random field. In: Clarke, et al. (eds.), pp. 311–318 (2007)
Metzler, D., Strohman, T., Croft, W.B.: A statistical view of binned retrieval models. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds.) ECIR 2008. LNCS, vol. 4956, pp. 175–186. Springer, Heidelberg (2008)
Robertson, S., Spärck Jones, K.: Relevance weighting of search terms. Journal of the American Society for Information Science 27(3), 129–146 (1976)
Robertson, S., Walker, S., Jones, S., Hancock-Beaulieu, M., Gatford, M.: Okapi at TREC–3. In: Harman, D. (ed.) Proc. Third Text REtrieval Conference (TREC–3), pp. 109–126. National Institute of Standards and Technology (Special Publication 500-225), Gaithersburg (1994)
Robertson, S., Walker, S., Beaulieu, M.: Okapi at TREC–7: automatic ad hoc, filtering, VLC and filtering tracks. In: Voorhees, E., Harman, D. (eds.) Proc. Seventh Text REtrieval Conference (TREC–7), November 1998, pp. 253–261. National Institute of Standards and Technology (Special Publication 500-242) (1998)
Salton, G., Buckley, C.: Term-weighting approaches in automatic text retrieval. Information Processing and Management 24(5), 513–523 (1988)
Singhal, A., Buckley, C., Mitra, M.: Pivoted document length normalization. In: Frei, H., Harman, D., Schäuble, P., Wilkinson, R. (eds.) Proc. 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 21–29. ACM Press, New York (1996)
Singhal, A., Salton, G., Mitra, M., Buckley, C.: Document length normalization. Information Processing and Management 32(5), 619–633 (1996)
Spärck Jones, K., Walker, S., Robertson, S.: A probabilistic model of information retrieval: Development and comparative experiments (Parts 1 and 2). Information Processing and Management 36, 779–808, 809–840 (2000)
Tao, T., Zhai, C.: An exploration of proximity measures in information retrieval. In: Clarke, et al. (eds.), pp. 295–302 (2007)
Zobel, J., Moffat, A.: Exploring the similarity space. ACM SIGIR Forum 32(1), 18–34 (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Anh, V.N., Wan, R., Moffat, A. (2008). Term Impacts as Normalized Term Frequencies for BM25 Similarity Scoring. In: Amir, A., Turpin, A., Moffat, A. (eds) String Processing and Information Retrieval. SPIRE 2008. Lecture Notes in Computer Science, vol 5280. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-89097-3_7
Download citation
DOI: https://doi.org/10.1007/978-3-540-89097-3_7
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-89096-6
Online ISBN: 978-3-540-89097-3
eBook Packages: Computer ScienceComputer Science (R0)