Abstract
Effectiveness is a primary concern in the information retrieval (IR) field. Various metrics for IR effectiveness have been proposed in the past; we take into account all the 44 metrics we are aware of, classifying them into a two-dimensional grid. The classification is based on the notions of relevance, i.e., if (or how much) a document is relevant, and retrieval, i.e., if (how much) a document is retrieved. To our knowledge, no similar classification has been proposed so far.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Belew, R.: Finding Out About. Cambridge Univ. Press, Cambridge (2000)
Borlund, P., Ingwersen, P.: Measures of relative relevance and ranked half-life: Performance indicators for interactive IR. In: 21st SIGIR, pp. 324–331 (1998)
Brache, R.: Personal communication (2005)
Buckley, C., Voorhees, E.: Retrieval evaluation with incomplete information. In: 27th SIGIR, pp. 25–32 (2004)
Cooper, W.S.: Expected search length: A single measure of retrieval effectiveness based on weak ordering action of retrieval systems. JASIST 19, 30–41 (1968)
de Vries, A., Kazai, G., Lalmas, M.: Tolerance to irrelevance: A user-effort oriented evaluation of retrieval systems without predefined retrieval unit. In: RIAO 2004 Conference Proceedings, pp. 463–473 (2004)
Della Mea, V., Mizzaro, S.: Measuring retrieval effectiveness: A new proposal and a first experimental validation. JASIST 55(6), 530–543 (2004)
Frei, H., Schauble, P.: Determining the effectiveness of retrieval algorithms. IPM 27(2), 153–164 (1991)
Järvelin, K., Kekäläinen, J.: Cumulated gain-based evaluation of IR techniques. TOIS 20, 422–446 (2002)
Kando, N., Kuriyama, K., Yoshioka, M.: Information retrieval system evaluation using multi-grade relevance judgments. In: IPSJ SIGNotes (2001)
Kazai, G.: Report of the INEX 2003 metrics working group. In: Proceedings of the 2nd INEX Workshop, pp. 184–190 (2004)
Kazai, G., Lalmas, M.: INEX, evaluation metrics (2005), http://inex.is.informatik.uni-duisburg.de/2005/inex-2005-metricsv4.pdf
Korfhage, R.R.: Information Storage and Retrieval. John Wiley & Sons, Chichester (1997)
Losee, R.M.: Upper bounds for retrieval performance and their use measuring performance and generating optimal boolean queries. Can it get any better than this? IPM 30(2), 193–204 (1994)
Mizzaro, S.: Relevance: The whole history. JASIS 48(9), 810–832 (1997)
Piwowarski, B., Gallinari, P.: Expected ratio of relevant units: A measure for structured information retrieval. In: Proceedings of INEX 2003, pp. 158–166 (2004)
Sakai, T.: New performance metrics based on multigrade relevance: Their application to question answering. In: NTCIR 4 Meeting Working Notes (2004)
Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw-Hill, New York (1984)
van Rijsbergen, C.J.: Information Retrieval. Butterworths, 2nd edn. (1979)
Yao, Y.Y.: Measuring retrieval effectiveness based on user preference of documents. JASIS 46(2), 133–145 (1995)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Demartini, G., Mizzaro, S. (2006). A Classification of IR Effectiveness Metrics. In: Lalmas, M., MacFarlane, A., Rüger, S., Tombros, A., Tsikrika, T., Yavlinsky, A. (eds) Advances in Information Retrieval. ECIR 2006. Lecture Notes in Computer Science, vol 3936. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11735106_48
Download citation
DOI: https://doi.org/10.1007/11735106_48
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-33347-0
Online ISBN: 978-3-540-33348-7
eBook Packages: Computer ScienceComputer Science (R0)