skip to main content
10.1145/1645953.1646294acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
poster

Measuring system performance and topic discernment using generalized adaptive-weight mean

Published: 02 November 2009 Publication History

Abstract

Standard approaches to evaluating and comparing information retrieval systems compute simple averages of performance statistics across individual topics to measure the overall system performance. However, topics vary in their ability to differentiate among systems based on their retrieval performance. At the same time, systems that perform well on discriminative queries demonstrate notable qualities that should be reflected in the systems' evaluation and ranking. This motivated research on alternative performance measures that are sensitive to the discriminative value of topics and the performance consistency of systems. In this paper we provide a mathematical formulation of a performance measure that postulates the dependence between the system and topic characteristics. We propose the Generalized Adaptive-Weight Mean (GAWM) measure and show how it can be computed as a fixed point of a function for which the Brouwer Fixed Point Theorem applies. This guarantees the existence of a scoring scheme that satisfies the starting axioms and can be used for ranking of both systems and topics. We apply our method to TREC experiments and compare the GAWM with the standard averages used in TREC.

References

[1]
Aslam, J. and Pavlu, V. 2007. Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions. In Proceedings of ECIR, 198--209.
[2]
Carmel, D., Yom-Tov, E., Darlow, A., and Pelleg, D. 2006. What makes a query difficult?. In Proc. of SIGIR, 390--397.
[3]
Cronen-Townsend, S., Zhou, Y., and Croft, W. B. 2002. Predicting query performance. In Proc. of SIGIR, 299--306.
[4]
Diaz, F. 2007. Performance prediction using spatial autocorrelation. In Proc. of SIGIR, 583--590.
[5]
Griffel, D. H. Applied Functional Analysis. Dover Publications, Jun 2002.
[6]
Ignjatović, A., Lee, C. T., Kutay, C., Guo H. and Compton, P. 2009. Computing Marks from Multiple Assessors Using Adaptive Averaging. In Proc. of ICEE&ICEER.
[7]
Mizzaro, S. and Robertson, S. 2007. HITS hits TREC: exploring IR evaluation results with network analysis. In Proc. of SIGIR, 479--486.
[8]
Rijsbergen, C. J. van. Information Retrieval, Butterworths, London, 1979.
[9]
Robertson, S. 2006. On GMAP: and other transformations. In Proc. of CIKM, 78--83.
[10]
Vinay, V., Cox, I. J., Milic-Frayling, N., and Wood, K. 2006. On ranking the effectiveness of searches. In Proc. of SIGIR, 398--404.
[11]
Voorhees, E. M. and Harman, D. K. 2005 TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing). The MIT Press.
[12]
Voorhees, E. M. 2006. The TREC 2005 robust track. SIGIR Forum 40(1), 41--48.
[13]
Webber, W., Moffat, A., and Zobel, J. 2008. Score standardization for inter-collection comparison of retrieval systems. In Proc. of SIGIR, 51--58.
[14]
Zhou, Y. and Croft, W. B. 2006. Ranking robustness: a novel framework to predict query performance. In Proc. of CIKM, 567--574.

Cited By

View all
  • (2016)A Simple and Effective Approach to Score StandardisationProceedings of the 2016 ACM International Conference on the Theory of Information Retrieval10.1145/2970398.2970399(95-104)Online publication date: 12-Sep-2016
  • (2010)Measuring the variability in effectiveness of a retrieval systemProceedings of the First international Information Retrieval Facility conference on Adbances in Multidisciplinary Retrieval10.1007/978-3-642-13084-7_7(70-83)Online publication date: 31-May-2010

Index Terms

  1. Measuring system performance and topic discernment using generalized adaptive-weight mean

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge management
      November 2009
      2162 pages
      ISBN:9781605585123
      DOI:10.1145/1645953
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 02 November 2009

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. performance metrics
      2. system performance
      3. topic discernment

      Qualifiers

      • Poster

      Conference

      CIKM '09
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

      Upcoming Conference

      CIKM '25

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)3
      • Downloads (Last 6 weeks)0
      Reflects downloads up to 16 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2016)A Simple and Effective Approach to Score StandardisationProceedings of the 2016 ACM International Conference on the Theory of Information Retrieval10.1145/2970398.2970399(95-104)Online publication date: 12-Sep-2016
      • (2010)Measuring the variability in effectiveness of a retrieval systemProceedings of the First international Information Retrieval Facility conference on Adbances in Multidisciplinary Retrieval10.1007/978-3-642-13084-7_7(70-83)Online publication date: 31-May-2010

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media